ILP issue

[asumann]

Hi,

I am running CARNIVAL using saezlab/transcriptutorial. I tried using IBM cplex but it didn't work out and I kept getting this:

Error in resList[[1]] : subscript out of bounds
In addition: Warning messages:
1: In controlNodeIdentifiers(netObj = netObj) :
  Your network contains identifiers with '-' symbol and they will
            be replaced with '_'
2: In controlNodeIdentifiers(netObj = netObj) :
  Your network contains identifiers with '/' symbol and they will
            be replaced with '_'

Finally, I installed lpSolve in R & made a ~/.Renviron file with R_MAX_VSIZE=100Gb & removed unused variable from the environment. And I got this one:

> carnival_result = CARNIVAL::runCARNIVAL(inputObj= iniciators,
+                                measObj = tfList$t, 
+                                netObj = sif, 
+                                weightObj = progenylist$score, 
+                                solverPath = "/var/folders/5h/nb97vjfj5w13g__ksqy9z0b40000gn/T/RtmpzOFfha/downloaded_packages/lpSolve/R/lpSolve", 
+                                solver = "lpSolve")
Writing constraints...
Solving LP problem...

── Column specification ─────────────────────────────────────────────
cols(
  `enter Problem` = col_character()
)

Error: vector memory exhausted (limit reached?)
In addition: Warning messages:
1: In controlNodeIdentifiers(netObj = netObj) :
  Your network contains identifiers with '-' symbol and they will
            be replaced with '_'
2: In controlNodeIdentifiers(netObj = netObj) :
  Your network contains identifiers with '/' symbol and they will
            be replaced with '_'

Here is my R session:

R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] lpSolve_5.6.15      visNetwork_2.0.9    OmnipathR_2.1.1     yaml_2.2.1          rappdirs_0.3.3     
 [6] tidyr_1.1.2         dplyr_1.0.4         readr_1.4.0         RCurl_1.98-1.2      purrr_0.3.4        
[11] magrittr_2.0.1      progress_1.2.2      jsonlite_1.7.2      vsn_3.58.0          Biobase_2.50.0     
[16] BiocGenerics_0.36.0

loaded via a namespace (and not attached):
 [1] httr_1.4.2            mixtools_1.2.0        UniProt.ws_2.30.0     bit64_4.0.5           splines_4.0.2        
 [6] foreach_1.5.1         assertthat_0.2.1      BiocFileCache_1.14.0  BiocManager_1.30.10   affy_1.68.0          
[11] stats4_4.0.2          RBGL_1.66.0           blob_1.2.1            Category_2.56.0       viper_1.24.0         
[16] pillar_1.4.7          RSQLite_2.2.3         lattice_0.20-41       glue_1.4.2            limma_3.46.0         
[21] digest_0.6.27         colorspace_2.0-0      htmltools_0.5.1.1     preprocessCore_1.52.1 Matrix_1.3-2         
[26] GSEABase_1.52.1       XML_3.99-0.5          pkgconfig_2.0.3       logger_0.1            genefilter_1.72.1    
[31] CARNIVAL_1.2.0        zlibbioc_1.36.0       xtable_1.8-4          scales_1.1.1          affyio_1.60.0        
[36] annotate_1.68.0       tibble_3.0.6          generics_0.1.0        IRanges_2.24.1        ggplot2_3.3.3        
[41] ellipsis_0.3.1        cachem_1.0.3          cli_2.3.0             survival_3.2-7        crayon_1.4.0         
[46] memoise_2.0.0         doParallel_1.0.16     MASS_7.3-53           segmented_1.3-1       class_7.3-18         
[51] graph_1.68.0          tools_4.0.2           prettyunits_1.1.1     hms_1.0.0             lifecycle_0.2.0      
[56] S4Vectors_0.28.1      kernlab_0.9-29        munsell_0.5.0         AnnotationDbi_1.52.0  packrat_0.5.0        
[61] compiler_4.0.2        e1071_1.7-4           rlang_0.4.10          grid_4.0.2            iterators_1.0.13     
[66] rstudioapi_0.13       htmlwidgets_1.5.3     igraph_1.2.6          bitops_1.0-6          gtable_0.3.0         
[71] codetools_0.2-18      curl_4.3              DBI_1.1.1             R6_2.5.0              fastmap_1.1.0        
[76] bit_4.0.4             KernSmooth_2.23-18    Rcpp_1.0.6            vctrs_0.3.6           dbplyr_2.1.0         
[81] tidyselect_1.1.0     

I hope you can help me in that issue.

All the best,
Asuman

[enio23]

Hi Ansuman,

You having issues running the examples on the transcriptutorial or is it your own data? After installing CPLEX did you specify correctly the path where you have the cplex executable stored or you used the path which was set on the example? Can you please show us how you are using the runCARNIVAL function?

The memory issues with lpSolve are unfortunately to be expected since it relies on a matrix formulation of the ILP problems which can get quite large in size for real-case applications of CARNIVAL. We have only implemented lpSolve to run small test examples.

Cheers,
Enio

[asumann]

Hi Enio,

I couldn't successfully install CPLEX. It looks like a common problem for OSx Catalina users. They suggest installing light mode but that is not available online to download right now..

I am testing the tutorial and for this I use the data given in the tutorial. Here is the CARNIVAL part:

> carnival_result = CARNIVAL::runCARNIVAL(inputObj= iniciators,
+                                measObj = tfList$t, 
+                                netObj = sif, 
+                                weightObj = progenylist$score, 
+                                solverPath = "/var/folders/5h/nb97vjfj5w13g__ksqy9z0b40000gn/T/RtmpzOFfha/downloaded_packages/lpSolve/R/lpSolve", 
+                                solver = "lpSolve")

In lpSolve case, I give my path. But, it gives memory error so I assume path is not the problem. I installed CBC and got the same error I got from lpSolve. The problem is I cannot identify the error I got..

I tried online suggestions to successfully install CPLEX but it didn't work out. Here is the error I got after installing IBM CPLEX, maybe you can give some advice so that I can run CARNIVAL:

------Executable: cd "/Users/asu"; "/var/folders/5h/nb97vjfj5w13g__ksqy9z0b40000gn/T//IA759858543/IA759858543/jre/Contents/Home/jre/bin/java" -Xms16m -Xmx48m -cp "/Users/asu/cplex_studio2010-osx.app/Contents/Resources/Java/Execute.zip:/Users/asu/cplex_studio2010-osx.app/Contents/Resources/Java/IAClasses.zip:/Users/asu/cplex_studio2010-osx.app/Contents/Resources/Java/installscript.zip:/Users/asu/cplex_studio2010-osx.app/Contents/Resources/Java:/Users/asu/cplex_studio2010-osx.app/Contents/Resources/Java/linking.zip" -Dfile.encoding=UTF-8 -Djava.library.path=.:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:"/Users/asu":"/var/folders/5h/nb97vjfj5w13g__ksqy9z0b40000gn/T/IA759858543/IA759858543/jre/Contents/Home/jre/lib":"/Users/asu/cplex_studio2010-osx.app/Contents/Resources/Java":"/Users/asu/cplex_studio2010-osx.app/Contents/MacOS":"/Users/asu/cplex_studio2010-osx.app/Contents/Resources" -Xdock:name="CPLEX Optimization Studio 20.1" -Xdock:icon="/Users/asu/cplex_studio2010-osx.app/Contents/Resources/SelfExtractor.icns" com.zerog.lax.LAX "/var/folders/5h/nb97vjfj5w13g__ksqy9z0b40000gn/T/app.properties"-------

Invocation of this Java Application has caused an InvocationTargetException. This application will now exit. (LAX)

Stack Trace:
java.awt.IllegalComponentStateException: The frame is decorated
	at java.awt.Frame.setBackground(Frame.java:986)
	at Flexeraat7.setBackground(Unknown Source)
	at com.zerog.ia.installer.AAMgr.initAAMgr(Unknown Source)
	at com.zerog.ia.installer.LifeCycleManager.a5(Unknown Source)
	at com.zerog.ia.installer.LifeCycleManager.executeApplication(Unknown Source)
	at com.zerog.ia.installer.Main.main(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.zerog.lax.LAX.launch(Unknown Source)
	at com.zerog.lax.LAX.main(Unknown Source)
This Application has Unexpectedly Quit: Invocation of this Java Application has caused an InvocationTargetException. This application will now exit. (LAX)

-Asuman

[enio23]

Yes, it is best if you get CPLEx which you can get with no-cost here if you plan using it for academic purposes. You can just download it and find the cplex executable file and then store it anywhere on the computer.

For lpSolve it is not needed to set a path, however again lpSolve will not work for large/real case-studies.

Cheers,
Enio

[asumann]

How about this error:

Error: vector memory exhausted (limit reached?)
In addition: Warning messages:
1: In controlNodeIdentifiers(netObj = netObj) :
  Your network contains identifiers with '-' symbol and they will
            be replaced with '_'
2: In controlNodeIdentifiers(netObj = netObj) :
  Your network contains identifiers with '/' symbol and they will
            be replaced with '_'

I am now having this with CPLEX. How can I solve it?

All the best,
Asuman

[enio23]

I think you are still using lpSolve.

Did you set solver='cplex' and solverPath to the path where the cplex is?

[asumann]

Yes, I did that.

[enio23]

Can you please send me the part of the script where you are using the runCARNIVAL function? :) We usually did not face memory issues when using CPLEX.

[asumann]

Just to make sure i also removed lpSolve package from library.

carnival_result = CARNIVAL::runCARNIVAL(inputObj= iniciators,
                               measObj = tfList$t, 
                               netObj = sif, 
                               weightObj = progenylist$score) 
                               solverPath = "/Users/asu/cplex_studio2010-osx.app/Contents/MacOS/cplex_studio2010-osx", 
                               solver = "cplex")
                               #timelimit=7200,
                               #mipGAP=0,
                               #poolrelGAP=0 )

Many thanks Enio!!

[enio23]

Hmm ok. Not sure if the path you are setting does not actually point to the executable file. Does the cplex_studio2010-osx file you are pointing looks something like this. If yes, maybe can you also store this file somewhere else (i.e. Desktop, Documents, ..) and update the solverPath accordingly because not sure if using a dot character on the path string might mess up something when we read the solverPath parameter.

[enio23]

Maybe I figured the possible error. Try running:

carnival_result = CARNIVAL::runCARNIVAL(inputObj= iniciators,
                               measObj = tfList$t, 
                               netObj = sif, 
                               weightObj = progenylist$score,
                               solverPath = "/Users/asu/cplex_studio2010-osx.app/Contents/MacOS/cplex_studio2010-osx", 
                               solver = "cplex",
                               timelimit=7200,
                               mipGAP=0,
                               poolrelGAP=0 )

[asumann]

Hi Enio,

It is working now!! I want to explain how so it may help someone else save their time.

First of all, IBM CPLEX was not successfully installed to my OSx Catalina, thus runCARNIVAL was giving errors. Installation problem was solved after I disabled Dark Mode on my Mac. It was somehow interfering with the CPLEX installation. Unfortunately, I do not know about the bottom reasons.

After installing CPLEX successfully, RStudio session was aborted every time I run runCARNIVAL. I run the code in Terminal R console. Again interestingly, I had solverPath issue even though cplex exe path was correct as in:

 > file.exists("/Applications/CPLEX_Studio201/cplex/bin/x86-64_osx/cplex")
[1] TRUE

I moved the .exe to another directory:

> file.exists("/Users/asu/cplex")
[1] TRUE

Finally, it run smoothly though it was computationally heavy. Next time, I will try to run on uni's HPC.

All the best,

Asuman