R installations possibly broken #919

martenson · 2015-10-14T22:44:53Z

ping @yhoogstrate @mvdbeek @bgruening @jmchilton

bgruening · 2015-10-15T06:41:09Z

working on this ...
Potential fix here: https://github.com/galaxyproject/galaxy/compare/dev...bgruening:R_return?expand=1
Just want to search for more test computers with different R versions and OS ...
R is not really portable and error handling seems to be very tricky :(

nsoranzo · 2015-10-15T11:42:52Z

@bgruening @yhoogstrate Your fix is wrong, it will return 0 also if Rscript fails for a random reason. The original problem is with the syntax of the error parameter of tryCatch. Can you test if this commit fixes the problem for you:

nsoranzo@73ca4c7

It works correctly for me on the command line.

bgruening · 2015-10-15T11:49:49Z

@nsoranzo for some reason the tryCatch is not working for all users.
Please see the latest discussion here:
bgruening#8

Including a new idea using Python to catch the error.

nsoranzo · 2015-10-15T12:02:47Z

I guess tryCatch was not working because of the problem fixed in nsoranzo/galaxy@73ca4c7 . I think my solution is less hacky and does not rely on grepping the Rscript stdout. I'm doing more testing now.

bgruening · 2015-10-15T12:05:46Z

@nsoranzo even the old version was working for me in one R version :)
If you think this works for every R version including R2 than I'm happy to take this one.

bgruening · 2015-10-15T12:13:22Z

This is what I get:

root@4e0ee2d18014:/galaxy-central# R --version
R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.

root@4e0ee2d18014:/galaxy-central# Rscript -e "tryCatch({ install.packages(c('BiocGenerics_0.14.0.tar.gz-'), lib = '/galaxy-central/tool_deps/deseq2/1.8.2/iuc/package_deseq2_1_8_2/00b14423d00f', repos=NULL, dependencies=FALSE) }, error = function(e) { print(e); quit(status = 1) } )"
Warning: invalid package ‘BiocGenerics_0.14.0.tar.gz-’
Error: ERROR: no packages specified
Warning message:
In install.packages(c("BiocGenerics_0.14.0.tar.gz-"), lib = "/galaxy-central/tool_deps/deseq2/1.8.2/iuc/package_deseq2_1_8_2/00b14423d00f",  :
  installation of package ‘BiocGenerics_0.14.0.tar.gz-’ had non-zero exit status
root@4e0ee2d18014:/galaxy-central# echo $?
0

yhoogstrate · 2015-10-15T13:24:27Z

$ R --version
R version 3.2.0 (2015-04-16) -- "Full of Ingredients"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.

Test 1: correct package, incorrect destination:

$ Rscript -e "tryCatch({ install.packages(c('BiocGenerics_0.14.0.tar.gz'), lib = '/dev/null', repos=NULL, dependencies=FALSE) }, error = function(e) { print(e); quit(status = 1) } )"
Warning in install.packages(c("BiocGenerics_0.14.0.tar.gz"), lib = "/dev/null",  :
  'lib = "/dev/null"' is not writable
<simpleError in install.packages(c("BiocGenerics_0.14.0.tar.gz"), lib = "/dev/null",     repos = NULL, dependencies = FALSE): unable to install packages>
$ echo $?
1

Test 2: incorrect package, correct destination:

$ Rscript -e "tryCatch({ install.packages(c('/dev/null'), lib = '/tmp/deseq2/inst', repos=NULL, dependencies=FALSE) }, error = function(e) { print(e); quit(status = 1) } )"
ERROR: cannot extract package from ‘/dev/null’
Warning message:
In install.packages(c("/dev/null"), lib = "/tmp/deseq2/inst", repos = NULL,  :
  installation of package ‘/dev/null’ had non-zero exit status
$ echo $?
0

Edit
Test 3: corrupt .tar.gz file, correct destination:

$ Rscript -e "tryCatch({ install.packages(c('Bio2.tar.gz'), lib = '/tmp/deseq2/inst2', repos=NULL, dependencies=FALSE) }, error = function(e) { print(e); quit(status = 1) } )"
Error in untar2(tarfile, files, list, exdir, restore_times) : 
  incomplete block on file
Warning message:
In install.packages(c("Bio2.tar.gz"), lib = "/tmp/deseq2/inst2",  :
  installation of package ‘Bio2.tar.gz’ had non-zero exit status
$ echo $?
0

nsoranzo · 2015-10-15T13:27:02Z

Yep, it looks like an invalid package name raise a warning instead of an error. Should we exit with status = 1 also for warnings?

bgruening · 2015-10-15T13:53:55Z

@nsoranzo I haven't done this because I fear that during compilations there are a lot of warnings.

nsoranzo · 2015-10-15T14:11:40Z

Should be fixed by nsoranzo@f666b05

yhoogstrate · 2015-10-15T14:19:50Z

@nsoranzo if ( grep('had non-zero exit status', as.character(w)) Pure ownage. Shame on us @bgruening 😆

The three tests cases I wrote in the comment above work currectly using nsoranzo@f666b05

bgruening · 2015-10-15T15:51:04Z

@nsoranzo your last one works for all of my examples. You probably want to include a few more strings to grep for, as shown by @yhoogstrate. Not sure if all are errors or a few of them are warnings.
Also we should consider that we need to add more filters later if R changes the error messages.

Please decide yourself if you want to have this one-liner or if we want to create a Python template without tryCatch(). The grep functions seems to be stable since 2.11 so this could be a valid solution.

nsoranzo · 2015-10-15T16:02:32Z

@yhoogstrate How can I reproduce unable to install packages and Execution halted errors/warnings?

yhoogstrate · 2015-10-16T06:41:22Z

@nsoranzo

$ Rscript -e "install.packages(c('BiocGenerics_0.14.0.tar.gz'),lib='/dev/null', repos=NULL, dependencies=FALSE)" 
Warning in install.packages(c("BiocGenerics_0.14.0.tar.gz"), lib = "/dev/null",  :
  'lib = "/dev/null"' is not writable
Error in install.packages(c("BiocGenerics_0.14.0.tar.gz"), lib = "/dev/null",  : 
  unable to install packages
Execution halted

This makes me realize that your grep should not be able to capture this warning, but it still returns an error. I think we should look a bit closer into it, unless I made a mistake in copy pasting:

$ Rscript -e "tryCatch( { install.packages(c('BiocGenerics_0.14.0.tar.gz'), lib = '/dev/null', repos = NULL, dependencies = FALSE) }, error = function(e) { print('--Error--'); quit(status = 1) }, warning = function(w) { if ( grep('ZZZZZZZZZZZZZZZZZZZZZZZZZZZZz', as.character(w)) ) { print('--Warning--'); quit(status = 1) } } )"
Error in if (grep("ZZZZZZZZZZZZZZZZZZZZZZZZZZZZz", as.character(w))) { : 
  argument is of length zero
Calls: tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted

nsoranzo · 2015-10-16T10:54:42Z

@yhoogstrate The problem is that I should have used grepl instead of grep (which returns an empty vector if there is no match). It should be finally fixed with nsoranzo@c9294a6 , where I also added all warnings found in the install.packages() source code.

yhoogstrate · 2015-10-16T11:32:44Z

@nsoranzo Thanks, it works here. I think you should make a PR and in the meantime we should test the packages within:

package_dexseq_1_14
package_deseq_2_8_2
package_r_ggplot2_0_9_3
package_r_edger_3_11_0

martenson · 2015-10-19T14:37:07Z

Awesome, thank you guys!

this is still not ideal, as the entire response will be loaded into memory. This is a problem with streaming responses. encode/starlette#1012 (comment) is actually enlightening here: > "This means this class will either load the entirety of streaming requests into memory (this issue) and run the background before returning the response (galaxyproject#919 ), or if we fix those problems, that it will then encourage users to leave resources in a pending or open state, an arguably worse result. In short, it's problematic." Which is exactly what we'd be doing with a middleware, keeping resources open for longer than necessary, which we need to avoid if we ever want to run background / async tasks. I think the solution here what we already had, path operation dependencies.

martenson added the kind/bug label Oct 14, 2015

nsoranzo mentioned this issue Oct 16, 2015

Fix syntax of error parameter of tryCatch in setup_r_environment action. #929

Merged

jmchilton closed this as completed in #929 Oct 19, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

R installations possibly broken #919

R installations possibly broken #919

martenson commented Oct 14, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

bgruening commented Oct 15, 2015

bgruening commented Oct 15, 2015

yhoogstrate commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

yhoogstrate commented Oct 15, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

yhoogstrate commented Oct 16, 2015

nsoranzo commented Oct 16, 2015

yhoogstrate commented Oct 16, 2015

martenson commented Oct 19, 2015

R installations possibly broken #919

R installations possibly broken #919

Comments

martenson commented Oct 14, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

bgruening commented Oct 15, 2015

bgruening commented Oct 15, 2015

yhoogstrate commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

yhoogstrate commented Oct 15, 2015

bgruening commented Oct 15, 2015

nsoranzo commented Oct 15, 2015

yhoogstrate commented Oct 16, 2015

nsoranzo commented Oct 16, 2015

yhoogstrate commented Oct 16, 2015

martenson commented Oct 19, 2015