py_install() hardwires the paths for virtualenv instead of following PATH

[cboettig]

Here's a reprex:

library(reticulate)
  # fails
py_install("h5py", "/opt/venv")
#> Error: Prerequisites for installing Python packages not available.
#> 
#> Please install the following Python packages before proceeding: virtualenv
  # but this works
virtualenv_install("/opt/venv", "h5py")
#> Using virtual environment '/opt/venv' ...
  
  ## here's py_config:
py_config()
#> python:         /opt/venv/bin/python
#> libpython:      /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
#> pythonhome:     /usr:/usr
#> virtualenv:     /opt/venv/bin/activate_this.py
#> version:        2.7.13 (default, Sep 26 2018, 18:42:22)  [GCC 6.3.0 20170516]
#> numpy:          /opt/venv/local/lib/python2.7/site-packages/numpy
#> numpy_version:  1.16.3
#> 
#> python versions found: 
#>  /opt/venv/bin/python
#>  /usr/bin/python
#>  /usr/bin/python3

^{Created on 2019-05-12 by the reprex package (v0.2.1)}

Not clear to me why py_install believes virtualenv is not installed. Testing this on Debian 9 with this setup

[cboettig]

Digging into the reticulate code, it looks like this is due to the hardwired binary paths in reticulate:::python_unix_binary("virtualenv"), which is called by checks inside py_install() but not by virtualenv_install().

reticulate:::python_unix_binary
function (bin) 
{
    locations <- file.path(c("/usr/bin", "/usr/local/bin", path.expand("~/.local/bin")), 
        bin)
    locations <- locations[file.exists(locations)]
    if (length(locations) > 0) 
        locations[[1]]
    else NULL
}
<bytecode: 0x55f506dd6fd0>
<environment: namespace:reticulate>

Is there a reason why reticulate choses to hardwire some common unix paths while ignore PATH (and ignoring the path set with either use_virtualenv() or RETICULATE_PYTHON)? Is there a way to override this?

Also, here's sessionInfo() for reference:

sessionInfo()
#> R version 3.6.0 (2019-04-26)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Debian GNU/Linux 9 (stretch)
#> 
#> Matrix products: default
#> BLAS/LAPACK: /usr/lib/libopenblasp-r0.2.19.so
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] compiler_3.6.0  magrittr_1.5    tools_3.6.0     htmltools_0.3.6
#>  [5] yaml_2.2.0      Rcpp_1.0.1      stringi_1.4.3   rmarkdown_1.12 
#>  [9] highr_0.8       knitr_1.22      stringr_1.4.0   xfun_0.6       
#> [13] digest_0.6.18   evaluate_0.13

Thanks!

[ha0ye]

@cboettig There's a general problem that R isn't configured to see PATH as generally used on linux & Mac. I think this is due to system(), which invokes the Bourne shell (/bin/sh), thus bypassing any configurations that update PATH in e.g. .bash_profile.

You can test this out by comparing some results with Sys.which() in R vs. which on the command line.

[kevinushey]

I think this issue should be resolved now -- we no longer use python_unix_binary(); rather, we use the version of Python associated with the virtual environment that was requested.

[cboettig]

Thanks @kevinushey , things are so much better with the current github version than the current CRAN version (1.15).

I still cannot switch between two different virtualenvs in the same R session:

library(reticulate)
## Set up two separate virtualenvs
virtualenv_install(envname="/opt/venv/tf1", 
                                             packages = c("tensorflow==1.14.0", 
                                                                   "tensorflow_probability==0.7.0") )
virtualenv_install(envname="/opt/venv/tf2", 
                                             packages = c("tensorflow==2.0.0") )

## we can successfully switch into venv 1:
use_virtualenv("/opt/venv/tf1")
tensorflow::tf_version()

## but then we're stuck with it until we restart the R session
## no warning or error message that we're still in the former env
use_virtualenv("/opt/venv/tf2")
tensorflow::tf_version()

Let me know if this would be better put in a different thread.

[kevinushey]

Thanks @kevinushey , things are so much better with the current github version than the current CRAN version (1.15).

I'm confused -- the current CRAN version basically is the GitHub version right now?

The issue you indicate is known, though -- Python can only be initialized once per session; if you want to re-initialize Python with a new virtual environment you need to restart R.

[cboettig]

Thanks @kevinushey , I was a release behind on CRAN.

Re virtualenv, thanks, that makes sense as well. Would it be possible for use_virualenv() to throw an error or warning about this? It's a little surprising to run the function and not get an error or the effect the function says it will have.

[kevinushey]

This is the behavior of use_virtualenv(..., required = TRUE). By default, the behavior is advisory. (I'd like to change this in the future if possible, but worry a bit about breaking existing scripts in the wild)

[cboettig]

Makes sense, thanks for the reply and sorry I didn't get that earlier!