.globals$py_config is not updated when changing conda environment

[GreenGrassBlueOcean]

Hi @kevinushey and @t-kalinowski,

I am working on an r-package and I want to use a specific miniconda python environment in the r package.
I notice however that after I use reticulate::use_miniconda(condaenv = "r-eikon", required = T) the internal global package value .globals$required_python_version is updated to the r-eikon conda environment but the value of .globals$py_config is left at the old conda environment (r-reticulate).

This leads to very weird behavior. E.g. when calling reticulate::py_module_available will return that already installed python modules are not available. However when opening miniconda and running conda activate r-eikon and conda list these are shown as installed in the r-eikon environment.

A possible solution for this would be to again initialize_python as currently it is only initialized once and when changing environments it is not initialized again.

reticulate/R/package.R

Lines 29 to 56 in da51570

	is_python_initialized <- function() {
	!is.null(.globals$py_config)
	}
	ensure_python_initialized <- function(required_module = NULL) {
	# nothing to do if python is initialized
	if (is_python_initialized())
	return()
	# give delay load modules priority
	use_environment <- NULL
	if (!is.null(.globals$delay_load_module)) {
	required_module <- .globals$delay_load_module
	use_environment <- .globals$delay_load_environment
	.globals$delay_load_module <- NULL # one shot
	.globals$delay_load_environment <- NULL
	.globals$delay_load_priority <- 0
	}
	# notify front-end (if any) that Python is about to be initialized
	callback <- getOption("reticulate.python.beforeInitialized")
	if (is.function(callback))
	callback()
	# perform initialization
	.globals$py_config <- initialize_python(required_module, use_environment)

So maybe in use_python() after the following line:

reticulate/R/use_python.R

Line 112 in 1523ef5

.globals$use_python_versions <- unique(c(.globals$use_python_versions, python))

When there is a difference between .globals$required_python_version and .globals$py_config

to add something analogue to this:
.globals$py_config <- initialize_python(some variables)

Reproducible bug:

> reticulate::conda_create(envname = "r-eikon", python_version = "3.8")
> reticulate::use_miniconda(condaenv = "r-eikon", required = T)
> reticulate::py_config()
python:         C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython:      C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate/python38.dll
pythonhome:     C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate
version:        3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:19:05) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.22.1
> reticulate::conda_list()
          name                                                                            python
1         base                     C:\\Users\\XXXX\\AppData\\Local\\r-miniconda/python.exe
2      r-eikon      C:\\Users\\XXXX\\AppData\\Local\\r-miniconda\\envs\\r-eikon/python.exe
3 r-reticulate C:\\Users\\XXXX\\AppData\\Local\\r-miniconda\\envs\\r-reticulate/python.exe
> reticulate::use_miniconda(condaenv = "r-eikon", required = T)
> reticulate::py_config()
python:         C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython:      C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate/python38.dll
pythonhome:     C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate
version:        3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:19:05) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.22.1
> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
system code page: 65001

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 reticulate_1.24 (and many more ofcourse)      

[GreenGrassBlueOcean]

I have forked the project and added .globals$py_config <- initialize_python(some variables) to use_python:
GreenGrassBlueOcean@b3f1abf

Now reticulate::py_config() shows the correct conda environment.
But still the python module cannot be found.

> use_miniconda(condaenv = "r-eikon", required = T)
> reticulate::py_config()
python:         C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon/python.exe
libpython:      C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon/python38.dll
pythonhome:     C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon
version:        3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:19:05) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/XXXX/AppData/Local/r-miniconda/envs/r-eikon/Lib/site-packages/numpy
numpy_version:  1.22.1

NOTE: Python version was forced by use_python function
> reticulate::py_module_available(module = "eikon")
[1] FALSE

However in the miniconda terminal the package can be spotted. This means that still not the correct conda environment is used.

(base) PS C:\Users\XXXX> conda activate r-eikon
(r-eikon) PS C:\Users\XXXX> conda list
# packages in environment at C:\Users\XXXX\AppData\Local\R-MINI~1\envs\r-eikon:
#
# Name                    Version                   Build  Channel
anyio                     3.5.0                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
ca-certificates           2021.10.8            h5b45459_0    conda-forge
certifi                   2021.10.8                pypi_0    pypi
chardet                   3.0.4                    pypi_0    pypi
charset-normalizer        2.0.11                   pypi_0    pypi
datetime                  4.3                      pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
deprecation               2.1.0                    pypi_0    pypi
eikon                     1.1.6.post2              pypi_0    pypi
eventemitter              0.2.0                    pypi_0    pypi
h11                       0.12.0                   pypi_0    pypi
h2                        3.2.0                    pypi_0    pypi
hpack                     3.0.0                    pypi_0    pypi
httpcore                  0.13.7                   pypi_0    pypi
httpx                     0.19.0                   pypi_0    pypi
hyperframe                5.2.0                    pypi_0    pypi
idna                      2.10                     pypi_0    pypi
nest-asyncio              1.3.3                    pypi_0    pypi
numpy                     1.22.1                   pypi_0    pypi
openssl                   3.0.0                h8ffe710_2    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    1.4.0                    pypi_0    pypi
pip                       22.0               pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.7                    pypi_0    pypi
python                    3.8.12          h900ac77_2_cpython    conda-forge
python-configuration      0.8.2                    pypi_0    pypi
python-dateutil           2.8.2                    pypi_0    pypi
python_abi                3.8                      2_cp38    conda-forge
pytz                      2021.3                   pypi_0    pypi
refinitiv-dataplatform    1.0.0a13                 pypi_0    pypi
requests                  2.27.1                   pypi_0    pypi
rfc3986                   1.5.0                    pypi_0    pypi
scipy                     1.7.3                    pypi_0    pypi
setuptools                60.5.0           py38haa244fe_0    conda-forge
six                       1.16.0                   pypi_0    pypi
sniffio                   1.2.0                    pypi_0    pypi
sqlite                    3.37.0               h8ffe710_0    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
urllib3                   1.26.8                   pypi_0    pypi
validators                0.18.2                   pypi_0    pypi
vc                        14.2                 hb210afc_6    conda-forge
vs2015_runtime            14.29.30037          h902a5da_6    conda-forge
watchdog                  2.1.6                    pypi_0    pypi
websocket-client          1.2.3                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
zope-interface            5.4.0                    pypi_0    pypi
(r-eikon) PS C:\Users\

[GreenGrassBlueOcean]

Hi @t-kalinowski,

please any response?

[t-kalinowski]

Hi, apologies for the taking so long to respond. I'm not quite sure I understand the issue.

Changing reticulate:::.globals$py_config won't change which Python reticulate bound to.

When reticulate is first loaded, reticulate:::.globals$py_config is NULL, and then that value gets updated when Python is initialized. Python can only be initialized once per R session; support for dynamically binding to and releasing multiple Pythons in one R session is not supported, and probably won't ever be. Updating the value of reticulate:::.globals$py_config once Python is initialized won't change the fact that the Python session is initialized already.

[GreenGrassBlueOcean]

@t-kalinowski Thanks a lot for your reply, much appreciated!

The issue occurs if you want to use two conda environments the default environment r-reticulate and a new conda environment r-eikon only for the python packages required by an r-package.

In that case reticulate::use_miniconda(condaenv = "r-eikon", required = T) will not switch to the new environments. This is proven by calling reticulate::py_module_available afterwards. This will only return TRUE if the python packages are listed in r-reticulate and FALSE when they are in the new environment r-eikon and not also in r-reticulate.

To workaround this you need to install all packages in the default environment r-reticulate which you don't want as you want to isolate the specific python packages I want to use from other already installed packages in r-reticulate environment. This is done in order to prevent dependency issues with already installed packages in r-reticulate.

While reading your response I was starting to get the (maybe wrong) impression that making a custom conda environment for an r package becomes useless as this cannot be used.

Hope this clarifies it a bit. It is indeed a complex issue. If my explanation is still lacking clarity please let me know!

[t-kalinowski]

I think that when you are writing an R package that uses reticulate, you can aim for convenience by hooking into reticulate::configure_environment, and/or by calling use_condaenv("r-eikon", required = FALSE) in .onLoad(). However requiring that the package works only with one specific python installation feels overly restrictive.

In the tensorflow ecosystem we like to provide a install_<name> function that users can call to install the Python modules into their Python installation of choice.

In all cases though, once reticulate's Python session is initialized and bound to one Python, it cannot then initialize another Python. If you absolutely need to use only one Python conda environment in your R package, then you would need to raise an error if "wrong" Python is already initialized.

[GreenGrassBlueOcean]

Many thanks for your insightful reply!

I have written an install_eikon function (similar to that of tensorflow) that creates a conda environment with only the required python packages. However I guess from reading your reply it is maybe better to just make a virtual environment (venv) inside the users' default conda environment and activate this virtual environment when loading the package.
Because I guess the chances are quite high that the default python environment is already activated when a user is loading my package.

The reason that I am advocating for using conda or venv is that I am very afraid of python package version dependency issues inside the user's default python session, this can possibly lead to all kind of unexpected behavior inside my r package which is very hard to predict and to check for in unit tests. In my eyes having a shielded python environment would be highly desirable.

I would like to end with the fact that reticulate is an amazing package which allows me to run many python packages inside r. Thanks a lot for this package.