Guidance on best practices for using renv with python virtualenvs

[mattwarkentin]

Hi Kevin,

I am starting to use {renv} more and more to make sure my R-centric projects are truly self-contained and reproducible. But I have a specific project that also happens to also use python quite a bit and I'm looking for a little guidance on best practices for setting up a virtualenv using {renv} with the least amount of friction.

As an example, I have a virtualenv which lives in the default location on my Mac (/Users/matt/.virtualenvs/r-reticulate). This virtualenv is the one I generally bind to when using {reticulate} because I have installed all of the python packages I need in that envrionment. So when I am ready to use python in my renv-initialized project, I call renv::use_python("/Users/matt/.virtualenvs/r-reticulate/bin/python") and point it toward the python binary found in this virtualenv. This creates a virtualenv in the renv/ directory, but that newly created virtual environment doesn't contain any packages.

I guess my expectation was that since I define my python dependencies in my R scripts via alias <- import("package") declarations, that I maybe expected the virtualenv would be set up in a similar way to how the R library is set up, such that renv would detect my imports and make sure they are included in the renv/python/virtualenvs that was created.

So it seems like I have two options to get the new virtual environment up to speed, (1) start installing python packages with reticulate::virtualenv_install() and they will now be installed in my renv/python/virtualenv/renv-python..., or (2) I can export the requirements.txt from the /Users/matt/.virtualenvs/r-reticulate and restore them. Both of these are a little more manual than I was hoping.

Any suggestions? Is my mental model of how python is managed by renv completely misguided?

Thanks for your help!

[mattwarkentin]

While I'm thinking out loud (and assuming my expectations are not entirely off the mark), I could envision two possible solutions:

• The simple one would probably be a function like renv::clone_pythonenv() which would work like renv::use_python() but it actually clones an entire virtual environment, and doesn't just the using the python binary.

• Or a more extreme solution: Add an argument template to use_python() whereby template points to a virtual environment that contains the necessary packages and versions that are desired in the new virtual environment. Then, scan any R and python scripts for imports/dependencies and use the template virtual environment to copy packages into the cache or find version numbers to install them fresh (then write them to requirements.txt as is done with renv::snapshot()).

Hope this makes some sense!

[kevinushey]

You're right, the current behavior of renv::use_python() is perhaps not what most would expect -- it uses the requested copy of Python to create its own project-local Python virtual environment.

We should definitely make it possible for renv to use non-local Python environments -- I'll have to think about how to best accomplish this.

[mattwarkentin]

We should definitely make it possible for renv to use non-local Python environments -- I'll have to think about how to best accomplish this.

I actually still like the idea of a project-local python environment, I guess I was just selfishly hoping that {renv} could detect and copy over the necessary python dependencies from an existing python environment, similar to what is already done for the R project-local library.

Or if copying over python packages from an existing environment is a bad idea, perhaps just use an existing python environment to automatically generate the requirements.txt file (and possibly call renv::restore() to install said packages).

[GitHunter0]

I actually still like the idea of a project-local python environment, I guess I was just selfishly hoping that {renv} could detect and copy over the necessary python dependencies from an existing python environment, similar to what is already done for the R project-local library.

Indeed that would be very helpful and avoid all the manual installation.

[mwrowe]

You're right, the current behavior of renv::use_python() is perhaps not what most would expect -- it uses the requested copy of Python to create its own project-local Python virtual environment.

We should definitely make it possible for renv to use non-local Python environments -- I'll have to think about how to best accomplish this.

I would definitely much prefer this solution-- I have a complicated Python virtualenv used by one of my R packages that eats up nearly 1 GB of disk space. I don't want to make a copy of it, or rebuild it, every time I do a new R analysis.

[kevinushey]

FWIW this should now be possible in the development version of renv. You can do something like:

renv::use_python("/path/to/virtualenv/bin/python")

and renv will set both the PATH and RETICULATE_PYTHON so that the requested version of Python is used.