Isolate conda R from system user library.

[jdblischak]

Checklist

• Used a fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

It's a constant frustration for me that conda R uses the R packages installed in the R user library. This is also a major limitation when trying to convince new conda users about the reproducibility benefits of conda.

This PR attempts to do the following:

• Disable the default setting of R_LIBS_USER by r-base
• Temporarily unsets any custom R_LIBS_USER set by the user
• Tests the isolation during package creation by creating an empty user library that matches the pattern that would be used on that OS

Note to future self: I created the patch by 1) cloning r-source, 2) editing and committing etc/Renviron.in, and 3) running git format-patch HEAD~1.

This issue has previously been discussed in #37.

cc: @mingwandroid @ebolyen @bgruening @isuruf

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[mingwandroid]

Thanks so much for doing this @jdblischak and sorry for the slow review.

Do you (or any of @conda-forge/r) think we should add an env var around all of this as an escape hatch to bring back the old behaviour?

[dpryan79]

An env solution to invoke the old behavior would be welcome, but not a deal breaker (I've lost track of the number of people who have been bitten by libraries in their home directories).

[jdblischak]

Do you (or any of @conda-forge/r) think we should add an env var around all of this as an escape hatch to bring back the old behaviour?

@mingwandroid I think that's fine. However, I think it would only work if the user has explicitly set R_LIBS_USER. The default search location for a local R library is disabled by the patch during the building of the conda package.

A few follow-up questions:

1. Could you share an example of a recipe that modifies it's behavior based on an environment variable that can be set by the user?
2. For consistency, do you plan to implement the same behavior for r-base in defaults?
3. Where should this be documented so that conda R users know about this behavior?

And a general question, what is the purpose of the subsection files: for tests:? I thought it would execute these files, but that only works if they are put in the commands: section. Does this remove this from the final recipe or something like that?

[mingwandroid]

@mingwandroid I think that's fine. However, I think it would only work if the user has explicitly set R_LIBS_USER. The default search location for a local R library is disabled by the patch during the building of the conda package.

Yeah, I propose adding env() checks to your patch. I'll try to schedule some time when I next update default's R to build on top of this PR.

For consistency, do you plan to implement the same behavior for r-base in defaults?

We do not want any difference between R in conda-forge and the R in defaults.

Where should this be documented so that conda R users know about this behavior?

Not sure! I guess conda-forge should document it and we should also document it.

And a general question, what is the purpose of the subsection files: for tests:?

This specifies which files (from the recipe, and not from ${SRC_DIR}, those are specified via source_files:) need to be copied into the test folder / CWD (this is not the test env.) during tests. Packages can be independently tested so we need to know exactly which files are necessary for the tests to work.

[jdblischak]

Yeah, I propose adding env() checks to your patch. I'll try to schedule some time when I next update default's R to build on top of this PR.

@mingwandroid That'd be great. I'm not familiar with env() checks.

We do not want any difference between R in conda-forge and the R in defaults.

100% agree.

Not sure! I guess conda-forge should document it and we should also document it.

Yeah, that is an Issue. Especially for conda-forge since its docs are about the channel, not how to use any given package that it distributes. The Anaconda docs has a section on using R, so that could be a potential options.

https://docs.anaconda.com/anaconda/user-guide/tasks/use-r-language/

This specifies which files (from the recipe, and not from ${SRC_DIR}, those are specified via source_files:) need to be copied into the test folder / CWD (this is not the test env.) during tests. Packages can be independently tested so we need to know exactly which files are necessary for the tests to work.

OK. Makes sense. Thanks for the explanation!

[jdblischak]

@mingwandroid I was just reminded about this PR from a colleague whose code broke because the R packages were being loaded from his user library instead of the conda environment. I think this isolation is very important for conda R users. If the env() checks are documented somewhere, could you please send me a link so that I can try to implement it?

[jdblischak]

I've added support for an environment variable CONDA_KEEP_R_LIBS_USER (name suggestions are welcome). If the user has this defined to anything, then the value of R_LIBS_USER will not be erased when the conda environment is activated.

[jdblischak]

@halldc I am updating activate-r-base.sh and am using the same strategy of checking environment variables that you used in PR #59. Could you please explain the function of the +x that you included in the variable name? I wasn't able to find any documentation on this.

r-base-feedstock/recipe/activate-r-base.sh

Line 5 in 9934a07

if [[ ! -z ${RSTUDIO_WHICH_R+x} ]]; then

[jdblischak]

@msarahan @jjhelmus @katietz Could you please review this PR to r-base? I am attempting to isolate R installed in a conda environment from the user's local R library (which always takes precedence when searching for R packages). Any feedback would be much appreciated.

If you approve of the approach, I will start working on support for Windows. Thanks!

[mingwandroid]

Sounds fine to me. Sorry not a detailed review in the least! Pinging @katietz

[halldc]

@jdblischak This is used to evaluate whether the parameter is set. ${var+x} is a parameter expansion which evaluates to nothing if var is unset, and substitutes the string x otherwise. See this Stackoverflow post for details.

[jdblischak]

${var+x} is a parameter expansion which evaluates to nothing if var is unset, and substitutes the string x otherwise.

@halldc Thanks for the explanation and link. The key for me from that SO answer was that using +x enables the script to "distinguish between a variable that is unset and a variable that is set to the empty string."

[jdblischak]

Just got the email from the R Project: "R 3.6.1 scheduled for July 5". It would be nice if we could get this feature into the conda-forge release of r-base 3.6.1.

[jdblischak]

For any users that find this thread and need a short-term solution, @dpryan79 provided a good solution in conda-forge/r-rjags-feedstock#6 (comment):

.libPaths(R.home("library"))

This will replace your default user library with the system library path (which is where conda R packages are installed). To automate this, you could put it inside of an .Rprofile file inside your project directory (you wouldn't want to put it in your home directory because then this would always ignore your user library, which you may want to use for other projects).

[h-vetinari]

Can this be retried now that 3.6.1 is out?

[katietz]

Sorry, that review took a bit. Change looks sensible to me.
As 3.6.1 is out now, is that patch applying to new version too?
Btw in my opionion we should update soon to 3.6.1 version

[jdblischak]

Can this be retried now that 3.6.1 is out?

@h-vetinari I rebased the PR against R 3.6.1. Could you please test it out?

[jdblischak]

Sorry, that review took a bit. Change looks sensible to me.

@katietz Thanks for the review! Can I interpret this as you being open to the same patch to the r-base recipe in the defaults channel? I'd prefer if the two behaved the same in regards to the user library.

[h-vetinari]

@jdblischak: @h-vetinari I rebased the PR against R 3.6.1. Could you please test it out?

I can build on my personal laptop, but in the environment where I actually need it, I can't pull stuff from github (but we have a proxy for conda-forge)... There's some complicated ways around it, but even then I'd have to wait until conda-forge supports an r=3.6.1 install. That migration is in progress, at least...

[katietz]

Yes. I am openminded to put patch also in r-base default channel for 3.6.1

[dpryan79]

@h-vetinari The migration is effectively finished if you can try things out.

[h-vetinari]

@dpryan79
Thanks for all the rebuilding effort!

In a fresh conda install, doing

conda config --add channels conda-forge \
    && conda update -n base conda \
    && conda update --all \
    && conda install --prune --strict-channel-priority R=3.6

fails with

[...]
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with current_repodata.json, will retry with next repodata source.
Initial quick solve with frozen env failed.  Unfreezing env and trying again.
Solving environment: ...working... failed with current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
Initial quick solve with frozen env failed.  Unfreezing env and trying again.
Solving environment: ...working... failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:
Package tk conflicts for:
python=3.7 -> tk[version='>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0']
Package libffi conflicts for:
python=3.7 -> libffi[version='>=3.2.1,<3.3.0a0|>=3.2.1,<4.0a0']
Package zlib conflicts for:
python=3.7 -> zlib[version='>=1.2.11,<1.3.0a0']
Package libstdcxx-ng conflicts for:
python=3.7 -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0']
Package sqlite conflicts for:
python=3.7 -> sqlite[version='>=3.24.0,<4.0a0|>=3.25.1,<4.0a0|>=3.25.2,<4.0a0|>=3.25.3,<4.0a0|>=3.26.0,<4.0a0|>=3.27.2,<4.0a0|>=3.28.0,<4.0a0']
Package xz conflicts for:
python=3.7 -> xz[version='>=5.2.3,<5.3.0a0|>=5.2.4,<5.3.0a0|>=5.2.4,<6.0a0']
Package libgcc-ng conflicts for:
python=3.7 -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
Package bzip2 conflicts for:
python=3.7 -> bzip2[version='>=1.0.6,<2.0a0']
Package openssl conflicts for:
python=3.7 -> openssl[version='>=1.0.2o,<1.0.3a|>=1.0.2p,<1.0.3a|>=1.1.1a,<1.1.2a|>=1.1.1b,<1.1.2a']
Package readline conflicts for:
python=3.7 -> readline[version='>=7.0,<8.0a0|>=8.0,<9.0a0']
Package ncurses conflicts for:
python=3.7 -> ncurses[version='>=6.1,<6.2.0a0|>=6.1,<7.0a0']
Package pip conflicts for:
python=3.7 -> pip
Note that strict channel priority may have removed packages required for satisfiability.

Removing --strict-channel-priority runs through, but interestingly, it does not pick up r-recommended or r from conda-forge, even though those have been rebuilt already:

  [...]
  pthread-stubs      conda-forge/linux-64::pthread-stubs-0.4-h14c3975_1001
  r                  pkgs/r/linux-64::r-3.6.0-r36_0
  r-base             conda-forge/linux-64::r-base-3.6.1-h6e652e1_3
  r-boot             conda-forge/noarch::r-boot-1.3_23-r36h6115d3f_2
  r-class            conda-forge/linux-64::r-class-7.3_15-r36hcdcec82_1001
  r-cluster          conda-forge/linux-64::r-cluster-2.1.0-r36h9bbef5b_1
  r-codetools        conda-forge/noarch::r-codetools-0.2_16-r36h6115d3f_1001
  r-foreign          conda-forge/linux-64::r-foreign-0.8_71-r36hcdcec82_1003
  r-kernsmooth       conda-forge/linux-64::r-kernsmooth-2.23_15-r36h9bbef5b_1003
  r-lattice          conda-forge/linux-64::r-lattice-0.20_38-r36hcdcec82_1001
  r-mass             conda-forge/linux-64::r-mass-7.3_51.4-r36hcdcec82_1
  r-matrix           conda-forge/linux-64::r-matrix-1.2_17-r36hcdcec82_1
  r-mgcv             conda-forge/linux-64::r-mgcv-1.8_28-r36hcdcec82_1
  r-nlme             conda-forge/linux-64::r-nlme-3.1_140-r36h9bbef5b_1
  r-nnet             conda-forge/linux-64::r-nnet-7.3_12-r36hcdcec82_1003
  r-recommended      pkgs/r/linux-64::r-recommended-3.6.0-r36_0
  r-rpart            conda-forge/linux-64::r-rpart-4.1_15-r36hcdcec82_1
  r-spatial          conda-forge/linux-64::r-spatial-7.3_11-r36hcdcec82_1003
  r-survival         conda-forge/linux-64::r-survival-2.44_1.1-r36hcdcec82_1
  sed                conda-forge/linux-64::sed-4.7-h1bed415_1000
  [...]

[h-vetinari]

@dpryan79
I should have been more explicit in my answer: I can build this PR on linux, but I couldn't test it beyond that* until I can get a standard R-install from conda-forge. I opened conda-forge/r-feedstock#12 to track this.

*Or so I thought; actually, all the packages that make up r can already be installed directly. How did you envision that I test this PR?

[ebolyen]

I don't understand this. How does conda deviate from upstream?

This section of PEP 405 defines the behavior of venv the built-in environment manager w.r.t. user-local packages.

EDIT: there was a better subsection to link to

[ebolyen]

I acknowledge that as both a package manager and environment manager, it may be impossible for conda to perfectly handle both of these situations as there isn't a strong distinction between a "root" environment and a normal environment. So it is difficult to really disentangle these competing situations.

[dpryan79]

I've needed to use both cases for R. For example, I configured the rstudio server at my old institute to use various versions of conda-installed r-base. This worked quite well and was much more convenient than compiling and installing various versions manually. Users then used the normal methods to install packages in their home directories (the default locations).

At the same time our pipelines used r-base in preconfigured envs and then just had the following in their Rscripts:

.libPaths(R.home("library"))

Then users could run scripts and everything in their home directories would be ignored. So you don't need an environment variable, if you want only-conda libraries loaded just add that one line.

[jdblischak]

That's not necessarily true. People can use a conda installation as system if they want. (Maybe in /opt/conda)

@isuruf They could use sudo to install conda as system, and thus prevent themselves and others from installing packages there. But I would argue that is the less common case. In the conda installation instructions, it states:

We recommend you install Anaconda for the local user, which does not require administrator permissions and is the most robust type of installation. You can also install Anaconda system wide, which does require administrator permissions.

Thus I'd prefer the default behavior to work well with the recommended installation guidelines, and then provide workarounds for the non-standard system installation of conda (especially since a systems administrator installing R for a group of users should be comfortable with the extra steps).

The script can give specific instructions on how to fix this.

@isuruf Do you have suggestions for the workaround?

Here is a potential workaround I came up with. Users could put the following in their .Rprofile. If R is installed with conda, this will configure the library paths to only point to the conda R package directory:

local({
  conda <- Sys.getenv("CONDA_PREFIX", unset = NA)
  if (!is.na(conda) && startsWith(R.home(), conda)) {
    .libPaths(R.home("library"))
  }
})

Then users could run scripts and everything in their home directories would be ignored. So you don't need an environment variable, if you want only-conda libraries loaded just add that one line.

@dpryan79 While this works, I don't like that this solution requires R to be installed with conda. I don't think an R script should be concerned with implementation details like whether or not R was installed via conda or not.

[isuruf]

Here is a potential workaround I came up with.

That sounds like a good idea and maybe a conda-forge package just for that might fix this. We can warn in the activate script here and ask the user to install that package.

[jamesmyatt]

The following workaround doesn't work on Windows, because R.home uses '/' separators and Sys.getenv uses '\\' separators. Also I think you only need .libPaths("").

local({
  conda <- Sys.getenv("CONDA_PREFIX", unset = NA)
  if (!is.na(conda) && startsWith(R.home(), conda)) {
    .libPaths(R.home("library"))
  }
})

[jdblischak]

The following workaround doesn't work on Windows, because R.home uses '/' separators and Sys.getenv uses '\' separators.

@jamesmyatt Thanks for reporting this shortcoming with the proposed solution. Below is an updated version that should work on both Windows and Unix-alike operating systems. I tested it on Windows 10.

local({
  conda <- Sys.getenv("CONDA_PREFIX", unset = NA)
  rhome <- normalizePath(R.home())
  if (!is.na(conda) && startsWith(rhome, conda)) {
    .libPaths(R.home("library"))
  }
})

Also I think you only need .libPaths("").

You're correct that .libPaths("") and .libPaths(R.home("library")) produce the same result, so you can use whichever you like. I prefer the latter since it is more explicit about the change being made. The former requires knowing that .libPaths() returns unique(c(new, .Library.site, .Library)).

[jdblischak]

A quick update on this PR: As seen above, there is a debate about whether this "isolation" feature should be implemented. The main issue is that because conda is used in so many different ways, it is difficult to find the right behavior that will be intuitive for all of the use cases.

I've abandoned this effort for now because the situation is actually fixed well enough for me on Linux. I don't know why or how the change was made, but now the conda recipe changes the value of the "platform" variable when installing R on Linux.

# System R installed from CRAN via apt-get on Ubuntu
> R.version$platform
[1] "x86_64-pc-linux-gnu"

# R installed from conda-forge
> R.version$platform
[1] "x86_64-conda_cos6-linux-gnu"

Because the default path to the user library uses this platform variable, when I activate a conda environment in Ubuntu, it ignores my user packages. Thus it's properly isolated.

Unfortunately conda R is still not isolated in the following situations:

1. The user manually sets a custom path to their R user library via R_LIBS_USER.

2. On Windows. This is because the platform variable is the same. It would be nice to insert "conda" into the platform like in the Linux case, but I don't know how to do this. This is already how it works on Linux, so it would be more consistent if Windows did the same.

   # System R installed from CRAN on Windows
   > R.version$platform
   [1] "x86_64-w64-mingw32"
   
   # R installed from conda-forge
   > R.version$platform
   [1] "x86_64-w64-mingw32"
   

My macbook died, so I don't know what the current behavior is on macOS. I'd appreciate it if someone could run R.version$platform with the system and conda versions of R to see if these are the same or different on macOS.

[mfansler]

Might be worth mentioning that the conda-ecosystem-user-package-isolation package provides activation scripts for isolating Python and R from user-level packages. For users that want this as default behavior, they could add it to their create_default_packages.

[jdblischak]

Might be worth mentioning that the conda-ecosystem-user-package-isolation package provides activation scripts for isolating Python and R from user-level packages. For users that want this as default behavior, they could add it to their create_default_packages.

@mfansler Yes! That is very relevant. It's a shame that it's been around for over 3 years, and I had no idea about it. Unfortunately I assume the only way to really learn about it is through word of mouth. Thanks for sharing!

To clarify though, this solves the problem in the situation where a user has explicitly set R_LIBS_USER. However, it won't help in the existing situation on Windows (and likely macOS) where the default user library is defined by R itself (which is addressed by this PR). If a user has any packages installed in the default directory, conda-installed R will still search there first. The only reason this is no longer a problem on Linux is, as I mentioned above, it somehow inserts "conda" into the platform name.

Lastly, for the benefits of others that like me were not aware of create_default_packages, it is documented here.

[jdblischak]

My macbook died, so I don't know what the current behavior is on macOS. I'd appreciate it if someone could run R.version$platform with the system and conda versions of R to see if these are the same or different on macOS.

I currently have access to a macOS (Catalina 10.15.7) machine through work. I checked the platform, and unfortunately "conda" is not inserted into the name. However, you could get lucky if your version of macOS differs from the version in which the r-base binary was built. In my case, the user-installed packages wouldn't conflict, but they very easily could on someone else's machine.

# homebrew-installed R
# R 4.1.3
> R.version$platform
[1] "x86_64-apple-darwin19.6.0"

# conda-installed R
# r-base==4.1.2=h2b051ba_0
> R.version$platform
[1] "x86_64-apple-darwin13.4.0"

[jdblischak]

A few updates relevant to this PR:

• In the upcoming release of R 4.2.0, the default user library on Windows is being changed to the Local Application Data directory. My test scripts assume the user directory is in Documents/

The default personal library on Windows, folder ‘R\win-library\x.y’ where ‘x.y’ stands for R release ‘x.y.z’, is now a subdirectory of Local Application Data directory (usually a hidden directory ‘C:\Users\username\AppData\Local’). Use shell.exec(.libPaths()[1]) from R to open the personal library in Explorer when it is first in the list (PR#17842).

https://github.com/wch/r-source/blob/tags/R-4-2-0/doc/NEWS.Rd#L324
wch/r-source@9196047

• The /etc/Renviron.in file has changed since my original PR against R 4.0.0. My patch won't even work any more

wch/r-source@tags/R-4-0-0...tags/R-4-2-0#diff-1cbaac4768fd110525ba9086cb7a684aaf2c6555389c5446c913effbfec90c85

I'm not sure what to do with this PR. On the one hand, I'm aware at this point that it is going on almost 4 years, and has zero chance of being merged. Thus I'm not planning on updating it. On the other hand, I think it serves as very useful documentation. Users of conda-installed R are typically surprised when they find out that their conda envrionments are using their locally installed packages (and are thus not at all isolated or reproducible). I know I was. Furthermore, this PR makes it clear that it's not enough to simply temporarily clear R_LIBS_USER (this is obviously a great start, and is now possible with conda-ecosystem-user-package-isolation-feedstock). You also have to prevent R from looking for packages in the default user location. Otherwise, any time a user installs the same R version with conda as they already have installed on their machine, conda-installed R will use the local user packages instead of the conda binaries.

[roaldarbol]

Here 4 years after the last comment, 8 years after this PR was opened, I have been tripped up by this too many times to count now. Could there be any interest in revisiting this? Possibly as a new PR, but maybe revisiting this discussion?

[isuruf]

See #65 (comment)

[roaldarbol]

I'm well aware of conda-ecosystem-user-package-isolation, but it falls short on MacOS:

It sets R_LIBS_USER="-" to suppress custom user-defined R library paths. The problem is that on MacOS, R's installer adds a platform-specific default library path (~/Library/R/x.y/library) through a separate mechanism in R's startup code that is completely independent of R_LIBS_USER. This is a MacOS-only behaviour; on Linux, R_LIBS_USER is the only personal library path, so the package works perfectly there.

And I think fundamentally it makes sense that a conda package operates with what is provided inside an environment - not what's outside of it.

[isuruf]

We can introduce an environment variable to disable that here and set that variable in conda-ecosystem-user-package-isolation

[roaldarbol]

I have made an isolated r-base build that can be installed from the https://prefix.dev/isolated-r channel.

The recipe is really a modified branch of the this feedstock (https://github.com/roaldarbol/r-base-feedstock), so changes made here will make it across to this build easily.

[jdblischak]

And I think fundamentally it makes sense that a conda package operates with what is provided inside an environment - not what's outside of it.

💯

I have made an isolated r-base build that can be installed from the https://prefix.dev/isolated-r channel.

Nice! Thanks for updating my patch for the latest version of R.

From a quick skim, my only concern is that you modified my test scripts. It's been 8 years now, so things might have changed, but I recall that the problem of the default user library path would only surface when it already existed. Thus this will pass in CI, but on an R user's machine where they have the default user library path, then local packages could leak in again (if not properly isolated). In other words, I suspect that test-isolation.R would pass even without the patch.

[jdblischak]

We can introduce an environment variable to disable that here and set that variable in conda-ecosystem-user-package-isolation

I'm not following how this would work. The default user library path is defined when R is built in this feedstock. How could we introduce an environment variable that would make this build-time configuration option be influenced by a conda package installed at run-time?

[ebolyen]

I saw some activity on this thread and wanted to mention that we recently had a report of a new flag R_HOME which is likely to be of interest to those who are also monitoring this.

We're still using our own activate/deactivate hooks (they got a bit better recently).

Cheers from the past and likely future.