R 3.6.0 depends on which and that's not always present

[kenahoo]

Actual Behavior

SUMMARY: r version 3.6.0 is installed in an environment, but it's not functional. The built-in utils and stats packages can't be loaded, not sure whether there are other problems too.

I created a Docker image using the following Dockerfile:

# -*- dockerfile -*-
FROM centos:latest
RUN yum update -y
RUN yum install -y curl bzip2


# Install miniconda
ARG CONDA_HOME=/opt/miniconda
RUN curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh && \
  bash /tmp/miniconda.sh -b -p $CONDA_HOME && \
  rm /tmp/miniconda.sh && \
  ln -s $CONDA_HOME/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
  echo ". $CONDA_HOME/etc/profile.d/conda.sh" >> ~/.bashrc && \
  echo "conda activate base" >> ~/.bashrc && \
  find $CONDA_HOME/ -follow -type f -name '*.a' -delete && \
  find $CONDA_HOME/ -follow -type f -name '*.js.map' -delete && \
  $CONDA_HOME/bin/conda clean -afy

RUN $CONDA_HOME/bin/conda update -n base -c defaults conda

RUN $CONDA_HOME/bin/conda create -n r-3.6 r=3.6.0

CMD /bin/bash

Built by running the simple command docker build -t conda-r-test -f conda-R-test ..
Everything appears to go fine.

However, the installation of R is broken:

% docker run -it conda-r-test
(base) [root@1e3c39c2bd44 /]# /opt/miniconda/envs/r-3.6/bin/R

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Error: package or namespace load failed for 'utils':
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: error in running command
Error: package or namespace load failed for 'stats':
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: error in running command
During startup - Warning messages:
1: package 'utils' in options("defaultPackages") was not found 
2: package 'stats' in options("defaultPackages") was not found 

The same errors are reported if I first do conda activate r-3.6 and then invoke R.

Expected Behavior

R should be installed correctly.

Note: using the exact same incantations, R 3.4 and 3.5 were installed successfully. Only R 3.6 seems to have the problem.

Steps to Reproduce

Build Docker image using the above.

Anaconda or Miniconda version:

# /opt/miniconda/bin/conda --version
conda 4.7.10

Operating System:

conda info

     active environment : None
       user config file : /root/.condarc
 populated config files : 
          conda version : 4.7.10
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages : 
       base environment : /opt/miniconda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/miniconda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/miniconda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.10 requests/2.21.0 CPython/3.7.3 Linux/4.9.125-linuxkit centos/7.6.1810 glibc/2.17
                UID:GID : 0:0
             netrc file : None
           offline mode : False

conda list --show-channel-urls

# packages in environment at /opt/miniconda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
asn1crypto                0.24.0                   py37_0    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
ca-certificates           2019.5.15                     0    defaults
certifi                   2019.6.16                py37_0    defaults
cffi                      1.12.2           py37h2e261b9_1    defaults
chardet                   3.0.4                    py37_1    defaults
conda                     4.7.10                   py37_0    defaults
conda-package-handling    1.3.11                   py37_0    defaults
cryptography              2.6.1            py37h1ba5d50_0    defaults
idna                      2.8                      py37_0    defaults
libarchive                3.3.3                h5d8350f_5    defaults
libedit                   3.1.20181209         hc058e9b_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc-ng                 8.2.0                hdf63c60_1    defaults
libstdcxx-ng              8.2.0                hdf63c60_1    defaults
libxml2                   2.9.9                hea5a465_1    defaults
lz4-c                     1.8.1.2              h14c3975_0    defaults
lzo                       2.10                 h49e0be7_2    defaults
ncurses                   6.1                  he6710b0_1    defaults
openssl                   1.1.1c               h7b6447c_1    defaults
pip                       19.0.3                   py37_0    defaults
pycosat                   0.6.3            py37h14c3975_0    defaults
pycparser                 2.19                     py37_0    defaults
pyopenssl                 19.0.0                   py37_0    defaults
pysocks                   1.6.8                    py37_0    defaults
python                    3.7.3                h0371630_0    defaults
python-libarchive-c       2.8                     py37_11    defaults
readline                  7.0                  h7b6447c_5    defaults
requests                  2.21.0                   py37_0    defaults
ruamel_yaml               0.15.46          py37h14c3975_0    defaults
setuptools                41.0.0                   py37_0    defaults
six                       1.12.0                   py37_0    defaults
sqlite                    3.27.2               h7b6447c_0    defaults
tk                        8.6.8                hbc83047_0    defaults
tqdm                      4.32.1                     py_0    defaults
urllib3                   1.24.1                   py37_0    defaults
wheel                     0.33.1                   py37_0    defaults
xz                        5.2.4                h14c3975_4    defaults
yaml                      0.1.7                had09818_2    defaults
zlib                      1.2.11               h7b6447c_3    defaults
zstd                      1.3.7                h0b5b093_0    defaults

Thanks.

[kenahoo]

I think I've figured out the problem - R 3.6.0 seems to need a functional which command in the default shell. If I do yum install -y which and then run R again (don't need to re-install it), things seem fixed:

(base) [root@6fabbb0f8e16 /]# yum install -y which
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirrors.usinternet.com
 * extras: mirror.steadfastnet.com
 * updates: mirrors.usinternet.com
Resolving Dependencies
--> Running transaction check
---> Package which.x86_64 0:2.20-7.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================================================================
 Package                                           Arch                                               Version                                                  Repository                                        Size
======================================================================================================================================================================================================================
Installing:
 which                                             x86_64                                             2.20-7.el7                                               base                                              41 k

Transaction Summary
======================================================================================================================================================================================================================
Install  1 Package

Total download size: 41 k
Installed size: 75 k
Downloading packages:
which-2.20-7.el7.x86_64.rpm                                                                                                                                                                    |  41 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : which-2.20-7.el7.x86_64                                                                                                                                                                            1/1 
install-info: No such file or directory for /usr/share/info/which.info.gz
  Verifying  : which-2.20-7.el7.x86_64                                                                                                                                                                            1/1 

Installed:
  which.x86_64 0:2.20-7.el7                                                                                                                                                                                           

Complete!
(base) [root@6fabbb0f8e16 /]# /opt/miniconda/envs/r-3.6/bin/R

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(base)
> 

Is this something that should be set as a formal prerequisite for the R package? Or should users just have to know that which is necessary, and ensure that it's present?

[msarahan]

The only way that conda could have that be a prereq is if there were a conda package for which. I think stripped-down docker images are kind of exceptional here. A lack of which is pretty extreme, in my humble opinion. This dependency is certainly good to know about, but I wonder if that 41 kB was really worth saving in your docker image.

[kenahoo]

I agree that it's silly to be missing which in this Docker image. Quite happy to add yum install -y which in my Dockerfile, I'm not purposefully trying to shave bits off the image.

The base image I'm starting from is centos:latest, though, not some super stripped-down OS variant. It's their decision to leave out which, not mine. It would be nice if there were some way to surface the missing requirement earlier, before runtime, otherwise this is a pretty obscure stumbling block to trip over.

Anyway, up to you how/if you want to handle this.

[msarahan]

Yes, the symptom is especially misleading. I agree that we should find some way to express this dependency better. CentOS stripping this out is nutty, but similar stuff happens with docker images pretty regularly. Thanks for isolating the issue for us.

[Ilhicas]

I don't think nutty would be the term, I'm in the middle of building a distroless R image (quite smaller than the official one) Having this sort of dependencies is "nutty", such as "bash" to run a lot of the dependencies.

[mingwandroid]

We (the Anaconda Distribution, or conda-forge) do not make any additions to the R interpreter to add any (additional) use of the which command to it. If you wish to bring this up with the R core developers then please do so.

One consideration here is that R often compiles software when you install it (particularly on Linux) and to do so it expects common Unix build tools (the Autotools suite in particular which includes shell code, m4 code and Perl code). For this reason, R users on Unix have long been expected to "just know" they need these things.

[moritzschaefer]

@mingwandroid anaconda's R (version 4.0.2) does not run properly run under NixOS, possibly due to this issue (I get the same error, i.e. utils and stats packages can't be loaded). Are you still not considering to patch this in Anaconda (I am also asking since this issue is still Open).

@kenahoo have you raised this with the R dev team?

[unode]

I can confirm what @moritzschaefer said and also that adding which to /usr/bin/which fixes the problem.
The path is hardcoded, simply having which in $PATH is not sufficient.

[aosingh]

This dependency on which needs to be documented. With the error message, the dependency on which is not clear

I am building a slim Docker image for conda. Each dependency requirement has to be installed during image build. The base Linux OS image doesn't come with which by default.

The conda Docker container works fine for Python conda environments without which but not for R environments.

[interpreteruser@811147bc33c4 ~]$ conda activate myrenv                      
[interpreteruser@811147bc33c4 ~]$ R                                          
                                                                             
R version 4.3.3 (2024-02-29) -- "Angel Food Cake"                            
Copyright (C) 2024 The R Foundation for Statistical Computing                
Platform: x86_64-conda-linux-gnu (64-bit)                                    
                                                                             
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

During startup - Warning messages:
1: package 'utils' in options("defaultPackages") was not found 
2: package 'stats' in options("defaultPackages") was not found 
> library(tidyverse)
Error: package or namespace load failed for 'tidyverse':
 .onAttach failed in attachNamespace() for 'tidyverse', details:
  call: NULL
  error: package or namespace load failed for 'ggplot2':
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: error in running command
> library(tidyr)
Error: package or namespace load failed for 'tidyr':
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: error in running command
> library(ggplot2)
Error: package or namespace load failed for 'ggplot2':
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: error in running command
> quit()

Adding yum install which to the Dockerfile fixes the problem.