conda shell function injects $CONDA_PREFIX into $PATH causing incorrect behavior in conda run

[chenghlee]

Description

What happened?

The $PATH/%PATH% set by conda run includes directories from the base environment; conda activate, on the other hand, removes base environment components from the PATH. This can be problematic, as conda run and conda activate can have different executables (and on Windows, DLLs) available to the user.

A basic replicating case:

(base) $ conda create -n py39 python=3.9
(base) $ conda run -n py39 python -c 'import os; print(os.environ["PATH"]);'
${CONDA_ROOT}/envs/py39/bin:${CONDA_ROOT}/bin:${CONDA_ROOT}/condabin:${HOME}/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
(base) $ conda activate py39
(py39) $ python -c 'import os; print(os.environ["PATH"]);'
${CONDA_ROOT}/envs/py39/bin:${CONDA_ROOT}/condabin:${HOME}/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

(Note that ${CONDA_ROOT}/bin gets removed from $PATH in the activated environment, but not when using conda run.)

Conda Details

conda info

active environment : base
    active env location : ${CONDA_ROOT}
            shell level : 1
       user config file : ${HOME}/.condarc
 populated config files : ${HOME}/.condarc
          conda version : 4.11.0
    conda-build version : 3.21.7
         python version : 3.8.12.final.0
       virtual packages : __osx=10.16=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : ${CONDA_ROOT}  (writable)
      conda av data dir : ${CONDA_ROOT}/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : ${CONDA_ROOT}/pkgs
                          ${HOME}/.conda/pkgs
       envs directories : ${CONDA_ROOT}/envs
                          ${HOME}/.conda/envs
               platform : osx-64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.8.12 Darwin/20.6.0 OSX/10.16
                UID:GID : 502:20
             netrc file : None
           offline mode : False

conda config

==> ${HOME}/.condarc <==
restore_free_channel: False
conda_build:
  error_overdepending: True
  error_overlinking: True

conda list

# packages in environment at /Users/clee/Applications/miniconda3:
#
# Name                    Version                   Build  Channel
anaconda-client           1.9.0            py38hecd8cb5_0    defaults
attrs                     21.4.0             pyhd3eb1b0_0    defaults
beautifulsoup4            4.10.0             pyh06a4308_0    defaults
brotlipy                  0.7.0           py38h9ed2024_1003    defaults
bzip2                     1.0.8                h1de35cc_0    defaults
ca-certificates           2021.10.26           hecd8cb5_2    defaults
certifi                   2021.10.8        py38hecd8cb5_2    defaults
cffi                      1.15.0           py38hc55c11b_1    defaults
chardet                   4.0.0           py38hecd8cb5_1003    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
clyent                    1.2.2                    py38_1    defaults
conda                     4.11.0           py38hecd8cb5_0    defaults
conda-build               3.21.7           py38hecd8cb5_0    defaults
conda-content-trust       0.1.1              pyhd3eb1b0_0    defaults
conda-package-handling    1.7.3            py38h9ed2024_1    defaults
conda-token               0.3.0              pyhd3eb1b0_0    defaults
coreutils                 8.32                 haf1e3a3_0    defaults
cryptography              36.0.0           py38hf6deb26_0    defaults
filelock                  3.4.2              pyhd3eb1b0_0    defaults
glob2                     0.7                pyhd3eb1b0_0    defaults
icu                       58.2                 h0a44026_3    defaults
idna                      3.3                pyhd3eb1b0_0    defaults
importlib-metadata        4.8.2            py38hecd8cb5_0    defaults
importlib_metadata        4.8.2                hd3eb1b0_0    defaults
ipython_genutils          0.2.0              pyhd3eb1b0_1    defaults
jinja2                    2.11.3             pyhd3eb1b0_0    defaults
jq                        1.6               h9ed2024_1000    defaults
jsonschema                3.2.0              pyhd3eb1b0_2    defaults
jupyter_core              4.9.1            py38hecd8cb5_0    defaults
libarchive                3.4.2                ha0e9c3a_0    defaults
libcxx                    12.0.0               h2f01273_0    defaults
libffi                    3.3                  hb1e8313_2    defaults
libiconv                  1.16                 h1de35cc_0    defaults
liblief                   0.10.1               h0a44026_0    defaults
libxml2                   2.9.12               hcdb78fc_0    defaults
lz4-c                     1.9.3                h23ab428_1    defaults
markupsafe                2.0.1            py38h9ed2024_0    defaults
nbformat                  5.1.3              pyhd3eb1b0_0    defaults
ncurses                   6.3                  hca72f7f_2    defaults
oniguruma                 6.9.7.1              h9ed2024_0    defaults
openssl                   1.1.1m               hca72f7f_0    defaults
packaging                 21.3               pyhd3eb1b0_0    defaults
pip                       21.2.4           py38hecd8cb5_0    defaults
pkginfo                   1.8.2              pyhd3eb1b0_0    defaults
psutil                    5.8.0            py38h9ed2024_1    defaults
py-lief                   0.10.1           py38haf313ee_0    defaults
pycosat                   0.6.3            py38h1de35cc_1    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 21.0.0             pyhd3eb1b0_1    defaults
pyparsing                 3.0.4              pyhd3eb1b0_0    defaults
pyrsistent                0.18.0           py38hca72f7f_0    defaults
pysocks                   1.7.1                    py38_1    defaults
python                    3.8.12               h88f2d9e_0    defaults
python-dateutil           2.8.2              pyhd3eb1b0_0    defaults
python-libarchive-c       2.9                pyhd3eb1b0_1    defaults
python.app                3                py38hca72f7f_0    defaults
pytz                      2021.3             pyhd3eb1b0_0    defaults
pyyaml                    6.0              py38hca72f7f_1    defaults
readline                  8.1.2                hca72f7f_1    defaults
requests                  2.27.1             pyhd3eb1b0_0    defaults
ripgrep                   12.1.1                        0    defaults
ruamel.yaml               0.16.12          py38haf1e3a3_1    defaults
ruamel.yaml.clib          0.2.6            py38hca72f7f_0    defaults
ruamel_yaml               0.15.100         py38h9ed2024_0    defaults
setuptools                58.0.4           py38hecd8cb5_0    defaults
six                       1.16.0             pyhd3eb1b0_0    defaults
soupsieve                 2.3.1              pyhd3eb1b0_0    defaults
sqlite                    3.37.0               h707629a_0    defaults
tk                        8.6.11               h7bc2e8c_0    defaults
tqdm                      4.62.3             pyhd3eb1b0_1    defaults
traitlets                 5.1.1              pyhd3eb1b0_0    defaults
urllib3                   1.26.7             pyhd3eb1b0_0    defaults
wget                      1.20.1               h051b688_0    defaults
wheel                     0.37.1             pyhd3eb1b0_0    defaults
xz                        5.2.5                h1de35cc_0    defaults
yaml                      0.2.5                haf1e3a3_0    defaults
zipp                      3.7.0              pyhd3eb1b0_0    defaults
zlib                      1.2.11               h4dc903c_4    defaults
zstd                      1.5.0                hcb37349_1    defaults

Resolution

It would appear that the __add_sys_prefix_to_path shell function added in conda 4.6.12 is the culprit here.

Duplicate Issues

• Running using conda run inside already activated environment should be the same as running it outside #11305
• conda run -n MY-ENV python FILE.py uses the base interpreter instead of environment interpreter. #9587
• conda run sometimes uses python in the base environment #8450
• Different basic environment activate the same environment but lead to different places #10786
• conda run python in it's own activated environment not using proper python #9571

[jezdez]

@chenghlee do you think this is related to #11072?

[chenghlee]

@jezdez That's my suspicion, but I can't exactly prove it.

[beeankha]

Related to issue #11231

[jezdez]

We've reviewed the work-around in #11257 and decided to remove this from the upcoming 4.12.0 release since we don't have the confidence that it would be an appropriate fix.

We'll need to dig more into why #8528 added the __add_sys_prefix_to_path function to the shell scripts (and related Windows versions). @kenodegard's comment also sheds some light on the subtleties of the problem.

[karrtikr]

Potentially related to #11305

[jezdez]

As mentioned in yesterday's community call we're bumping this up in severity since it seems to affect a lot of users of Visual Studio Code.

[imatiach-msft]

I was able to work around this by calling conda deactivate first, like this (and then passing to subprocess.check_call in my case):

    command = [
        "conda",
        "deactivate",
        "&&",
        "conda",
        "run",
        "-n",
        self.inference_conda_env,
        "python",
        get_wrapmodel_filepath(),
        "--model_path",
        self.model_path,
        "--wrapped_model_path",
        self.wrapped_model_path,
    ]

related issues:
#11305
#2379
#11174