Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conda 23.7.3 start-up time extremely slow compared to 23.5.2 #13067

Closed
asalaria-cisco opened this issue Sep 6, 2023 · 9 comments · Fixed by #13082 or #13071
Closed

Conda 23.7.3 start-up time extremely slow compared to 23.5.2 #13067

asalaria-cisco opened this issue Sep 6, 2023 · 9 comments · Fixed by #13082 or #13071
Assignees
@dholth
Labels
duplicate indicate issues/PRs that are duplicates of another in-progress issue is actively being worked on severity::2 critical; broken functionality with an unacceptably complex workaround source::community catch-all for issues filed by community members tag::performance related to degraded performance type::bug describes erroneous operation, use severity::* to classify the type

Comments

@asalaria-cisco
Copy link

asalaria-cisco commented Sep 6, 2023 •
edited by kenodegard

What happened?

Today I did a fresh install of conda. This installed conda 23.5.2. Conda start-up time is excellent:

$ time conda env list
# conda environments:
#
base                  *  /home/XXX/miniconda3


real    0m0.298s
user    0m0.268s
sys     0m0.030s

However, when I upgrade conda, it installs version 23.7.3 and the start-up time shoots up to more than 45 seconds!

$ conda upgrade conda
...
$ time conda env list
# conda environments:
#
base                  *  /home/XXX/miniconda3


real    0m46.997s
user    0m0.544s
sys     0m2.865s

This is not limited to conda env, any other invocation of conda has this 45 seconds start-up overhead!

Conda Details

conda info 
active environment : base
    active env location : /home/xxx/miniconda3
            shell level : 1
       user config file : /home/xxx/.condarc
 populated config files : /home/xxx/.condarc
          conda version : 23.7.3
    conda-build version : not installed
         python version : 3.11.4.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.0=0
                          __glibc=2.35=0
                          __linux=5.15.90.1=0
                          __unix=0=0
       base environment : /home/xxx/miniconda3  (writable)
      conda av data dir : /home/xxx/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/xxx/miniconda3/pkgs
                          /home/xxx/.conda/pkgs
       envs directories : /home/xxx/miniconda3/envs
                          /home/xxx/.conda/envs
               platform : linux-64
             user-agent : conda/23.7.3 requests/2.31.0 CPython/3.11.4 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.3 glibc/2.35 solver/libmamba conda-libmamba-solver/23.5.0 libmambapy/1.4.1
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
conda config 
==> /home/xxx/.condarc <==
channels:
  - defaults
solver: libmamba

==> cmd_line <==
debug: False
json: False
conda list 
# packages in environment at /home/xxx/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
boltons                   23.0.0          py311h06a4308_0    defaults
brotlipy                  0.7.0           py311h5eee18b_1002    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
c-ares                    1.19.1               h5eee18b_0    defaults
ca-certificates           2023.05.30           h06a4308_0    defaults
certifi                   2023.7.22       py311h06a4308_0    defaults
cffi                      1.15.1          py311h5eee18b_3    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
conda                     23.7.3          py311h06a4308_0    defaults
conda-content-trust       0.1.3           py311h06a4308_0    defaults
conda-libmamba-solver     23.5.0          py311h06a4308_0    defaults
conda-package-handling    2.2.0           py311h06a4308_0    defaults
conda-package-streaming   0.9.0           py311h06a4308_0    defaults
cryptography              39.0.1          py311h9ce1e76_2    defaults
fmt                       9.1.0                hdb19cb5_0    defaults
icu                       73.1                 h6a678d5_0    defaults
idna                      3.4             py311h06a4308_0    defaults
jsonpatch                 1.32               pyhd3eb1b0_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
krb5                      1.20.1               h143b758_1    defaults
ld_impl_linux-64          2.38                 h1181459_1    defaults
libarchive                3.6.2                h6ac8c49_2    defaults
libcurl                   8.2.1                h251f7ec_0    defaults
libedit                   3.1.20221030         h5eee18b_0    defaults
libev                     4.33                 h7f8727e_1    defaults
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgomp                   11.2.0               h1234567_1    defaults
libmamba                  1.4.1                h2dafd23_1    defaults
libmambapy                1.4.1           py311h2dafd23_1    defaults
libnghttp2                1.52.0               h2d74bed_1    defaults
libsolv                   0.7.22               he621ea3_0    defaults
libssh2                   1.10.0               hdbd6064_2    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
libuuid                   1.41.5               h5eee18b_0    defaults
libxml2                   2.10.4               hf1b16e4_1    defaults
lz4-c                     1.9.4                h6a678d5_0    defaults
ncurses                   6.4                  h6a678d5_0    defaults
openssl                   3.0.10               h7f8727e_2    defaults
packaging                 23.1            py311h06a4308_0    defaults
pcre2                     10.37                he7ceb23_1    defaults
pip                       23.1.2          py311h06a4308_0    defaults
pluggy                    1.0.0           py311h06a4308_1    defaults
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.4           py311h5eee18b_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 23.2.0          py311h06a4308_0    defaults
pysocks                   1.7.1           py311h06a4308_0    defaults
python                    3.11.4               h955ad1f_0    defaults
readline                  8.2                  h5eee18b_0    defaults
reproc                    14.2.4               h295c915_1    defaults
reproc-cpp                14.2.4               h295c915_1    defaults
requests                  2.31.0          py311h06a4308_0    defaults
ruamel.yaml               0.17.21         py311h5eee18b_0    defaults
setuptools                67.8.0          py311h06a4308_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.41.2               h5eee18b_0    defaults
tk                        8.6.12               h1ccaba5_0    defaults
toolz                     0.12.0          py311h06a4308_0    defaults
tqdm                      4.65.0          py311h92b7b1e_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
urllib3                   1.26.16         py311h06a4308_0    defaults
wheel                     0.38.4          py311h06a4308_0    defaults
xz                        5.4.2                h5eee18b_0    defaults
yaml-cpp                  0.7.0                h295c915_1    defaults
zlib                      1.2.13               h5eee18b_0    defaults
zstandard                 0.19.0          py311h5eee18b_0    defaults
zstd                      1.5.5                hc292b87_0    defaults
@asalaria-cisco asalaria-cisco added the type::bug describes erroneous operation, use severity::* to classify the type label Sep 6, 2023
@kenodegard kenodegard added source::community catch-all for issues filed by community members severity::2 critical; broken functionality with an unacceptably complex workaround tag::performance related to degraded performance labels Sep 6, 2023
@kenodegard
Copy link
Contributor

@asalaria-cisco interesting, a few followup questions:

  1. Do you see a similar slowdown between 23.7.2 and 23.7.3?
  2. Do you happen to have a lot of paths defined in your $PATH and/or are any of those paths network locations?

@kenodegard kenodegard added the pending::feedback indicates we are waiting on feedback from the user label Sep 6, 2023
@aaravind100
Copy link

@kenodegard i just noticed this happening to me too on 23.7.3. Have around 22 entries in $PATH, this is in wsl (Ubuntu 22.04.3 LTS) and 7 are mounted paths from windows.

  • on 23.7.3
$ time conda --version
conda 23.7.3

real    0m17.788s
user    0m0.290s
sys     0m1.825s
  • after downgrading to 23.7.2
$ time conda --version
conda 23.7.2

real    0m0.154s
user    0m0.156s
sys     0m0.011s
conda info 
     active environment : None
            shell level : 0
       user config file : /home/aravia1/.condarc
 populated config files : /home/aravia1/.condarc
          conda version : 23.7.3
    conda-build version : not installed
         python version : 3.11.4.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.0=0
                          __glibc=2.35=0
                          __linux=5.15.90.1=0
                          __unix=0=0
       base environment : /home/aravia1/miniconda3  (writable)
      conda av data dir : /home/aravia1/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/aravia1/miniconda3/pkgs
                          /home/aravia1/.conda/pkgs
       envs directories : /home/aravia1/miniconda3/envs
                          /home/aravia1/.conda/envs
               platform : linux-64
             user-agent : conda/23.7.3 requests/2.31.0 CPython/3.11.4 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.3 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
conda config

NA

conda list 
# packages in environment at /home/aravia1/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                       1_gnu
boltons                   23.0.0          py311h06a4308_0
brotlipy                  0.7.0           py311h5eee18b_1002
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2023.08.22           h06a4308_0
certifi                   2023.7.22       py311h06a4308_0
cffi                      1.15.1          py311h5eee18b_3
charset-normalizer        2.0.4              pyhd3eb1b0_0
conda                     23.7.3          py311h06a4308_0
conda-content-trust       0.2.0           py311h06a4308_0
conda-package-handling    2.2.0           py311h06a4308_0
conda-package-streaming   0.9.0           py311h06a4308_0
cryptography              41.0.2          py311h22a60cf_0
idna                      3.4             py311h06a4308_0
jsonpatch                 1.32               pyhd3eb1b0_0
jsonpointer               2.1                pyhd3eb1b0_0
ld_impl_linux-64          2.38                 h1181459_1
libffi                    3.4.4                h6a678d5_0
libgcc-ng                 11.2.0               h1234567_1
libgomp                   11.2.0               h1234567_1
libstdcxx-ng              11.2.0               h1234567_1
libuuid                   1.41.5               h5eee18b_0
ncurses                   6.4                  h6a678d5_0
openssl                   3.0.10               h7f8727e_2
packaging                 23.1            py311h06a4308_0
pip                       23.0.1          py311h06a4308_0
pluggy                    1.0.0           py311h06a4308_1
pycosat                   0.6.4           py311h5eee18b_0
pycparser                 2.21               pyhd3eb1b0_0
pyopenssl                 23.2.0          py311h06a4308_0
pysocks                   1.7.1           py311h06a4308_0
python                    3.11.4               h955ad1f_0
readline                  8.2                  h5eee18b_0
requests                  2.31.0          py311h06a4308_0
ruamel.yaml               0.17.21         py311h5eee18b_0
setuptools                67.8.0          py311h06a4308_0
sqlite                    3.41.2               h5eee18b_0
tk                        8.6.12               h1ccaba5_0
toolz                     0.12.0          py311h06a4308_0
tqdm                      4.65.0          py311h92b7b1e_0
tzdata                    2023c                h04d1e81_0
urllib3                   1.26.16         py311h06a4308_0
wheel                     0.38.4          py311h06a4308_0
xz                        5.4.2                h5eee18b_0
zlib                      1.2.13               h5eee18b_0
zstandard                 0.19.0          py311h5eee18b_0

@conda-bot conda-bot added pending::support indicates user is waiting on support from triage engineers and removed pending::feedback indicates we are waiting on feedback from the user labels Sep 7, 2023
@aaravind100
Copy link

I did some further testing with two machines, native linux, native windows and wsl. For me, It seem to run slow only in wsl when the $PATH contains the windows mounted paths.

$ echo $PATH
/home/aaravind/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/system32/wbem:/mnt/c/windows/system32/windowspowershell/v1.0/:/mnt/c/windows/system32/openssh/:/mnt/c/program files/git/cmd:/mnt/c/program files/dotnet/:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/ajith/miniconda3:/mnt/c/Users/ajith/miniconda3/Library/mingw-w64/bin:/mnt/c/Users/ajith/miniconda3/Library/usr/bin:/mnt/c/Users/ajith/miniconda3/Library/bin:/mnt/c/Users/ajith/miniconda3/Scripts:/mnt/c/Users/ajith/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/ajith/AppData/Local/Programs/Microsoft VS Code/bin

$ TEMP_PATH=$PATH

$ time conda --version  # this runs slow
conda 23.7.3

real    0m8.785s
user    0m0.150s
sys     0m1.021s

$ PATH=/home/aaravind/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib

$ time conda --version  # this runs fast after removing the windows mounted path
conda 23.7.3

real    0m0.152s
user    0m0.151s
sys     0m0.000s

$ PATH=$TEMP_PATH

$ time conda --version  # adding the windows mounted path makes it slow again
conda 23.7.3

real    0m9.109s
user    0m0.181s
sys     0m1.020s

@kenodegard
Copy link
Contributor

@aaravind100 thanks for running these tests, this confirms my suspicion that the recent change #13006 caused this slowdown

@kenodegard kenodegard added in-progress issue is actively being worked on and removed pending::support indicates user is waiting on support from triage engineers labels Sep 7, 2023
@jezdez jezdez mentioned this issue Sep 7, 2023
Closed
3 tasks
@dholth dholth mentioned this issue Sep 7, 2023
Merged
3 tasks
@jdblischak
Copy link

I am also experiencing the slow conda start-up time in WSL Ubuntu. I'm happy to help test out any release candidates that address this issue. Thanks for your quick response.

time conda --version
## conda 23.7.3
## 
## real    0m33.811s
## user    0m0.741s
## sys     0m3.465s

mamba install conda=23.7.2

time conda --version
## conda 23.7.2
## 
## real    0m0.319s
## user    0m0.269s
## sys     0m0.040s

@dholth
Copy link
Contributor

dholth commented Sep 8, 2023 •
edited

You can try the canary build off conda's main branch conda install -c conda-canary/label/dev conda. The canary build is available for osx-64 win-64 and linux-64 platforms. It's automatically built when the main branch changes.

@jdblischak
Copy link

Thanks @dholth! I confirmed that the canary build has a quick startup time on WSL:

conda install -c conda-canary/label/dev conda
time conda --version
## conda 23.7.4.dev62
## 
## real    0m0.589s
## user    0m0.231s
## sys     0m0.077s

However, I see that your PR #13070 from yesterday hasn't been merged into main yet. I assume this means the update in PR #13035 is responsible for the improvement

@dholth
Copy link
Contributor

dholth commented Sep 8, 2023

That's correct. Yesterday's PR is a tiny improvement on the existing fix.

@kenodegard kenodegard added the duplicate indicate issues/PRs that are duplicates of another label Sep 8, 2023
@kenodegard kenodegard mentioned this issue Sep 11, 2023
Merged
3 tasks
This was linked to pull requests Sep 11, 2023
23.7.x find commands regression picked from main #13082
Merged
use scandir() in find_commands.py to avoid stat() calls against 23.7.x branch #13071
Merged
@kenodegard
Copy link
Contributor

conda 23.7.4 has been released to defaults and corrects this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dholth dholth

Labels
duplicate indicate issues/PRs that are duplicates of another in-progress issue is actively being worked on severity::2 critical; broken functionality with an unacceptably complex workaround source::community catch-all for issues filed by community members tag::performance related to degraded performance type::bug describes erroneous operation, use severity::* to classify the type
Projects
🧭 Planning
Archived in project
Milestone
No milestone
6 participants
@dholth @jdblischak @kenodegard @conda-bot @aaravind100 @asalaria-cisco