Pip install logs not shown with verbose option. #10556
Comments
|
Some notes: I think the "hanging" is because This SO post illustrates how to read the output from the subprocess and stream it to the conda install stdout. |
|
Can confirm, faced the same issue (Azure ML instances). |
|
I am also experiencing this issue. |
|
I am also experiencing this issue, trying to debug what is actually going wrong with my pip installs that takes this long. Any progress on this? |
|
Dirty workaround to see pip logs in real time. Unfortunately this shows verbose pip logs and not only pip stdout. export PIP_LOG="/tmp/pip_log.txt"
touch ${PIP_LOG}
tail -f ${PIP_LOG} &
conda env create --prefix ${CONDA_ENV_PATH} -f "${TMP_ENV_DIR}/train/conda_dependencies.yml"
killall tail
rm ${PIP_LOG}Horrendous oneliner for dockerfile: RUN export PIP_LOG="/tmp/pip_log.txt" && touch ${PIP_LOG} && tail -f ${PIP_LOG} & conda env create --prefix ${CONDA_ENV_PATH} -f "${TMP_ENV_DIR}/train/conda_dependencies.yml" && killall tail && rm ${PIP_LOG} |
|
Also hitting this issue on Azure ML model deploy. |
|
Still running into this issue. Conda version 4.12.0, hangs forever on "Installing pip dependencies". Running macOS Monterey on Intel Macbook pro. |
|
Same on Windows 10 with version 4.13.0. |
Looks wonderfu! But no cigar for me. Just some details in case someone else is still not seeing output: Somehow this does not work for me when using Mamba in Docker on a Mac: silent as ever, until done and then showing the logging after the fact, after first printing " I did not investigate a lot yet; the clipping may even be related to BuildKit for what I've read. Exact command that does not print logging until done |
|
@avbentem I tested with your small example and it works for me on linux (docker 19.03.05). Replaced mamba with conda in your example. logs when building on linuxlpasselin@appliedai-ws1:~/tmp$ ls -lahtr
total 16K
-rw-rw-r-- 1 lpasselin lpasselin 234 Sep 7 13:44 conda_environment.yml
-rw-rw-r-- 1 lpasselin lpasselin 275 Sep 7 13:44 Dockerfile
drwxr-xr-x 18 lpasselin lpasselin 4.0K Sep 7 13:44 ..
drwxrwxr-x 2 lpasselin lpasselin 4.0K Sep 7 13:44 .
lpasselin@appliedai-ws1:~/tmp$ docker build . --no-cache
Sending build context to Docker daemon 3.072kB
Step 1/3 : FROM continuumio/miniconda3:latest
---> ce7d119281a1
Step 2/3 : COPY conda_environment.yml conda_environment.yml
---> 2411d2b767c4
Step 3/3 : RUN export PIP_LOG="/tmp/pip_log.txt" && touch ${PIP_LOG} && tail -f ${PIP_LOG} & conda env create -n myenv --file conda_environment.yml && conda clean --all && killall tail && rm ${PIP_LOG}
---> Running in 37e573cd9e1c
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
==> WARNING: A newer version of conda exists. <==
current version: 4.12.0
latest version: 4.14.0
Please update conda by running
$ conda update -n base -c defaults conda
Downloading and Extracting Packages
libgomp-11.2.0 | 560 KB | ########## | 100%
_openmp_mutex-5.1 | 20 KB | ########## | 100%
ld_impl_linux-64-2.3 | 732 KB | ########## | 100%
readline-8.1.2 | 423 KB | ########## | 100%
wheel-0.37.1 | 31 KB | ########## | 100%
libstdcxx-ng-11.2.0 | 6.1 MB | ########## | 100%
certifi-2022.6.15 | 156 KB | ########## | 100%
xz-5.2.5 | 389 KB | ########## | 100%
ncurses-6.3 | 1.1 MB | ########## | 100%
sqlite-3.39.2 | 1.5 MB | ########## | 100%
ca-certificates-2022 | 131 KB | ########## | 100%
tk-8.6.12 | 3.3 MB | ########## | 100%
libffi-3.3 | 54 KB | ########## | 100%
_libgcc_mutex-0.1 | 3 KB | ########## | 100%
libgcc-ng-11.2.0 | 8.5 MB | ########## | 100%
openssl-1.1.1q | 3.8 MB | ########## | 100%
zlib-1.2.12 | 130 KB | ########## | 100%
setuptools-61.2.0 | 1.3 MB | ########## | 100%
pip-21.1.2 | 2.0 MB | ########## | 100%
python-3.8.13 | 22.7 MB | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/conda/envs/myenv/bin/python', '-m', 'pip', 'install', '-U', '-r', '/condaenv.qcjrhpho.requirements.txt']
Pip subprocess output:
Collecting seaborn
Downloading seaborn-0.12.0-py3-none-any.whl (285 kB)
Collecting numpy>=1.17
Downloading numpy-1.23.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Collecting matplotlib>=3.1
Downloading matplotlib-3.5.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.3 MB)
Collecting pandas>=0.25
Downloading pandas-1.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
Collecting packaging>=20.0
Downloading packaging-21.3-py3-none-any.whl (40 kB)
Collecting fonttools>=4.22.0
Downloading fonttools-4.37.1-py3-none-any.whl (957 kB)
Collecting pyparsing>=2.2.1
Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting python-dateutil>=2.7
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting cycler>=0.10
Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting pillow>=6.2.0
Downloading Pillow-9.2.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.2 MB)
Collecting pytz>=2020.1
Downloading pytz-2022.2.1-py2.py3-none-any.whl (500 kB)
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pyparsing, pytz, python-dateutil, pillow, packaging, numpy, kiwisolver, fonttools, cycler, pandas, matplotlib, seaborn
Successfully installed cycler-0.11.0 fonttools-4.37.1 kiwisolver-1.4.4 matplotlib-3.5.3 numpy-1.23.2 packaging-21.3 pandas-1.4.4 pillow-9.2.0 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.2.1 seaborn-0.12.0 six-1.16.0
done
#
# To activate this environment, use
#
# $ conda activate myenv
#
# To deactivate an active environment, use
#
# $ conda deactivate
Cache location: /opt/conda/pkgs
Will remove the following tarballs:
/opt/conda/pkgs
---------------
python-3.8.13-h12debd9_0.tar.bz2 22.7 MB
openssl-1.1.1q-h7f8727e_0.tar.bz2 3.8 MB
xz-5.2.5-h7f8727e_1.tar.bz2 389 KB
certifi-2022.6.15-py38h06a4308_0.tar.bz2 156 KB
setuptools-61.2.0-py38h06a4308_0.tar.bz2 1.3 MB
_openmp_mutex-5.1-1_gnu.tar.bz2 20 KB
ca-certificates-2022.07.19-h06a4308_0.tar.bz2 131 KB
libstdcxx-ng-11.2.0-h1234567_1.tar.bz2 6.1 MB
ncurses-6.3-h5eee18b_3.tar.bz2 1.1 MB
libgcc-ng-11.2.0-h1234567_1.tar.bz2 8.5 MB
zlib-1.2.12-h7f8727e_2.tar.bz2 130 KB
wheel-0.37.1-pyhd3eb1b0_0.tar.bz2 31 KB
readline-8.1.2-h7f8727e_1.tar.bz2 423 KB
_libgcc_mutex-0.1-main.tar.bz2 3 KB
tk-8.6.12-h1ccaba5_0.tar.bz2 3.3 MB
sqlite-3.39.2-h5082296_0.tar.bz2 1.5 MB
pip-21.1.2-py38h06a4308_0.tar.bz2 2.0 MB
ld_impl_linux-64-2.38-h1181459_1.tar.bz2 732 KB
libgomp-11.2.0-h1234567_1.tar.bz2 560 KB
libffi-3.3-he6710b0_2.tar.bz2 54 KB
---------------------------------------------------
Total: 52.9 MB
Proceed ([y]/n)?
Removed python-3.8.13-h12debd9_0.tar.bz2
Removed openssl-1.1.1q-h7f8727e_0.tar.bz2
Removed xz-5.2.5-h7f8727e_1.tar.bz2
Removed certifi-2022.6.15-py38h06a4308_0.tar.bz2
Removed setuptools-61.2.0-py38h06a4308_0.tar.bz2
Removed _openmp_mutex-5.1-1_gnu.tar.bz2
Removed ca-certificates-2022.07.19-h06a4308_0.tar.bz2
Removed libstdcxx-ng-11.2.0-h1234567_1.tar.bz2
Removed ncurses-6.3-h5eee18b_3.tar.bz2
Removed libgcc-ng-11.2.0-h1234567_1.tar.bz2
Removed zlib-1.2.12-h7f8727e_2.tar.bz2
Removed wheel-0.37.1-pyhd3eb1b0_0.tar.bz2
Removed readline-8.1.2-h7f8727e_1.tar.bz2
Removed _libgcc_mutex-0.1-main.tar.bz2
Removed tk-8.6.12-h1ccaba5_0.tar.bz2
Removed sqlite-3.39.2-h5082296_0.tar.bz2
Removed pip-21.1.2-py38h06a4308_0.tar.bz2
Removed ld_impl_linux-64-2.38-h1181459_1.tar.bz2
Removed libgomp-11.2.0-h1234567_1.tar.bz2
Removed libffi-3.3-he6710b0_2.tar.bz2
WARNING: /root/.conda/pkgs does not exist
Cache location: /opt/conda/pkgs
Will remove the following packages:
/opt/conda/pkgs
---------------
_libgcc_mutex-0.1-main 7 KB
_openmp_mutex-5.1-1_gnu 102 KB
---------------------------------------------------
Total: 109 KB
Proceed ([y]/n)?
removing _libgcc_mutex-0.1-main
removing _openmp_mutex-5.1-1_gnu
/bin/sh: 1: killall: not found
The command '/bin/sh -c export PIP_LOG="/tmp/pip_log.txt" && touch ${PIP_LOG} && tail -f ${PIP_LOG} & conda env create -n myenv --file conda_environment.yml && conda clean --all && killall tail && rm ${PIP_LOG}' returned a non-zero code: 127Dockerfile and conda_environment.ymlDockerfile ````Dockerfile FROM continuumio/miniconda3:latest COPY conda_environment.yml conda_environment.yml RUN export PIP_LOG="/tmp/pip_log.txt" && touch ${PIP_LOG} && tail -f ${PIP_LOG} & conda env create -n myenv --file conda_environment.yml && conda clean --all && killall tail && rm ${PIP_LOG} ````conda_environment.yml |
|
For later phases of the env creating the log-file works great. |
|
So subprocess has issues with large stdout buffers. Essentially once it's full it will stop consuming, and might even block the underlying subprocess if it's implemented to block on stdout. the executor package does have a solution to this where it writes to a temporary file, and optionally reads that back for real-time monitoring of logs https://executor.readthedocs.io/en/latest/api.html#executor.ExternalCommand.buffered Since pip can really go off the rails with output, it's not a good idea to try and stream the output of pip using subprocess without some extra stream buffer handling. |
Current Behavior
Installing a conda environemnt with
conda env create -vv --file ./env.ymlseems to get stuck at the pip installation.It seems that the
verboseoption-vvdoes not enable pip install logs to be shown.Steps to Reproduce
Where
env.ymlhas pip packages to install.This seems to wait at
Installing pip dependencies: ...working...without forwarding pip install log output untill the pip install is finished.This "hangs" for quite a while without any feedback if custom pip-install compile flags are being used because it doesn't forward any logs until the compilation is finished. When pip install finishes, it prints the non-verbose pip install log after
Ran pip subprocess with arguments:For example use
conda env create -vv --file env.ymlwhereenv.ymlhas the following content:Expected Behavior
--verboseand-vvoptions.Environment Information
`conda info``conda config --show-sources``conda list --show-channel-urls`The text was updated successfully, but these errors were encountered: