Seems lots of package installing issues with requirements.txt

[ymcdull]

I have tried to run "neptune send experiment_manager.py --environment keras-2.0-gpu-py3 --worker gcp-gpu-medium --config neptune_config.yaml -- train_evaluate_predict_pipeline --pipeline_name glove_lstm"

but got lots of package installation issues. I tried to comment out some in requirements.txt, but seems they are so many. Any solution for this? Or I just missed something in configuration? Thanks.

[jakubczakon]

I have just cloned the repo from master to new env on ubuntu 16.04, ran pip install neptune-cli --upgrade and ran the script you pasted above. My model is training.
What OS are you using?

[ymcdull]

I am using Mac OS now. Are they really related to the OS? Cause I have not set up the local environment on my laptop, and just send the files to Neptune cloud. I guess all the packages in requirements.txt will be installed in the Neptune machine, correct? And my local machine just played the 'sending' role.

[kamil-kaczmarek]

@jakubczakon @ymcdull,

Can you send us entire error message? We will take a closer look at it.

In principle, it is correct, that all requirements are installed on the VM that runs your experiment in the cloud.

I assume that you have installed neptune-cli on your machine. Recently, we have updated neptune to the new version, that is 2.6. Make sure that you have the latest version installed locally: pip3 install neptune-cli -U.

Best,

Kamil

[ymcdull]

@kamil-kaczmarek thanks Kamil. I have tried to pip upgrade my neptune-cli, but the same error happens here:

[pip] Collecting attrdict==2.0.0 (from -r /tmp/tmp5bi6stj8 (line 1))
[pip] Downloading attrdict-2.0.0.tar.gz
[pip] Collecting affine==2.0.0.post1 (from -r /tmp/tmp5bi6stj8 (line 2))
[pip] Downloading affine-2.0.0.post1-py3-none-any.whl
[pip] Collecting appdirs==1.4.3 (from -r /tmp/tmp5bi6stj8 (line 3))
[pip] Downloading appdirs-1.4.3-py2.py3-none-any.whl
[pip] Collecting Babel==1.3 (from -r /tmp/tmp5bi6stj8 (line 4))
[pip] Downloading Babel-1.3.tar.gz (3.4MB)
[pip] Collecting backports-abc==0.5 (from -r /tmp/tmp5bi6stj8 (line 5))
[pip] Downloading backports_abc-0.5-py2.py3-none-any.whl
[pip] Collecting backports.functools-lru-cache==1.4 (from -r /tmp/tmp5bi6stj8 (line 6))
[pip] Downloading backports.functools_lru_cache-1.4-py2.py3-none-any.whl
[pip] Collecting backports.shutil-get-terminal-size==1.0.0 (from -r /tmp/tmp5bi6stj8 (line 7))
[pip] Downloading backports.shutil_get_terminal_size-1.0.0-py2.py3-none-any.whl
[pip] Collecting backports.ssl-match-hostname==3.5.0.1 (from -r /tmp/tmp5bi6stj8 (line 8))
[pip] Downloading backports.ssl_match_hostname-3.5.0.1.tar.gz
[pip] Collecting backports.weakref==1.0.post1 (from -r /tmp/tmp5bi6stj8 (line 9))
[pip] Downloading backports.weakref-1.0.post1-py2.py3-none-any.whl
[pip] Collecting beautifulsoup4==4.4.1 (from -r /tmp/tmp5bi6stj8 (line 10))
[pip] Downloading beautifulsoup4-4.4.1-py3-none-any.whl (81kB)
[pip] Collecting bleach==1.5.0 (from -r /tmp/tmp5bi6stj8 (line 11))
[pip] Using cached bleach-1.5.0-py2.py3-none-any.whl
[pip] Collecting boto==2.47.0 (from -r /tmp/tmp5bi6stj8 (line 12))
[pip] Downloading boto-2.47.0-py2.py3-none-any.whl (1.4MB)
[pip] Collecting bunch==1.0.1 (from -r /tmp/tmp5bi6stj8 (line 13))
[pip] Downloading bunch-1.0.1.zip
[pip] Collecting bz2file==0.98 (from -r /tmp/tmp5bi6stj8 (line 14))
[pip] Downloading bz2file-0.98.tar.gz
[pip] Collecting bzr-etckeeper==0.0.0 (from -r /tmp/tmp5bi6stj8 (line 15))
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit

What I did is comment out the bzr-etckeeper line in requirements.txt, but later other packages will have the similar issue.

[ymcdull]

@kamil-kaczmarek following are messages from stderr:

[pip] Could not find a version that satisfies the requirement bzr-etckeeper==0.0.0 (from -r /tmp/tmp5bi6stj8 (line 15)) (from versions: )
[pip] No matching distribution found for bzr-etckeeper==0.0.0 (from -r /tmp/tmp5bi6stj8 (line 15))
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/deepsense/neptune/job_wrapper.py", line 138, in
execute()
File "/usr/local/lib/python3.6/dist-packages/deepsense/neptune/job_wrapper.py", line 119, in execute
install_requirements()
File "/usr/local/lib/python3.6/dist-packages/deepsense/neptune/job_wrapper.py", line 51, in install_requirements
install_pip_requirements(os.environ['PIP_REQUIREMENTS'])
File "/usr/local/lib/python3.6/dist-packages/deepsense/neptune/job_wrapper.py", line 44, in _check_pip_install_result
raise RuntimeError('pip failed to install the requirements. '
RuntimeError: pip failed to install the requirements. For more details, see the stdout/stderr channels.

[kamil-kaczmarek]

@jakubczakon @ymcdull ,

1. I assume that you successfully installed the latest neptune-cli, that is 2.6.1.
2. Please double check, if you are using the latest version of the requirements.txt on branch master.
3. We will try to reproduce your error.

[ymcdull]

@kamil-kaczmarek thanks for your reply. I finally find out the problem: my version is not the latest one. I git cloned the repo several days ago and haven't checked the version again. The earlier requirements.txt file has hundreds of packages. Now I git pull the latest version and don't have the pip install issue anymore, and it is doing the training now on Neptune. Thanks a lot for your help.