conda install doesn't properly pull in dependencies

[dbkeitel]

Which documentation page is causing an issue?
https://legwork.readthedocs.io/en/latest/install.html

What is the problem?
On my Debian 11 system with conda 4.11.0, the current instructions don't give me a complete install anymore:

conda create --name legwork
conda activate legwork
pip install legwork

Then it seems pip sees some dependencies on the system path is happy with those, but doesn't actually put them into the conda environment, leaving it incomplete:

Collecting legwork
  Using cached legwork-0.2.3-py3-none-any.whl (24.5 MB)
Collecting seaborn>=0.11.1'
  Using cached seaborn-0.11.2-py3-none-any.whl (292 kB)
Requirement already satisfied: astropy>=4.0' in /usr/lib/python3/dist-packages (from legwork) (4.2)
Requirement already satisfied: scipy>=1.5.0' in /usr/lib/python3/dist-packages (from legwork) (1.6.0)
Collecting numba<0.54',>=0.50
  Using cached numba-0.53.1-cp39-cp39-manylinux2014_x86_64.whl (3.4 MB)
Requirement already satisfied: matplotlib>=3.3.2' in /usr/lib/python3/dist-packages (from legwork) (3.3.4)
Collecting schwimmbad>=0.3.2'
  Using cached schwimmbad-0.3.2-py3-none-any.whl
Requirement already satisfied: numpy<1.20',>=1.16 in /usr/lib/python3/dist-packages (from legwork) (1.19.5)
Collecting llvmlite<0.37,>=0.36.0rc1
  Using cached llvmlite-0.36.0-cp39-cp39-manylinux2010_x86_64.whl (25.3 MB)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from numba<0.54',>=0.50->legwork) (52.0.0)
Collecting pandas>=0.23
  Using cached pandas-1.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
Collecting pytz>=2020.1
  Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/lib/python3/dist-packages (from pandas>=0.23->seaborn>=0.11.1'->legwork) (2.8.1)
Installing collected packages: pytz, pandas, llvmlite, seaborn, schwimmbad, numba, legwork
Successfully installed legwork-0.2.3 llvmlite-0.36.0 numba-0.53.1 pandas-1.4.0 pytz-2021.3 schwimmbad-0.3.2 seaborn-0.11.2

And then any imports of legwork fail with missing scipy/numpy/astropy/matplotlib.

Suggested solution (optional)
I think you can't get around using conda installation of the dependencies, even if that means double-maintaining them in an extra .yml file or the instructions page.

[TomWagg]

Haha it seems every "fix" we do for these dependencies breaks it in a different way 😂

I think it probably makes sense to just include the version numbers manually on the instruction page since users would need to download that .yml file manually anyway right? Might as well just have a copy-able code block instead.

I'll add these back and add notes to the requirement.txt file to remember to always update in both locations.

[dbkeitel]

The advantage of providing a .yml is that at a later point, people can re-download it and do

conda env update --file environment.yml  --prune

to get all dependencies updated consistently.

But your suggestion is certainly the simplest.

[TomWagg]

Okay I see what you mean. I don't know that much about package managers and whatnot but I think that if, at some point in the future, a user wants to update legwork they could run

pip install legwork --upgrade

My question is then, when that upgrade occurs, would it also check whether the dependencies could be upgraded without needing to update the environment with the new .yml file?

[TomWagg]

Having googled this a bit I've realised that pip upgrades are completely invisible to conda so I do see how the environment.yml file could be very useful here.

I'm going to add it for the stable case but I think there isn't much point with the development version since you always have to use pip to install it anyway.

[dbkeitel]

Trying to understand the differences between the stable and dev instructions, I've stumbled about non-intuitive (for me at least) behaviour regarding how conda isolates (or doesn't isolate) an env from system packages. Apparently this depends on whether the env's python version matches the system's:

• If I create an env with the same python version as my system, then...
  • the current stable instructions fail as described above
  • the current dev instructions sort-of-work, with dependencies not being actively installed into the env but available
• If I create an env with a different python version, then...
  • the current stable instructions work, with pip actually pulling in all dependencies and installing them into the env
  • the current dev instructions work the same

So unless you understand these things better than I do and know of a perfectly clean solution, I think changing the stable instructions to use a yml and leaving the dev case as it currently stands should at least give working installations in each situation.

[dbkeitel]

PS on your question regarding pip install legwork --upgrade my (also limited) understanding is that it may be more fragile than the yaml route.