Use environment.yml mechanism for specifying conda dependencies

[hackalog]

Anaconda has a yml-based specification file for indicating dependencies. We should use this instead
of pip if we are using conda to manage virtual environments

• Add a configuration-time question to choose between conda and virtualenv explicitly
• Use an environment.yml to specify the conda dependencies
• Install dependencies at venv-creation time, since we already have this info

As an added bonus, this makes it a little easier to add pipenv support by extending the VIRTUALENV choices

[pjbull]

Thanks for the PR and the thoughts!

We've been tracking conda's environment.yml for a while.

In the past, we have not recommended it for one big reason and a couple of small reasons. The big reason is that the yml specification was platform (windows, macos, linux) specific, and the conda maintainers recommended the machine-generated exported specification (which includes platform) rather than hand-editing dependencies. For the projects we work on, the fact that pip does the right thing per platform made it the right choice. Maybe this has changed since we last looked?

The little reasons include (again, maybe have changed?): no equivalent of -e for the src module, moving target in terms of what channels are recommended/secure, steeper learning curve for yml than requirements.txt. The final reason, is that requirements.txt is much more common, and it works regardless of your choice for environment manager (that is, you can pip install in a conda environment without any problems).

All this to say that I'm not quite over the hump yet in terms of recommending environment.yml for every conda project as the default since requirements.txt works and will continue to work across environment managers.

[hackalog]

Thanks for the quick reply!

I have definitely encountered version conflicts in mixing pip and conda-installed libraries, which is why my typical workflow involves installing everything I can using conda, and using pip as a last resort. I'd love a solution that would permit mixing the two.

conda seems to have merged in functionality to pass parameters to pip in this PR. Does this seem like it addresses some of your concerns?

conda/conda#3969

Also, I'd love to find out if conda has changed from recommending platform-specific .yml files. Can you point me to where this recommendation was previously made?

[mfgray]

Just like to provide a concrete example of a conda + pip mixing issue with a package issues/3378

[hackalog]

It looks like yes, conda/conda#3969 did add the ability to pass options to pip from the conda environment.yml, so I've added in the pip install -e . functionality.

It also looks like since conda/conda#6221, conda now allows the environment to be exported in a non-platform-dependent way

[hackalog]

I've moved things around a bit. Though the pip flags were working on linux in recent conda, my macbook was having issues, so now I'm using environment.yml for the conda modules, and using the requirements.txt for the pip-only packages (by default, just -e .)

[knaaptime]

just wanted to vote +1 on this. I use geopandas a lot, which depends on GDAL, so the option to use conda over pip would be a massive boon

Both conda-forge and defaults are quite stable these days so it sounds like many of the lingering issues blocking this have been resolved?

[rodgzilla]

Hi everyone,

I am working with your project template (which is awesome by the way) using conda environments.

During setup I had a crazy problem and I think it may help @hackalog to create its patch. When creating a conda virtual environment and exporting it using conda env export > environment.yml I had some problems with the resulting file. More precisely, the file contained the following lines:

[...]
  - pip:
    - src==0.1.0
[...]

I didn't really know what the src package so I thought it was a dependency to something I installed. However while importing the environment, I kept getting the following message:

Could not find a version that satisfies the requirement src==0.1.0 (from -r [...]/condaenv.x6os_xmh.requirements.txt (line 1)) (from versions: 0.0.1, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7)
No matching distribution found for src==0.1.0 (from -r [...]/condaenv.x6os_xmh.requirements.txt (line 1))

At which point I activated the original environment to see what was going on with this package and I got:

pip search src
src (0.0.7)                        - UNKNOWN
  INSTALLED: 0.1.0
  LATEST:    0.0.7

Apparently, pip is thinking that the src folder (that contains a __init__.py) is a pip module and is trying to export it. I think this is happening because there is a package named src in the pip database. Finding this problem was REALLY frustrating, I hope this message can prevent someone else of going through the same trouble.

[hackalog]

I think the best thing to have there is:

  - pip:
    - -e .

With a recent conda, this works just Fine. I've been putting it through the paces for a couple of months now, so I think it's time to respin and bump this PR.

[pjbull]

Continuing this discussion in #118

[hackalog]

Okay, I've updated this patch to reflect the way I've been using conda the past few months. In short, I think this addresses all the issues raised by @pjbull

I'd love to hear test results from others using conda...

[hackalog]

I noticed a merge conflict crept in after some recent fixes to master. I've merged them and respun this PR. Any feedback? @pjbull?

[pjbull]

Thanks for the ping @hackalog.

I'm planning to take a look this month. Long story short is that I think we need a coherent take on both environment management and dependency specification that: (1) is simple for users, (2) reliably works together / is tested (since it expands the surface area of what is "supported"). I'd like to have the chance to test this, pipenv, and classic requirements.txt and expose those options without overwhelming.

[avivrosenberg]

@hackalog Thanks for this PR. I've tried it and ran into an error due to your Makefile modifications.
In recent versions, it seems that conda is actually a shell function, not an executable on the $PATH. So invoking e.g. conda env create from the Makefile as you did resulted in

make: conda: No such file or directory
make: *** [create_environment] Error 1

The conda function is defined in $CONDA_PATH/etc/profile.d/conda.sh (where anaconda/miniconda is installed). This script is what they recomend to source in bashrc/zshrc etc.
The script defines the $CONDA_EXE shell variable which is the location of the actual conda binary.

So, to make it work I had to change the invocations of conda in your Makefile to ${CONDA_EXE}, e.g.

${CONDA_EXE} env create --name $(PROJECT_NAME) -f environment.yml

Perhaps this will help someone.

[hackalog]

Yes. In fact, I have this very workaround in my cookiecutter-easydata fork already. I will update the PR with something similar.

Nowadays, I tend to also ask for the location of the conda binary as one of the cookiecutter questions, and I use this in case CONDA_EXE is not set (which is possible if the system's root python is not anaconda, and no conda environment is yet activated)

[pjbull]

Thanks @hackalog for this PR and the discussion here! I'm closing this since we're planning to get #162 in this month, which has environment.yml support inspired by your work. Much appreciated!