Cross-platform requirements files

[ryanpdwyer]

I really like using conda for creating and managing environments with scientific python packages. Is there a way to create a cross-platform requirements file? Right now, if I create a requirements file on Mac, and then install on Linux, some dependencies will be missed, and likewise, some Mac dependencies have no equivalent on Linux or Windows.

If I specify a list of requirements at the command line (specifying only packages in a setup.py install_requires, for example),

conda create -n newenv python=2.7 numpy=1.9 scipy=0.14

conda can resolve them correctly. I end up manually write shell scripts like this as a workaround. Is there a way to do something like this automatically, so that the proper requirements would be specified regardless of platform?

[dwyatte]

+1 for resolving this issue.

I'd like to migrate to conda from pip as to not deal with the scientific python stack baggage (blas, etc., when building from source on linux) as well as the reap the performance benefits associated with the recent mkl redistribution. However, I'd also like to maintain the cross-platform nature that pip provides e.g., between Mac and Linux.

There are two problems, as far as I can tell.

1.) Generating an environment.yml ("conda env export") or spec-file.txt ("conda list -e") lists the version dependency as well as the specific build. The build component is not cross-platform (might exist in one OS channel but not the other) :
numpy=1.10.4=py27_1
version: 1.10.4
build: py27_0

2.) Non-python components are not cross-platform. e.g., scipy on Linux depends on libgfortran, but scipy on Mac does not.

It has been pointed out elsewhere that hand-editing the environment file can create something that gets auto-resolved when a user does 'conda install ...' (https://www.continuum.io/blog/developer-blog/explicit-conda-environment-specifications#comment-2368808739) but I would really be interested in an intelligent environment export function that reduces the requirements to an OS agnostic set automatically.

[dwyatte]

Looks like this is also being tracked in anaconda ContinuumIO/anaconda-issues#546

[allanleal]

For those interested in a solution to maintain a single environment file that can be used in Linux, macOS, and Windows, please check the conda-devenv tool at https://github.com/ESSS/conda-devenv.

[thompsonmj]

The approach for reproducible, upgradable, and cross-platform-compatible environments that seems to require the least overhead (i.e. manually configuring environment files) I've come across is to use conda-lock.

Learned about it through this helpful article.

Haven't thoroughly tested it, but I'll try incorporating it into my projects and see how it goes!

e.g. usage with mamba for faster environment setup

conda create --name myenv
conda activate myenv
conda install -c conda-forge mamba
mamba install -c conda-forge python conda-lock jupyterlab jupytext jupyterlab-git
conda deactivate
conda env export -n myenv -c conda-forge --from-history > environment.yml 
conda activate myenv
conda-lock # creates lock files for osx-64, win-64, and linux-64 by default
# run the previous 4 lines every time a new package is added to the environment
conda activate myenv
# use the environment

# on a new machine, build the environment with one of the following:
# Ubuntu
conda create --name myenv --file conda-linux-64.lock
# OSX
conda create --name myenv --file conda-osx-64.lock
# Windows
conda create --name myenv --file conda-win-64.lock

# or install mamba in the base environment and use it to create the virtual environments from the lock files

[gregorywaynepower]

The approach for reproducible, upgradable, and cross-platform-compatible environments that seems to require the least overhead (i.e. manually configuring environment files) I've come across is to use conda-lock.

Learned about it through this helpful article.

Haven't thoroughly tested it, but I'll try incorporating it into my projects and see how it goes!

e.g. usage with mamba for faster environment setup

conda create --name myenv
conda activate myenv
conda install -c conda-forge mamba
mamba install -c conda-forge python conda-lock jupyterlab jupytext jupyterlab-git
conda deactivate
conda env export -n myenv -c conda-forge --from-history > environment.yml 
conda activate myenv
conda-lock # creates lock files for osx-64, win-64, and linux-64 by default
# run the previous 4 lines every time a new package is added to the environment
conda activate myenv
# use the environment

# on a new machine, build the environment with one of the following:
# Ubuntu
conda create --name myenv --file conda-linux-64.lock
# OSX
conda create --name myenv --file conda-osx-64.lock
# Windows
conda create --name myenv --file conda-win-64.lock

# or install mamba in the base environment and use it to create the virtual environments from the lock files

How has your luck been with conda-lock?

[thompsonmj]

@gregorywaynepower Haha, well to be honest, I had started using Julia primarily shortly after this, and environments were in general much easier to manage. Now that I'm getting back into more Python work recently, I will pick this back up.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[shahrokh-bahtooei]

Perhaps, using Docker on all platforms will be the ultimate solution to this issue…

[wolfv]

You could take a look at pixi which does this & much more: https://pixi.sh/