Properly include channels and channel prefix into conda env export --from-history

[liqimai]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

In short, I want the package list created by conda env export --from-history to be concise, cross-platform, and highly reproducible. There are two drawbacks in its current output.

1. the missing channel problem

Now, conda env export --from-history includes packages that I’ve explicitly asked for. It works well when I only install packages from default channels. If I have installed some packages from third-party channels, such as pytorch. It would not include the third-party channels.

e.g. Here is the command to install pytorch.

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

Here is the output of conda env export --from-history

name: motion
channels:
  - defaults
  - conda-forge
dependencies:
  - python==3.10
  - torchaudio
  - pytorch
  - pytorch-cuda=11.8
  - torchvision
prefix: /opt/conda/envs/motion

The channel pytorch is missed, because I did NOT add 'pytorch' channel by conda config --add channels pytorch, but just temporally add it by conda install -c pytorch.

2. the channel ambiguity

Nowadays, users usually install a package from a channel by conda install -c command. For the sake of cross-platform reproducibility, we should also include the channel prefix in the output of conda env export --from-history. e.g.

  - pytorch::pytorch-cuda=11.8    # add prefix "pytorch::"
  - pytorch::pytorch              # add prefix "pytorch::"

This is because pytorch exists both in conda-forge and pytorch channel. To avoid the ambiguity, we need to explicitly specify the pytorch in pytorch channel.

Why is this needed?

It should include all channels these explicit packages come from, and add precise channel prefixes to the packages that come from non-default channels. see "What should happen?" for an example.

What should happen?

Here is an environment.yml I created manually. It represents the output of conda env export --from-history I expected and requested in this issue.

It differs from the current output of conda env export --from-history in the following ways:

1. all channels are included, as long as any package user asked comes from the channel, even if the channel is not added to config by conda config --add channels.
2. add a channel prefix to the package, if the package does not come from the default channel.

name: myenv
channels:
  - pytorch3d
  - pytorch
  - huggingface
  - nvidia
  - defaults
  - conda-forge
dependencies:
  # packages from the default channel
  - python==3.10
  - dm-tree
  - matplotlib
  - pytest
  - rich
  - ipython
  - ipykernel
  - scipy

  # packages with channel prefix, i.e., installed by `conda install -c <channel> <package>`
  - pytorch::pytorch-cuda=11.8
  - pytorch::pytorch
  - pytorch::torchaudio
  - pytorch::torchvision
  - conda-forge::yapf~=0.32
  - conda-forge::ipywidgets
  - conda-forge::wandb
  - conda-forge::accelerate
  - huggingface::transformers
  - huggingface::datasets
  - pytorch3d::pytorch3d

  # pip packages
  - pip:
      - mpmath==1.2.1
      - pyqt5-sip==12.11.0
prefix: /opt/conda/envs/myenv

Additional Context

No response

[liqimai]

The missing channel problem is more like a bug, not a feature request.
The channel prefix is a real feature request.

[travishathaway]

Hi @liqimai,

Thanks for submitting this feature request. We really appreciate how detailed it is. Below are my thoughts on what you have proposed.

The first item you suggested would be trickier to implement because the order in which those channels are listed reflect the priority given to it while resolving the dependencies for a project. When channels are given at the command line for single commands they are temporarily given the highest level of priority while installing those particular dependencies. You can imagine how that may then become tricky when we are later determining the priority of a channel for the environment file we want to export. What if we specific multiple different channels this way? Which one deserves a spot at the top?

But, I think the feature you proposed (channel prefix) may be the best way to circumvent this confusion. Every dependency could easily be locked to its channel by using the channel prefix when listing the dependencies. I ran some tests myself and determined that is not currently happening.

Unfortunately, we cannot promise when we would get around to implementing this for conda env export, but it sounds like an item that could be included were we to do a greater refactor of this command. As always, we would gladly welcome any pull requests to add this bit of functionality.

In the meantime, we'll keep this feature request open and welcome any further conversation surrounding this issue.

[liqimai]

@travishathaway thanks for your reply! Does the same issue exist in conda env export command without --from-history? I guess not, because I notice that conda env export includes package hash values. I think the hash values are enough to resolve the channel ambiguity.

[travishathaway]

Hi @liqimai,

It appears not to exist for the conda env export command.