pip-installed packages not in "conda list --explicit"

[rgommers]

Current Behavior

conda list --explicit does not include any pip-installed packages, while conda list does. It's unclear to me if that is on purpose or not, but if it is then that needs mentioning in https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments because users will be ending up with incomplete environments if they have any pip-installed packages.

Steps to Reproduce

$ conda --version
conda 4.6.7
$ conda create -n explicit
$ conda install pip
$ pip install Pillow
$ conda list  # output will include Pillow
$ conda list --explicit  # output will not include Pillow

Expected Behavior

I understand that conda cannot easily give a hard guarantee about the exact binaries or sdists installed by pip, however including the pip-installed packages in the output of conda list --explicit (just copy conda list behavior) still seems way more useful than leaving them out.

Environment Information

$ conda info

     active environment : explicit
    active env location : /home/rgommers/anaconda3/envs/explicit
            shell level : 1
       user config file : /home/rgommers/.condarc
 populated config files : /home/rgommers/.condarc
          conda version : 4.6.7
    conda-build version : 3.4.1
         python version : 3.6.8.final.0
       base environment : /home/rgommers/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/rgommers/anaconda3/pkgs
                          /home/rgommers/.conda/pkgs
       envs directories : /home/rgommers/anaconda3/envs
                          /home/rgommers/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.7 requests/2.19.1 CPython/3.6.8 Linux/4.20.12-arch1-1-ARCH antergos/ glibc/2.28
                UID:GID : 1000:985
             netrc file : None
           offline mode : False

`conda config --show-sources`

$ conda config --show-sources
==> /home/rgommers/.condarc <==
channels:
  - defaults
  - conda-forge

[prjemian]

I agree. This situation has burned me in the past and it would be very useful to include mention of the pip-installed (or other non conda-installed) packages as comments. Such as:

...
http://conda.anaconda.org/psi4/linux-64/boost-1.57.0-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2018.10.15-ha4d7672_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/conda-env-2.6.0-1.tar.bz2
# no URL for: dateutil-2.1-py27_2
http://conda.anaconda.org/psi4/linux-64/dkh-1.2-h173d85e_2.tar.bz2
http://conda.anaconda.org/psi4/linux-64/erd-3.0.6-0.tar.bz2
# no URL for: cf-units                  1.1.3                     <pip>
...

[dpizetta]

Options with conda list

-c, --canonical       Output canonical names of packages only. Implies --no-pip.
    --explicit        List explicitly all installed "conda packaged" with URL 
                       (output may be used by conda create --file).
-e, --export          Output requirement string only 
                       (output may be used by conda create --file).
    --no-pip          Do not include pip-only installed packages.

If I understand with "--explicit" it lists only installed by conda, not including pip (not clear). However, if you use only --no-pip option, it lists everything, exactly identical to conda list, that it is equal (in packages, not how they are described) to --export. So it seems that they are not working properly.

Some outputs in the files. Maybe another issue for conda --no-pip as it is not working.

conda-list.txt
conda-list-canonical.txt
conda-list-explicit.txt
conda-list-export.txt
conda-list-no-pip.txt

[wshanks]

I just want to emphasize what @prjemian showed in his suggestion. He shows the pip packages as lines beginning with #. I have scripts that export conda environments using conda list --explicit so that the packages can be installed in another environment with conda install --file. I don't believe there is a way to format the URL for a pip package that works with conda install. I'd like the conda list --explicit -> conda install --file workflow not to be broken.

[robertcatgithub]

This is confusing.

So to create an EXACT clone of a working env on another host, - with ALL installed pkgs - Is this the recommended way ?

1. conda env export > environment.yml
2. conda env create -f environment.ym

[ronif]

Hey! Is there any update regarding this?

The issue is important since conda list --explicit is the only way to recreate an exact copy of a given env. conda env export does not work since the repodata.json in the channels changes and breaks dependencies: #9257

[ivanlen]

I know it is not a definitely solution,
but if I am in a conda virtual env, first I need to install python in that particular virtual env,

conda install python,

then I install the packages using pip, and then if I conda list it will list correctly all the conda and the pip installed packages.

If I don't install python in that particular when I conda list I do not see anything.

[christian-cahig]

I know it is not a definitely solution,
but if I am in a conda virtual env, first I need to install python in that particular virtual env,

conda install python,

then I install the packages using pip, and then if I conda list it will list correctly all the conda and the pip installed packages.

If I don't install python in that particular when I conda list I do not see anything.

This did not work for me though :(

[epassaro]

Any update on this?

[mforbes]

This issue needs a very high priority: reproducible computing requires the ability to reproduce the computing environment. I thought that the point of Conda was to promote reproducibility, but without this basic functionality, or a clearly documented workaround, conda fails in this objective. Unless things have changed, the process of getting a package currently only available with pip converted and built as a conda package is sufficiently difficult as to be not a reasonable option for regular users workflows.

Until this is address, what is the official recommendation from the conda team? Or is my notion that conda aims to facilitate reproducible computing in error?

Perhaps picky-conda provides a solution? It is really not very customizable, but may do what we need.

[Ellutze]

Does anyone know if an work around exists now? i.e. is there a way to reliably clone existing environment including pip installed packages?

[epassaro]

conda env export works differently but lists the pip packages.

[vivianamarquez]

I usually save the previous environment with:

• conda env export > environment.yml
• conda list --explicit > spec-file.txt
• pip freeze > requirements.txt

and then I create my new environment using:

• conda create --name NEWENV --file spec-file.txt
• conda env update --name NEWENV --file environment.yml
• pip install -r requirements.txt

is there a better/more complete way to do this?

[wtfzambo]

I usually save the previous environment with:

* `conda env export > environment.yml`

* `conda list --explicit > spec-file.txt`

* `pip freeze > requirements.txt`

and then I create my new environment using:

* `conda create --name NEWENV --file spec-file.txt`

* `conda env update --name NEWENV --file environment.yml`

* `pip install -r requirements.txt`

is there a better/more complete way to do this?

Wouldn't pip freeze also include all the conda installed packages? I am currently facing the same issue but I'm not sure how to address it

[wshanks]

Wouldn't pip freeze also include all the conda installed packages?

Yes, but typically pip should see the conda installed packages and not overwrite them. This is not guaranteed though.

[wtfzambo]

Wouldn't pip freeze also include all the conda installed packages?

Yes, but typically pip should see the conda installed packages and not overwrite them. This is not guaranteed though.

How safe is that option?

[ivanlen]

Wouldn't pip freeze also include all the conda installed packages?

Yes, but typically pip should see the conda installed packages and not overwrite them. This is not guaranteed though.

How safe is that option?

I broked a couple of virtual envs mixing conda and pip.

In general my approach is trying to install everything with conda (which manages ok the dependencies) and at the end if there is something that can't be installed with conda I install it with pip.

[wtfzambo]

Wouldn't pip freeze also include all the conda installed packages?

Yes, but typically pip should see the conda installed packages and not overwrite them. This is not guaranteed though.

How safe is that option?

I screw a couple of virtual envs mixing conda and pip.

In general my approach is trying to install everything with conda (which manages ok the dependencies) and at the end if there is something that can't be installed with conda I install it with pip.

I do the same, my only concern is re-creating the environment on a different machine. If my friend has a mac, I want to share with him only the top lvl packages that I installed in that env.

With the --from-history flag I can do it with conda packages, but then it won't include the pip installed ones.

How do you work around this?

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[mforbes]

Dear bot. This issue has not yet been resolved (at least with version 23.5.0): it might be more of a documentation issue, but someone needs to weigh in and fix this one way or the other. It is not stale, just old.

[charlesmarseille]

This is a real pain. It is a basic and common functionnality to recreate an env. What is the proposed approach please?

[kamurani]

Chiming in; I am currently in the midst of this exact issue. I have a local package with pip install -e . that I have only managed to get working on a single machine; every option I've used to preserve the conda environment that it works on, and move to another machine, has failed. Every single time I try to run it and use pip install on the new machine, despite building the identical conda environment from environment.yml, it still messes up some dependencies that require pip (i.e. no conda equivalent)

[GLARKI]

I usually save the previous environment with:

• conda env export > environment.yml
• conda list --explicit > spec-file.txt
• pip freeze > requirements.txt

and then I create my new environment using:

• conda create --name NEWENV --file spec-file.txt
• conda env update --name NEWENV --file environment.yml
• pip install -r requirements.txt

is there a better/more complete way to do this?

Is there a way to do this from script? might be the easiest workaround :P

---

EDIT:
Simplest way to clone the env is using yml:

1. Save environment in the cmd run: conda env export > environment.yml
2. Command out, or delete the prefix at the bottom of the environment.yml file

This also copies a lot of dependancies in that arent strictly needed. These can then be removed/cleaned.