New feature suggested in "pip list": option to differentiate packages installed by pip and conda.

[vx1920]

I am using Anaconda, some packages installed by conda, but some other avalable only from "pip install". I have problem with packages updates, since Anaconda packages should be updated with conda and pip packages should be updated respectively with pip (normally).

Updating of conda packages can be done properly using "conda update --all", but "pip list outdated" reports all outdated packages, including just updated conda ones. Problem is that some packages can be installed in both ways and ones from pip are oftenly newer, since Anaconda doing prpagation with some delay. Normally it is better to use slightly older conda-installed packages with conda, but pip-only packages have to be updated and I want to see who did install package originally.
There is optional column "Installer" in "pip list format=columns -verbose" with pip/conda values inside, but experiments shows this information is useless for me. It may just indicate that package
uses metadata in "*dist-info" folder format and it is true for many conda-installed packages.

Solution: I need additional column with additional information.
My new suggested output format can be enabled using new option "format=columnsx".
Pay attention to new column "Owner" with "Anaconda"/"ThePip" values.
You can also see that column "Installer" is useless, while new "Owner" is more useful...

Act: d:\InstSoft\Python\Conda3x64\Scripts\activate.bat 
Run: d:\InstSoft\Python\Conda3x64\Python.exe
Arg: -m pip list --format=columnsx
Package                              Version             Installer Owner   
------------------------------------ ------------------- --------- --------
adal                                 1.2.4               pip       ThePip  
alabaster                            0.7.12              pip       Anaconda
anaconda-client                      1.7.2                         Anaconda
anaconda-navigator                   1.9.12                        Anaconda
anaconda-project                     0.8.3               conda     Anaconda
ansiwrap                             0.8.4               pip       ThePip  
appdirs                              1.4.3                         Anaconda
applicationinsights                  0.11.9              pip       ThePip  
argh                                 0.26.2                        Anaconda
asn1crypto                           1.3.0               pip       Anaconda
astroid                              2.4.1               pip       Anaconda
astropy                              4.0.1.post1                   Anaconda
async-generator                      1.10                pip       ThePip  
atomicwrites                         1.4.0               conda     Anaconda
attrs                                19.3.0              conda     Anaconda
augment-nd                           0.1.1               pip       ThePip  
autopep8                             1.4.4               pip       Anaconda
azure-common                         1.1.25              pip       ThePip  
azure-core                           1.6.0               pip       ThePip  
azure-graphrbac                      0.61.1              pip       ThePip  
azure-identity                       1.3.1               pip       ThePip  
azure-mgmt-authorization             0.60.0              pip       ThePip  
azure-mgmt-containerregistry         2.8.0               pip       ThePip  
azure-mgmt-keyvault                  2.2.0               pip       ThePip  
azure-mgmt-network                   10.2.0              pip       ThePip  
azure-mgmt-resource                  10.0.0              pip       ThePip  
azure-mgmt-storage                   10.0.0              pip       ThePip  
azureml                              0.2.7               pip       ThePip  
azureml-automl-core                  1.6.0               pip       ThePip  
azureml-automl-runtime               1.6.0.post1         pip       ThePip  
azureml-contrib-notebook             1.6.0               pip       ThePip  
azureml-core                         1.6.0.post1         pip       ThePip  
azureml-dataprep                     1.7.0               pip       ThePip  
azureml-dataprep-native              14.2.0              pip       ThePip  
azureml-defaults                     1.6.0               pip       ThePip  
azureml-explain-model                1.6.0               pip       ThePip  
azureml-interpret                    1.6.0               pip       ThePip  
azureml-model-management-sdk         1.0.1b6.post1       pip       ThePip  
azureml-pipeline                     1.6.0               pip       ThePip  
azureml-pipeline-core                1.6.0               pip       ThePip  
azureml-pipeline-steps               1.6.0               pip       ThePip  
azureml-sdk                          1.6.0               pip       ThePip  
azureml-telemetry                    1.6.0               pip       ThePip  
azureml-train                        1.6.0               pip       ThePip  
azureml-train-automl                 1.6.0               pip       ThePip  
azureml-train-automl-client          1.6.0.post1         pip       ThePip  
azureml-train-automl-runtime         1.6.0               pip       ThePip  
azureml-train-core                   1.6.0.post1         pip       ThePip  
azureml-train-restclients-hyperdrive 1.6.0               pip       ThePip  
azureml-widgets                      1.6.0               pip       ThePip  
Babel                                2.8.0               conda     Anaconda
backcall                             0.1.0               pip       Anaconda
backports.functools-lru-cache        1.6.1               conda     Anaconda
backports.shutil-get-terminal-size   1.0.0               pip       Anaconda
backports.tempfile                   1.0                 pip       Anaconda
backports.weakref                    1.0.post1                     Anaconda
bcrypt                               3.1.7               conda     Anaconda
beautifulsoup4                       4.9.1               pip       Anaconda
bitarray                             1.2.1               pip       Anaconda
bkcharts                             0.2                           Anaconda
black                                19.10b0             conda     Anaconda
bleach                               3.1.4               conda     Anaconda
blinker                              1.4                           Anaconda
bokeh                                2.0.2               pip       Anaconda
boto                                 2.49.0                        Anaconda
boto3                                1.9.66                        Anaconda
botocore                             1.12.189            conda     Anaconda
Bottleneck                           1.3.2               pip       Anaconda
cachetools                           3.1.1               conda     Anaconda
certifi                              2020.4.5.1                    Anaconda
cffi                                 1.14.0              pip       Anaconda
chardet                              3.0.4               conda     Anaconda
click                                7.1.2               conda     Anaconda
cloudpickle                          1.4.1               conda     Anaconda
clyent                               1.2.2                         Anaconda
colorama                             0.4.3               conda     Anaconda
comtypes                             1.1.7                         Anaconda
conda                                4.8.3                         Anaconda
conda-build                          3.18.11                       Anaconda
conda-package-handling               1.7.0               conda     Anaconda
conda-verify                         3.4.2               conda     Anaconda
configparser                         5.0.0               conda     Anaconda
contextlib2                          0.6.0.post1         conda     Anaconda
coverage                             5.0                           Anaconda
cryptography                         2.9.2               pip       Anaconda
cycler                               0.10.0                        Anaconda
Cython                               0.29.17             pip       Anaconda
cytoolz                              0.10.1                        Anaconda
dask                                 2.17.2              conda     Anaconda
decorator                            4.4.2               conda     Anaconda
defusedxml                           0.6.0                         Anaconda
diff-match-patch                     20181111            conda     Anaconda
dill                                 0.3.1.1             conda     Anaconda
distributed                          2.17.0              pip       Anaconda
distro                               1.5.0               pip       ThePip  
docker                               4.2.1               pip       ThePip  
docutils                             0.16                pip       Anaconda
dotnetcore2                          2.1.14              pip       ThePip  
entrypoints                          0.3                           Anaconda
ephem                                3.7.7.1             pip       ThePip  
et-xmlfile                           1.0.1                         Anaconda
fastcache                            1.1.0               pip       Anaconda
ffmpeg                               1.4                 pip       ThePip  
filelock                             3.0.12              pip       Anaconda
fire                                 0.3.1               pip       ThePip  
flake8                               3.7.9               conda     Anaconda
Flask                                1.1.2               pip       Anaconda
fsspec                               0.7.4               conda     Anaconda
fusepy                               3.0.1               pip       ThePip  
future                               0.18.2              conda     Anaconda
gensim                               3.8.0               pip       Anaconda

This new option is most useful with "pip list --outdated". In the example below
it shows: all outdated items are from conda and pip update is not needed.

Act: d:\InstSoft\Python\Conda3x64\condabin\conda.bat activate
Run: d:\InstSoft\Python\Conda3x64\Python.exe 
Arg: -m pip list --outdated --format=columnsx
Package                  Version           Latest     Type  Installer Owner   
------------------------ ----------------- ---------- ----- --------- --------
appdirs                  1.4.3             1.4.4      wheel           Anaconda
astroid                  2.4.1             2.4.2      wheel pip       Anaconda
autopep8                 1.4.4             1.5.3      sdist pip       Anaconda
bitarray                 1.2.1             1.2.2      sdist pip       Anaconda
bleach                   3.1.4             3.1.5      wheel conda     Anaconda
boto3                    1.9.66            1.13.24    wheel           Anaconda
botocore                 1.12.189          1.16.24    wheel conda     Anaconda
cachetools               3.1.1             4.1.0      wheel conda     Anaconda
certifi                  2020.4.5.1        2020.4.5.2 wheel           Anaconda
coverage                 5.0               5.1        wheel           Anaconda
Cython                   0.29.17           0.29.19    wheel pip       Anaconda
dask                     2.17.2            2.18.0     wheel conda     Anaconda
distributed              2.17.0            2.18.0     wheel pip       Anaconda
flake8                   3.7.9             3.8.2      wheel conda     Anaconda
gensim                   3.8.0             3.8.3      wheel pip       Anaconda
gevent                   1.4.0             20.6.0     wheel           Anaconda
google-api-core          1.17.0            1.19.1     wheel pip       Anaconda
google-auth              1.14.1            1.16.1     wheel conda     Anaconda
google-cloud-storage     1.28.0            1.28.1     wheel conda     Anaconda
google-resumable-media   0.5.0             0.5.1      wheel conda     Anaconda
googleapis-common-protos 1.51.0            1.52.0     wheel pip       Anaconda
greenlet                 0.4.15            0.4.16     wheel pip       Anaconda
hypothesis               5.11.0            5.16.0     wheel conda     Anaconda
importlib-metadata       1.6.0             1.6.1      wheel pip       Anaconda
ipykernel                5.1.4             5.3.0      wheel pip       Anaconda
ipython                  7.13.0            7.15.0     wheel pip       Anaconda
jedi                     0.15.2            0.17.0     wheel pip       Anaconda
jmespath                 0.9.4             0.10.0     wheel pip       Anaconda
json5                    0.9.4             0.9.5      wheel conda     Anaconda
jupyterlab               1.2.6             2.1.4      wheel           Anaconda
jupyterlab-server        1.1.4             1.1.5      wheel conda     Anaconda
keyring                  21.1.1            21.2.1     wheel pip       Anaconda
lazy-object-proxy        1.4.3             1.5.0      wheel conda     Anaconda
lightgbm                 2.3.0             2.3.1      wheel pip       Anaconda
lxml                     4.5.0             4.5.1      wheel pip       Anaconda
matplotlib               3.1.3             3.2.1      wheel pip       Anaconda
nltk                     3.4.5             3.5        sdist pip       Anaconda
numpy                    1.18.1            1.18.5     wheel           Anaconda
numpydoc                 0.9.2             1.0.0      wheel conda     Anaconda
packaging                20.3              20.4       wheel conda     Anaconda
pandas                   1.0.3             1.0.4      wheel pip       Anaconda
parso                    0.5.2             0.7.0      wheel conda     Anaconda
path                     13.1.0            14.0.1     wheel pip       Anaconda
pathspec                 0.7.0             0.8.0      wheel conda     Anaconda
pip                      20.0.2            20.1.1     wheel           Anaconda
portalocker              1.5.2             1.7.0      wheel pip       Anaconda
prometheus-client        0.7.1             0.8.0      wheel conda     Anaconda
protobuf                 3.11.4            3.12.2     wheel           Anaconda
pyarrow                  0.11.1            0.17.1     wheel           Anaconda
pyasn1-modules           0.2.7             0.2.8      wheel           Anaconda
pycodestyle              2.5.0             2.6.0      wheel           Anaconda
pydocstyle               4.0.1             5.0.2      wheel conda     Anaconda
pyflakes                 2.1.1             2.2.0      wheel pip       Anaconda
pylint                   2.5.2             2.5.3      wheel pip       Anaconda
PyNaCl                   1.3.0             1.4.0      wheel           Anaconda
pyodbc                   4.0.0-unsupported 4.0.30     wheel pip       Anaconda
pytest                   5.4.2             5.4.3      wheel           Anaconda
pytest-cov               2.8.1             2.9.0      wheel conda     Anaconda
pytest-doctestplus       0.5.0             0.7.0      wheel conda     Anaconda
python-language-server   0.31.10           0.33.1     wheel conda     Anaconda
pyzmq                    18.1.1            19.0.1     wheel           Anaconda
QtAwesome                0.7.0             0.7.2      wheel conda     Anaconda
regex                    2020.5.14         2020.6.8   wheel pip       Anaconda
ruamel-yaml              0.15.87           0.16.10    wheel           Anaconda
s3transfer               0.1.13            0.3.3      wheel           Anaconda
scikit-image             0.16.2            0.17.2     wheel pip       Anaconda
scikit-learn             0.22.1            0.23.1     wheel pip       Anaconda
sortedcollections        1.1.2             1.2.1      wheel pip       Anaconda
sortedcontainers         2.1.0             2.2.2      wheel           Anaconda
sphinxcontrib-websupport 1.2.1             1.2.2      wheel conda     Anaconda
sympy                    1.5.1             1.6        wheel pip       Anaconda
tenacity                 6.0.0             6.2.0      wheel pip       Anaconda
tifffile                 0.15.1            2020.6.3   wheel           Anaconda
toml                     0.10.0            0.10.1     wheel pip       Anaconda
tqdm                     4.46.0            4.46.1     wheel conda     Anaconda
typing-extensions        3.7.4.1           3.7.4.2    wheel pip       Anaconda
ujson                    1.35              3.0.0      wheel           Anaconda
urllib3                  1.25.8            1.25.9     wheel pip       Anaconda
wcwidth                  0.1.9             0.2.4      wheel conda     Anaconda
wrapt                    1.11.2            1.12.1     sdist pip       Anaconda
XlsxWriter               1.2.8             1.2.9      wheel conda     Anaconda
yapf                     0.28.0            0.30.0     wheel conda     Anaconda
zope.interface           4.7.1             5.1.0      wheel pip       Anaconda

I have Win-X64, conda version : 4.8.3, conda-build version : 3.18.11, python version : 3.7.7.final.0.
I did modify my local pip 20.0.2, everything works good for me, see above. We can discuss these changes more if it will be considered as new useful feature for "pip list". Old behavior completely unchanged, new feature available only with new option.

[uranusjr]

pip already shows the installer information in the verbose output (pip list -v). It should satisfy your need if Conda provides that information (and you’ll need to ask Conda to provide the info if it doesn’t already do).

[vx1920]

I already wrote that "pip list -v" provide additional column "Installer" with irrelevant information. If you don't believe - open first Details marker in my original post and review first two lines. In my system I have package "adal" installed by pip, while next package "alabaster" installed by conda.
Your "pip list -v" marks both of them as "pip". With my new column I can see "alabaster" marked as "Anaconda", while "adal" marked "ThePip". I decide to combine both columns (old "Installer" and my new "Owner") to illustrate that "Installer" column contains something useless.

[uranusjr]

It’s a conda bug if packages it installs do not report the installer correctly. I don’t think pip should be responsible for fixing things other tools do wrong.

[vx1920]

I am not sure that it is conda bug, but I am sure that conda developers will tell you and me that there is no any bug in conda and conda itself can provide ability to update conda-only packages ("conda update --all"). Also "installer=pip" may legitimately indicate that conda uses "setuptools" (it is general tool used both by pip and conda) to install some particular package. Some other conda tools may set marker as "conda".
Conda has tracking items installed by conda and pip should do the same. If I use "pip list --outdated" - it should inform about outdated packages installed by pip and don't care about items installed by conda.
My option solves this problem somehow by adding extra marker column. Mybe it is better to have special option "pip list --outdatedpip", but with my option I could do some system cleanup by uninstalling pip packages and installing same name conda ones.
Maybe somebody else you can consider my option as useful. Any case: I have it now, I use now and I solve some of my problems.

[pfmoore]

I concur with @uranusjr - pip reports the value in the INSTALLER metadata file. If conda populates that fire correctly, pip will report as you wish. If conda don't do so, they are not respecting the existing interoperability standard, and it's not down to pip to work around that problem.

[redbar0n]

I've encountered the problem where pip list will report the newer version of a package installed by conda (and vice versa, see: conda/conda#9924). When I removed the newer package from conda then pip list will change to suddenly show the older version of that package. Presumably the one that was installed with pip in the first place.

I don't know where in the interaction between pip and conda that the problem lies. But in any case it seems inconsistent of the output of pip list to rely on activities conducted within conda.

[pfmoore]

@redbar0n As has been noted above, the fix here is for conda to follow the existing standards for recording installed packages. If they can't do that (they have their own processes, which may or may not work well alongside the packaging standards) then pip and conda will never be able to interoperate completely seamlessly.

As an aside, conda developers are more than welcome to participate in packaging standards discussions, in the same way that, for example, Linux distro packagers do. So far, we've had very little involvement from them, so it's unfortunate but not exactly unexpected that awkward interactions like this crop up occasionally.