Add / remove packages commands

[AlbertDeFusco]

This PR provides interactive dependency management. One use case enabled here is to initialize an empty project and then add dependencies as needed. With each add/remove command the full dependencies are re-locked from scratch and the live environment is replaced with new locked dependencies.

> conda project init
> conda project add -c defaults python=3.10
> conda project add conda-forge::pandas requests
> conda project add pandas<2
> conda project remove requests

Pip packages are specified with the prefix @pip::. The @pip:: prefix can also be used in the init command when the project is initialized.

> conda project init
> conda project add -c defaults python=3.10 @pip::pydantic
> conda project add @pip::pydantic[email]<2
> conda project remove @pip::pydantic

TODO:

• Explore support for [ENH] Option to re-lock a project with the smallest number of changes #11
• Support add/remove pip depdendencies

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (49aeb59) 97.75% compared to head (6592247) 98.07%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #112      +/-   ##
==========================================
+ Coverage   97.75%   98.07%   +0.31%     
==========================================
  Files           9        9              
  Lines         846      985     +139     
==========================================
+ Hits          827      966     +139     
  Misses         19       19              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[AlbertDeFusco]

Something to consider here: Right now running conda project init without a list of packages will still run the lock. Perhaps creating a lockfile with no dependencies is not strictly necessary

[AlbertDeFusco]

I notice that https://anaconda.org/pypi indicates that it is an index mirror of pypi perhaps we can allow pip packages to be added on the CLI with the prefix pypi::. This would not use anaconda.org for anything, the packages would be added to the pip key of the environment.yml file.

[mattkram]

Something to consider here: Right now running conda project init without a list of packages will still run the lock. Perhaps creating a lockfile with no dependencies is not strictly necessary

I've had the same thought. We should just dump a set of templated files at that point and skip the lock.

Maybe there's a more general feature like "if there are no packages specified for any environments, skip the lock". I'm not sure why that would ever be the case after init but I suppose it is possible if you cleared out your environment files to "restart".

[AlbertDeFusco]

Tests are failing for Conda versions 4.9 and 4.10 and I think we may just end up dropping support for those versions in conda-project.

Here's how to reproduce. The problem appears to be limited to scenarios where these old versions of conda attempt to install environments with Python 3.10 or newer.

It may be that the envs are created correctly, but conda list incorrectly reports the package manager.

> conda create -p ./old-conda python=3.8 conda=4.10

> ./old-conda/bin/conda create python=3.10 -p ./env
The following NEW packages will be INSTALLED:

  bzip2              pkgs/main/osx-arm64::bzip2-1.0.8-h620ffc9_4
  ca-certificates    pkgs/main/osx-arm64::ca-certificates-2023.05.30-hca03da5_0
  libffi             pkgs/main/osx-arm64::libffi-3.4.4-hca03da5_0
  ncurses            pkgs/main/osx-arm64::ncurses-6.4-h313beb8_0
  openssl            pkgs/main/osx-arm64::openssl-3.0.9-h1a28f6b_0
  pip                pkgs/main/osx-arm64::pip-23.1.2-py310hca03da5_0
  python             pkgs/main/osx-arm64::python-3.10.12-hb885b13_0
  readline           pkgs/main/osx-arm64::readline-8.2-h1a28f6b_0
  setuptools         pkgs/main/osx-arm64::setuptools-67.8.0-py310hca03da5_0
  sqlite             pkgs/main/osx-arm64::sqlite-3.41.2-h80987f9_0
  tk                 pkgs/main/osx-arm64::tk-8.6.12-hb8d0fd4_0
  tzdata             pkgs/main/noarch::tzdata-2023c-h04d1e81_0
  wheel              pkgs/main/osx-arm64::wheel-0.38.4-py310hca03da5_0
  xz                 pkgs/main/osx-arm64::xz-5.4.2-h80987f9_0
  zlib               pkgs/main/osx-arm64::zlib-1.2.13-h5a0b063_0

> ./old-conda/bin/conda list -p ./env
# packages in environment at ./env:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h620ffc9_4  
ca-certificates           2023.05.30           hca03da5_0  
libffi                    3.4.4                hca03da5_0  
ncurses                   6.4                  h313beb8_0  
openssl                   3.0.9                h1a28f6b_0  
pip                       23.1.2                   pypi_0    pypi
python                    3.10.12              hb885b13_0  
readline                  8.2                  h1a28f6b_0  
setuptools                67.8.0                   pypi_0    pypi
sqlite                    3.41.2               h80987f9_0  
tk                        8.6.12               hb8d0fd4_0  
tzdata                    2023c                h04d1e81_0  
wheel                     0.38.4                   pypi_0    pypi
xz                        5.4.2                h80987f9_0  
zlib                      1.2.13               h5a0b063_0  

[AlbertDeFusco]

The current implementation destroys the existing conda env and re-installs from the updated lock. Now that conda env update --prune has been fixed for the classic solver we might find an opportunity to use this for installs with forthcoming versions of conda

conda/conda#9614

[bkreider]

pypi:: is overloaded here. It is also a valid conda channel. But I think that might be ok as long there is a warning or a callout. Because conda install pypi::.... does not use pip. On a.org, luckily that channel is empty because it serves up pypi packages at pypi.anaconda.org not conda.anaconda.org.

What does it do if there is a valid conda channel that overlaps with the prefix name? I think it is totally fine if the answer is "don't name channels pypi or r".

[AlbertDeFusco]

Thanks, @bkreider . I can add a warning that pypi:: means pip install. Right now conda-project will always assume pypi:: means pip install even if your package server contains a channel called pypi with real conda packages. That means that -c pypi and pypi:: do different things.

If this is confusing do you think there might be a better way to express it for the CLI? I was trying to go for a single add/remove/init command to get both conda and pypi packages.

[bkreider]

The only thing i can think of to differentiate -c pypi (I think Anaconda should never suggest -c) or conda install pypi::<package> would be to change the delimiter from ::. I know that is ugly but it would be explicit. I don't know what you'd choose though. (@pypi:: might work though and give you a way to easily differentiate additional package managers)

• pypi=>pkg: Is very annoying to quote on the CLI
• pypi:=: is ugly and subtle
• pypi##: is ugly but obvious
• pypi=: is nicer, but looks like a version declaration
• @pypi:: This might work and cannot be used as a conda command but shows a similarity to the conda command?

Or just keep the current method and warn/document?

[AlbertDeFusco]

I'll go with @pypi:: @pip::. Thanks!