[ENH] - Safer pip integrations

[jaimergp]

Feature description

Right now, we have some kind of "extra" checks around pip requirements via the Specification schema, but other than that, we just delegate to conda env.

The idea here is to handle pip packages a bit more safely, as described in this draft.

Then we would modify this block so we only pass the conda deps to conda env, but we handle pip packages separately, as described in the draft.

Something like:

conda_pkgs, pip_pkgs = pkgs_from_environment_yml()
for pip_pkg in pip_pkgs:
   conda_deps, pip_deps = grayskull(pip_pkg, recursive=True)
   conda_pkgs.extend(conda_deps)
   pip_pkgs.extend(pip_deps)
conda_files = files_from_conda_solve(conda_pkgs)
pip_files = files_from_pip_install(pip_pkgs)
if pip_files.intersect(conda_files):
  # warn, or error
if not shared_objects(pip_files):
  # warn, or error
... # maybe other checks
conda_create(conda_pkgs)
pip_install(pip_pkgs)

Value and/or benefit

Safe pip integrations with reduced risk of clobbering and ABI incompatibilities.

Anything else?

This might start as a separate module, or maybe as a different repository.

[jaimergp]

CLI prototype at https://github.com/jaimergp/conda-pip

You can try it out in your base environment with:

$ conda install -nbase grayskull pip
$ pip install https://github.com/jaimergp/conda-pip/archive/main.tar.gz

And then run commands like:

$ conda pip install scipy  # should tell you it will get it from conda
$ conda pip install ragna
Analyzing dependencies: done
conda will install:
 - aiofiles
 - emoji
 - fastapi
 - huey
 - httpx
 - importlib-metadata[version='>=4.6']
 - packaging
 - panel[version='>=1.3,<1.4']
 - pydantic[version='>=2']
 - pydantic-core
 - pydantic-settings[version='>=2']
 - pyjwt
 - python-multipart
 - redis-py
 - questionary
 - rich
 - sqlalchemy[version='>=2']
 - tomlkit
 - typer
 - uvicorn
pip will install:
 - ragna==0.1.1
Proceed ([y]/n)? 
# will run 'conda install' and 'pip install --no-deps', respectively

[dharhas]

This is very promising. I like the direction. One longer term issue will be related to the UI. Whether we make pip packages easier to select and how we communicate what has been done under the hood. i.e. should we optionally spit out a new env.yaml with the packages moved to the conda section.

[trallard]

Perhaps we will need a new yaml for visibility.
If that were the case we should probably add some auto generated header too.

[nkaretnikov]

@jaimergp What's the status here? I see this was marked as Ready before.

[jaimergp]

See jaimergp/conda-pip. Still not ready for integration in conda-store, but hoping to make more progress during December.

[jaimergp]

Current status:

• I identified some issues with the grayskull based solver and implemented another based on pip. Planning to add another based on uv.
• I also added some very basic tests to ensure some things do work. Still, this tool would benefit from a lot of testing on real case scenarios.
• I also opened conda_env installers / providers plugin hooks conda/conda#13833 to discuss how to better integrate this tool with conda env so the safe pip is used instead of the regular pip when installing environment.yml files with - pip: sections.

I guess some documentation could be beneficial here, which I haven't added yet. I plan to do so after getting some consensus on the name of the project (see related issue), but I can do it already if someone on the team wants to test this locally?

[jaimergp]

Renamed to conda-pypi (conda-incubator/conda-pypi#18) and added documentation (conda-incubator/conda-pypi#19).

Useful reads (this is not yet on GH Pages):

• Quickstart
• Motivation
• Contribution guidelines

[trallard]

Will close this as it is completed for the purpose of Challenges. And the project has now been moved to https://github.com/conda-incubator/conda-pypi

[dharhas]

Is there another issue tracking moving towards using conda-pypi in conda-store to enable safer pip integration?

[trallard]

No there is not. I suppose @jaimergp @peytondmurray and I could discuss how this would look like and how we'd go about it.

[jaimergp]

Opened #845