Default PyPI dependencies for pyproject.toml

[pep-sanwer]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

My understanding is that as pyproject.toml (Poetry flavor) support currently stands, all dependencies default to conda channels unless explicitly stated as source = "pypi" for that particular dependency. From what I understand, this includes dependencies listed in [tool.poetry.dependencies] as well as [tool.conda-lock.dependencies].

The feature suggestion here is change this default behavior, so that dependencies listed in [tool.poetry.dependencies] default to PyPI, i.e. source = "pypi" without having to explicitly state it in pyproject.toml, while [tool.conda-lock.dependencies] behavior can remain the same, i.e. defaulting to conda channels unless explicitly stated otherwise in pyproject.toml.

This would enable workflows where you can conda-lock to solve an environment, conda-lock install to install a reproducible environment, and then poetry add pandas (for example) to add additional pyPI dependencies as the project evolves. As it currently stands, one would have to invoke poetry add pandas --source="pypi" (for example) per additional pyPI dependency.

Currently, the workflow I have landed upon is to lock and reproduce the environment with:

• conda-lock --conda mamba -f pyproject.toml
• conda-lock install --conda mamba --name test_env conda-lock.yml
• then an additional poetry install to update all pyPI packages from conda channels to their latest pyPI
• my concern is that that the previous step may cause package issues (I've been burned in the past mixing conda and pip installs in the same project env)

If I am misunderstanding how additional dependencies should be added to a project using conda-lock & pyproject.toml or how conda-lock should be utilized with Poetry, please correct me.

An additional question would also be, does conda-lock default to conda-lock --conda mamba if the only conda installed is from mamba-forge?

Appreciative of this wonderful tool!

Why is this needed?

No response

What should happen?

No response

Additional Context

No response

[maresb]

Thanks a lot @pep-sanwer for the thoughtful comments! I like your suggestion, and your explanation of your usage. To put it in my own words:

• PyPI dependencies should go in [tool.poetry.dependencies].
• Conda dependencies should go in [tool.conda-lock.dependencies].

But I do agree that favoring PyPI packages could lead to messy problems. In my typical workflow, I want to get as many packages from conda-forge as possible, and only then would I fall back to PyPI for those which are missing. So from that perspective I prefer the existing logic. (My biggest pain point ATM is that implicit dependencies aren't converted to conda-forge, but that's another issue...)

Another thing that bothers me is how we are giving Poetry preferential treatment, especially because it is so temperamental, and there are other great alternatives available. In particular, the source = "pypi" key-value pair is not standards-compliant, so unless I'm mistaken it can only really be used with Poetry. I'd really like to see this somehow eliminated.

In any case, it might be interesting to add a flag to [tool.conda-lock] to make [tool.poetry.dependencies] (and even better the standards-compliant [project.dependencies]) PyPI-only.

[pep-sanwer]

Thank you @maresb for taking the time to look into this.
I agree with you completely on keeping the current behavior - while exposing a flag parameter in [tool.conda-lock] that would allow the user to specify default source behavior for [tool.poetry.dependencies].

To your point on favoring Poetry - I agree as well. Hopefully we are moving towards a more standards compliant [project.dependencies] specification as an ecosystem. Until we get there, conda-lock and poetry have been a godsend.