conda-lock 1.4.1 fails to generate lock file from pyproject.toml with mixed conda-forge/PyPI dependencies

[cameronraysmith]

This issue is related to

• Solver can fail with mixed pip/conda dependencies conda/conda-lock#179
• Hanging on my Local Environment when including PyPi dependencies conda/conda-lock#315

Conda-lock configuration is specified near

pyrovelocity/pyproject.toml

Line 87 in 42c2475

[tool.conda-lock]

PyPI dependencies are marked explicitly as in

pyrovelocity/pyproject.toml

Line 43 in 42c2475

cospar = { version = "0.1.9", source = "pypi" }

A command similar to

conda-lock \
--conda mamba \
--extras dev \
--filter-extras \
--no-dev-dependencies \
--virtual-package-spec conda/virtual-packages.yml \
--log-level DEBUG \
-f pyproject.toml \
-p linux-64

from the repo root with conda-lock version 1.4.0 fails to produce a valid conda lock file due to failure to resolve PyPI dependencies whose name contains a hyphen vs underscore.

The specific error is related to the translation of package names between conda and PyPI with matplotlib_base vs matplotlib-base vs matplotlib

conda-lock stderr

Traceback (most recent call last):
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 488, in seperator_munge_get
    return d[key]
KeyError: 'matplotlib-base'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 491, in seperator_munge_get
    return d[key.replace("-", "_")]
KeyError: 'matplotlib_base'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".local/bin/conda-lock", line 8, in <module>
    sys.exit(main())
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 1353, in lock
    lock_func(
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 1083, in run_lock
    make_lock_files(
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 408, in make_lock_files
    lock_content = lock_content | create_lockfile_from_spec(
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 801, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 737, in _solve_for_arch
    pip_deps = solve_pypi(
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/pypi_solver.py", line 327, in solve_pypi
    src_parser._apply_categories(requested=pip_specs, planned=planned)
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 502, in _apply_categories
    for dep in seperator_munge_get(planned, item).dependencies
  File ".local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 493, in seperator_munge_get
    return d[key.replace("_", "-")]
KeyError: 'matplotlib-base'

The approach described here would likely work, but since it would still require maintenance of two separate lists of dependencies it is not necessarily better than manually synchronizing pyproject.toml with environment.yml.

[cameronraysmith]

fixed in conda/conda-lock#290

pipx install git+https://github.com/conda/conda-lock.git

installs conda-lock, version 1.4.1.dev119+g4a667bd on 5/20/2023.

conda-lock \
--conda mamba \
--filter-extras \
--no-dev-dependencies \
--virtual-package-spec conda/virtual-packages.yml \
--log-level INFO \
-f pyproject.toml \
-p linux-64

succeeds with appropriate annotation in pyproject.toml of packages only available on PyPI, e.g.

seaborn = { version = "0.11.2", source = "pypi" }

and the resulting conda-lock.yml is installable with

conda-lock install -n [ENVIRONMENT NAME] --file conda-lock.yml

[qinqian]

can we close this issue? @cameronraysmith

[cameronraysmith]

reopen this when switching to conda-lock