Poetry overwrites existing conda packages even if versions match

[dpp23]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name:
• Poetry version: 1.2.0
• Link of a Gist with the contents of your pyproject.toml file:

Issue

If running inside a conda environment, poetry 1.1.* is happy to use the conda packages if their versions matches, however this is not the case with poetry 1.2 which re-installs them.

This is problematic in the case of packages that are not easy to install from pip on Mac M1, like lightgbm

As you can see in the reproduction steps, poetry says Updating lightgbm (3.3.2 /Users/runner/miniforge3/conda-bld/lightgbm_1641600116991/work -> 3.3.2) so the version patches, but poetry decides to re-install it anyway.

reproduction steps:

conda create -y env foo
conda activate foo

conda install -y lightgbm=3.3.2

poetry install --dry-run
Installing dependencies from lock file

Package operations: 0 installs, 6 updates, 0 removals, 1 skipped

  • Updating numpy (1.23.2 /Users/runner/miniforge3/conda-bld/numpy_1660673193545/work -> 1.23.2)
  • Updating numpy (1.23.2 /Users/runner/miniforge3/conda-bld/numpy_1660673193545/work -> 1.23.2)
  • Updating joblib (1.1.0 /home/conda/feedstock_root/build_artifacts/joblib_1633637554808/work -> 1.1.0)
  • Updating joblib (1.1.0 /home/conda/feedstock_root/build_artifacts/joblib_1633637554808/work -> 1.1.0)
  • Updating scipy (1.9.1 -> 1.6.1)
  • Updating scipy (1.9.1 -> 1.6.1)
  • Updating threadpoolctl (3.1.0 /home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work -> 3.1.0)
  • Updating threadpoolctl (3.1.0 /home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work -> 3.1.0)
  • Updating scikit-learn (1.1.2 /Users/runner/miniforge3/conda-bld/scikit-learn_1659726153807/work -> 1.1.2)
  • Updating scikit-learn (1.1.2 /Users/runner/miniforge3/conda-bld/scikit-learn_1659726153807/work -> 1.1.2)
  • Updating lightgbm (3.3.2 /Users/runner/miniforge3/conda-bld/lightgbm_1641600116991/work -> 3.3.2)

  // pyproject.toml

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["foo <foo@bar.co>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
lightgbm = "3.3.2"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[neersighted]

This is likely because Conda does not conform to PEP 610 and write the origin URL to the package install directory. Poetry 1.2 is PEP 610 compliant and will not consider a package to be equivalent unless it comes from the same source (as determined by the PEP 610 record).

[dpp23]

@neersighted thank you for explaining! Do you think there are any workarounds for this or will this workflow remain unsupported?

[neersighted]

I don't think there are any workarounds -- Poetry expects to be the sole manager of the environment and will act like it. From a practical standpoint, Poetry is correct in the packages not being 100% the same as defined by your lockfile, as well...

Is there a reason your workflow requires mixing packages installed by Conda and Poetry? In the long run if Conda supports PEP 610 (and uses the same origins as Poetry) this will become 'supported' again, but I don't think it will ever by anything but happy chance, unless there's a compelling reason many users would benefit from this mixed workflow in Poetry itself.

[dimbleby]

I don't think poetry would write or expect to see a direct_url.json for a package that it installs from pypi.

I think the explanation in this thread is likely essentially right but it's sort of the opposite : conda does write direct_url.json saying something about the package, as of 1.2 poetry understands this to mean that it's not the same package as is requested (either directly or indirectly) by pyproject.toml, and so poetry updates.

[dpp23]

The reason we have this setup is that on M1 Macs some packages are not available on PyPi and the only way to install them is via conda. In our case, we use poetry to solely manage the whole environment on Linux, but for the folks that have an M1 machine, we were using conda + poetry as a workaround.

[neersighted]

Oh, fair point -- I forget that we don't write direct_url.json for repositories. In that case, I don't think there's really anything we can do from the Poetry end, even if this was desirable, unless we added some sort of installer.ignore-pep610 option (that would be another footgun along the lines of virtualenvs.create, in my opinion).

[neersighted]

@dpp23 Depending on how Conda authentication works (if it's even necessary for these packages), you could actually use a platform constraint for a direct URL dependency on Macs, using the URL taken from the conda-written direct_url.json -- in that case, in theory Poetry would do nothing when it encountered a Conda-installed wheel.

[dimbleby]

I experimented in the docker container continuumio/miniconda3, it looks as though the direct_url.json files that Conda writes are artifacts of its build process rather than anything useful

eg

{"dir_info": {}, "url": "file:///tmp/build/80754af9/lightgbm_1633019183505/work"}

I'm not familiar with the conda ecosystem at all but this feels like a bug at their end, delivering these files is surely an accident

Maybe you can try reporting this to them

[neersighted]

Interesting -- looks like those are artifacts of their build backend and this is a Conda issue after all. I'm going to close this issue since it's now clear that if Conda didn't deliver these files, things would work as @dpp23 expects. That being said, feel free to report back here if Conda resolves the junk artifact issue, for posterity's sake.

[dimbleby]

conda/conda#11580 (comment)

[dpp23]

Thank you @neersighted and @dimbleby for the quick responses! I've resolves this for now with a find <conda-env-path> -delete -name direct_url.json and will follow up on the conda issue

[lancechua]

Thank you @neersighted and @dimbleby for the quick responses! I've resolves this for now with a find <conda-env-path> -delete -name direct_url.json and will follow up on the conda issue

For others who wish to try this fix, put the -delete flag AFTER -name direct_url.json.
The command as is will wipe the whole conda-env-path.

The correct command is find <conda-env-path> -name direct_url.json -delete.

[geiszla]

This is likely because Conda does not conform to PEP 610 and write the origin URL to the package install directory. Poetry 1.2 is PEP 610 compliant and will not consider a package to be equivalent unless it comes from the same source (as determined by the PEP 610 record).

We're having the same problem. As I understand, Poetry considers conda package to have different version than the one it finds on PyPI. However, how does it determine if the current version needs to be upgraded? Would there be a way to tell Poetry that the conda version is higher than the PyPI one?

[ashanbrown]

I'm seeing an issue that I think might be related to this and I'd welcome some other perspectives. I'm seeing intermittent CI issues that seem to be due to missing files when reinstalling packages previously installed by conda. I don't see this issue when I use the workaround of removing all the direct_url.json files.

  • Updating virtualenv (20.24.2 /home/conda/feedstock_root/build_artifacts/virtualenv_1690273226872/work -> 20.24.2)
  • Installing xmltodict (0.13.0)
  FileNotFoundError
  [Errno 2] No such file or directory: '/opt/conda/envs/caliban-env/lib/python3.10/site-packages/virtualenv/create/via_global_ref/_virtualenv.py'
  at /opt/conda/envs/caliban-env/lib/python3.10/pathlib.py:1119 in open
      1115│         the built-in open() function does.
      1116│         """
      1117│         if "b" not in mode:
      1118│             encoding = io.text_encoding(encoding)
    → 1119│         return self._accessor.open(self, mode, buffering, encoding, errors,
      1120│                                    newline)
      1121│ 
      1122│     def read_bytes(self):
      1123│         """

My understanding is that poetry should be able to install packages concurrently, but is it possible that it doesn't realize that it needs to serialize these reinstalls, perhaps because it thinks it already has the correct version installed?

[pokapok-alban]

I just encountered the same issue...

[conda_stage 13/16] RUN poetry install --no-cache
#37 2.349 Installing dependencies from lock file
#37 2.349 
#37 2.349 Package operations: 23 installs, 2 updates, 0 removals
#37 2.349 
#37 2.349   • Updating numpy (1.25.2 /home/conda/feedstock_root/build_artifacts/numpy_1691056235090/work -> 1.25.2)
#37 2.349   • Installing six (1.16.0)
#37 2.349 
#37 2.349   CalledProcessError
#37 2.349 
#37 2.349   Command '['/home/serviceuser/.conda/envs/env-develop/bin/python', '-m', 'pip', 'uninstall', 'numpy', '-y']' returned non-zero exit status 2.

#37 2.349   OSError: [Errno 18] Invalid cross-device link: '/home/serviceuser/.conda/envs/env-develop/lib/python3.11/site-packages/numpy-1.25.2.dist-info/' -> '/home/serviceuser/.conda/envs/env-develop/lib/python3.11/site-packages/~umpy-1.25.2.dist-info'

The fact is removing the direct_url.json from my conda solved the poetry install hang.

here are the versions I use :

• python : 3.11.4
• pip : 32.2.1
• conda : 23.5.2
• poetry : 1.6.1

I have different execution environments, some are on linux ubuntu 22.04 laptops / desktops, macOS laptops, all using docker.

What I do not understand is that the previous version ( before removing direct_urls .. ) of my python app dockerfile ( ubuntu 22.04 image + conda install )

-> perfectly worked - on ubuntu desktop or server, it also worked on ubuntu VMs under proxmox,
-> It did not work under Proxmox LXC / Docker nesting ( tested with docker.io ubuntu release : Docker version 20.10.21 and Docker-ce : Docker version 24.0.5 ). I was struggling with other problems related to LXC / Docker nesting ( privileged container, apparmor, and so on.. ). But now I realize It had nothing to do with.

What other reason for docker build not being reproducible ? What the difference in building docker between 22.04 lxc image and 22.04 iso image ?

By the way, thank you, this thread saved my day !

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.