Breaking change in Warehouse breaks poetry lock file creation #5970

cyraxjoe · 2022-07-08T08:07:33Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Linux/Debian bookworm:
1.2.0b2:
Sample pyproject:

Issue

The warehouse project (backend of pypi.org) applied a breaking change that affects poetry lock the releases key is no longer present in the responses, required for the pypi_repository implementation.

It can be easily verified by removing the local cache and inspecting the generated lock file, it would have an empty metadata.files property, something like:

[[package]]
name = "beautifulsoup4"
version = "4.11.1"
description = "Screen-scraping library"
category = "main"
optional = false
python-versions = ">=3.6.0"

[package.dependencies]
soupsieve = ">1.2"

[package.extras]
html5lib = ["html5lib"]
lxml = ["lxml"]

[[package]]
name = "soupsieve"
version = "2.3.2.post1"
description = "A modern CSS selector implementation for Beautiful Soup."
category = "main"
optional = false
python-versions = ">=3.6"

[metadata]
lock-version = "1.1"
python-versions = "^3.10"
content-hash = "36fc53f6339e375f6295378c7a8ecea11e3a04ba0adcbfa7aeb812e30500aad0"

[metadata.files]
beautifulsoup4 = []
soupsieve = []

The text was updated successfully, but these errors were encountered:

Kurt-von-Laven · 2022-07-08T09:50:28Z

I can confirm that this issue is also affecting Poetry 1.1.13 as it is taking down all of our builds.

novotl · 2022-07-08T10:02:14Z

👍 We're seeing the same with Poetry 1.1.8

dimbleby · 2022-07-08T10:11:20Z

I suspect a lot of people are about to learn a hard lesson about the dangers of relying on public repositories....

diff --git a/src/poetry/repositories/pypi_repository.py b/src/poetry/repositories/pypi_repository.py
index 0e5beab6..142b29d8 100644
--- a/src/poetry/repositories/pypi_repository.py
+++ b/src/poetry/repositories/pypi_repository.py
@@ -204,7 +204,7 @@ class PyPiRepository(HTTPRepository):
         )

         try:
-            version_info = json_data["releases"][version]
+            version_info = json_data["urls"]
         except KeyError:
             version_info = []

looks like it should fix things on master.

mkniewallner · 2022-07-08T10:16:43Z

#5972 draft for now, because there may be other places where releases is used, don't know the codebase well enough.
Tested locally, and it seems to work fine.

Also backported the change for 1.1 in https://github.com/mkniewallner/poetry/commits/fix/fix-releases-key-pypi-repository-1.1

finswimmer · 2022-07-08T10:17:11Z

Thanks a lot everyone for reporting, debugging and starting a PR 👍 Will have a look at your PR @mkniewallner

python-poetry/poetry#5970

Resolves #6076 I've taken the JSON version of the simple API and converted it into a `LinkSource` so that the package-finding logic in the `PyPiRepository` is very similar to - but annoyingly not quite the same as! - the `LegacyRepository`. I've also taken the opportunity to refactor the `LegacyRepository` ever so slightly to emphasise that similarity. I think I've probably fixed a small bug re caching and pre-releases: previously the processing for ignored pre-releases was skipped when reading from the cache. I believe this change will tend to be a modest performance hit. Eg consider a package like `cryptography`, for which there are maybe a couple of dozen downloads available at each release: to get the available versions we now have to iterate over each of those files and parse their names, rather than simply reading the answer. However if the API that poetry currently uses is truly deprecated I see little choice but to suck that up - or risk being in an awkward spot when it is turned off. cf #5970, but worse. Most of the changes are in the test fixtures: - unversioned fixtures were generated from the existing fixtures: I didn't want to download fresh data and start getting different answers than the tests were expecting - new versioned fixtures were downloaded fresh

Resolves #6076 I've taken the JSON version of the simple API and converted it into a `LinkSource` so that the package-finding logic in the `PyPiRepository` is very similar to - but annoyingly not quite the same as! - the `LegacyRepository`. I've also taken the opportunity to refactor the `LegacyRepository` ever so slightly to emphasise that similarity. I think I've probably fixed a small bug re caching and pre-releases: previously the processing for ignored pre-releases was skipped when reading from the cache. I believe this change will tend to be a modest performance hit. Eg consider a package like `cryptography`, for which there are maybe a couple of dozen downloads available at each release: to get the available versions we now have to iterate over each of those files and parse their names, rather than simply reading the answer. However if the API that poetry currently uses is truly deprecated I see little choice but to suck that up - or risk being in an awkward spot when it is turned off. cf #5970, but worse. Most of the changes are in the test fixtures: - unversioned fixtures were generated from the existing fixtures: I didn't want to download fresh data and start getting different answers than the tests were expecting - new versioned fixtures were downloaded fresh (cherry picked from commit b61a4dd)

Natim · 2023-01-04T13:46:54Z

It seems that Poetry 1.2.2 fixed it but 1.3.1 is back with the same issue, any idea why?
However this code is still the one provided with it.

finswimmer · 2023-01-04T13:49:39Z

@Natim: Please open a new issue with steps to reproduce if you encounter any problems (I don't have any).

Kurt-von-Laven · 2023-01-10T07:47:50Z

We haven't encountered this issue on Poetry 1.3.1 either.

Natim · 2023-01-10T08:24:52Z

Sorry, you are both right, the issue is not related, the poetry.lock file structure changed in between 1.2.2 and 1.3.1 which made disappear the metadata section, but the hashes are correctly computed in the package file section in 1.3.1.

github-actions · 2024-02-29T21:06:34Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

cyraxjoe added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Jul 8, 2022

mkniewallner mentioned this issue Jul 8, 2022

fix(repositories): use urls for versioned JSON API instead of releases #5972

Merged

2 tasks

finswimmer closed this as completed Jul 8, 2022

finswimmer reopened this Jul 8, 2022

EcmaXp mentioned this issue Jul 8, 2022

Don't render all releases on version specific JSON pages pypi/warehouse#11775

Merged

finswimmer closed this as completed in #5972 Jul 8, 2022

mkniewallner removed the status/triage This issue needs to be triaged label Jul 9, 2022

radoering mentioned this issue Jul 9, 2022

improve performance of find_similar_names python-poetry/cleo#173

Merged

This was referenced Jul 14, 2022

export fails to include hashes of some requirements python-poetry/poetry-plugin-export#90

Closed

Exported requirements.txt fails to install #6041

Closed

Missing wheels (metadata.files all empty lists) #6045

Closed

jstriebel mentioned this issue Jul 21, 2022

upgrade wkw to v1.1.16, poetry to 1.1.14 scalableminds/webknossos-libs#769

Merged

aryarm added a commit to CAST-genomics/haptools that referenced this issue Jul 21, 2022

update poetry in response to a breaking change in the pypi API

6dded6b

python-poetry/poetry#5970

This was referenced Jul 26, 2022

Avoid the deprecated JSON API #6081

Merged

Cannot Install tensorflow 2.9.1 #6079

Closed

radoering mentioned this issue Aug 2, 2022

Difference vs. pip: installing cyptography #6102

Closed

dimaqq mentioned this issue Aug 3, 2022

cffi is missing when cryptography is installed using poetry pyca/cryptography#7467

Closed

dimbleby mentioned this issue Aug 19, 2022

poetry removes all wheels sometimes... randomly #6199

Closed

glennmatthews mentioned this issue Oct 25, 2022

Add toml to dev dependencies. The new GH setup only install dev dependencies networktocode/circuit-maintenance-parser#182

Merged

github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Breaking change in Warehouse breaks poetry lock file creation #5970

Breaking change in Warehouse breaks poetry lock file creation #5970

cyraxjoe commented Jul 8, 2022

Kurt-von-Laven commented Jul 8, 2022

novotl commented Jul 8, 2022

dimbleby commented Jul 8, 2022

mkniewallner commented Jul 8, 2022

finswimmer commented Jul 8, 2022

Natim commented Jan 4, 2023 •

edited

finswimmer commented Jan 4, 2023

Kurt-von-Laven commented Jan 10, 2023

Natim commented Jan 10, 2023 •

edited

github-actions bot commented Feb 29, 2024

Breaking change in Warehouse breaks poetry lock file creation #5970

Breaking change in Warehouse breaks poetry lock file creation #5970

Comments

cyraxjoe commented Jul 8, 2022

Issue

Kurt-von-Laven commented Jul 8, 2022

novotl commented Jul 8, 2022

dimbleby commented Jul 8, 2022

mkniewallner commented Jul 8, 2022

finswimmer commented Jul 8, 2022

Natim commented Jan 4, 2023 • edited

finswimmer commented Jan 4, 2023

Kurt-von-Laven commented Jan 10, 2023

Natim commented Jan 10, 2023 • edited

github-actions bot commented Feb 29, 2024

Natim commented Jan 4, 2023 •

edited

Natim commented Jan 10, 2023 •

edited