Don't render all releases on version specific JSON pages

[dstufft]

• Reverts the streaming JSON, it didn't work and complicates the code/tests
• Version specific legacy JSON pages only have information for that version now.
• Non version specific legacy JSON pages still have information on all releases.

This is a breaking change, but it's a needed one if we're going to keep PyPI stable.

[ewdurbin]

probably want to update https://github.com/pypi/warehouse/blob/main/docs/api-reference/json.rst with the new form and a note on the breaking change?

otherwise, yes. its time.

[EcmaXp]

With this change, I think the poetry lock process is now failing: python-poetry/poetry#5970

I checked the metrics in PyPI Public Dashboard, and it makes sense that this change had to be merged quickly.

However, I still think that there is a way to revert the previous PR or rollback to the previous version rather than making a breaking change in the API.

It would be nice to have a way to notify the package manager maintainer if there is a case where API breaking changes are made in the future.

Thanks for your contribution. I'm glad that PyPI is stable for now. 🎉

[mkniewallner]

If the change was to be reverted, on Poetry side we would probably only need to bring back the version requested in /pypi/<project_name>/<version>/json endpoint in releases key, because that's exactly what we did before this change.

So for instance, if we query /pypi/requests/2.28.1/json, we would only need 2.28.1 in releases, not the other versions.

I understand that a revert (even a partial one as suggested that would add back only the requested version) is cumbersome and a pretty big request (especially since this PR is about PyPI stability, which is obviously important), but just to bring awareness about the impact, basically all released versions of Poetry (as of writing this) are affected, preventing the retrieval of hashes when locking dependencies, since they rely on releases key, so the impact is quite high for Poetry end users.

We are in the process of releasing an express new Poetry 1.1 version that relies on urls key instead of releases, so people will have a workaround, but people with older versions will have no other choice than updating, which can be quite disruptive.

[dstufft]

I went back and forth on whether to leave the releases key there with just the current version in it or not.

I ultimately decided not to for one major, and one minor reason.

The minor reason is just that the data already exists on the response in the urls key, so it was duplicative.

The major reason though is that I think breaking loudly is better than breaking quietly, and while poetry's specific use here wouldn't be affected, it's just as easy to imagine someone iterating over the list of releases rather than looking for a specific release. If they did that, then they would get silently wrong data instead of breaking loudly.

[mkniewallner]

The major reason though is that I think breaking loudly is better than breaking quietly, and while poetry's specific use here wouldn't be affected, it's just as easy to imagine someone iterating over the list of releases rather than looking for a specific release. If they did that, then they would get silently wrong data instead of breaking loudly.

This is perfectly understandable. I've only recently started contributing more and more to Poetry (so disclaimer, not a core contributor), so I lack a bit of context to know if it should/could have used urls from the beginning, but on a broader level, it's a bit frightening that Poetry relies on an API that could break (even if the reasons to break things are perfectly valid, so I'm mostly reasoning from a Poetry end user and maintainer point of view).

Was there a way to know beforehand about the removal of releases in the API? Maybe in a mailing list for instance?

I've noticed #11559 following another issue that recently affected Poetry, and I agree that it would be awesome to document a bit more what to expect from the different APIs in terms of stability. Having deprecation notices, even if short ones, would at least make it possible to update the API requests on Poetry side, which would be a better experience for Poetry users since it would give them time to upgrade rather than being forced to.

[richaagarwal]

Hi there - thanks for the context on this change. I'm curious if/when the releases key will be removed on the non-versioned json API endpoint as well. I noticed that the documentation was updated to include a note that it may be deprecated in the future; any timeline you are able to share with me would be helpful!

[dstufft]

As far as we can tell, there's not currently any major problems stemming from that specific key since it's one extra response to cache (versus an unbounded number of extra responses to cache when it was on the versioned URL). Given that, there's currently not any specific plan to remove the releases key from the non-versioned URL, so the answer to that is basically "nobody knows, it might never be removed, or it might be in the future if it starts to cause problems because it makes the responses huge".

That warning was put there mostly to guide people to attempt to find an alternative solution if one exists to relying on that key OR if an alternative solution doesn't exist, to follow up on the issue tracker or on discuss.python.org with their use cases that aren't handled currently by the other keys in the JSON API or, even better, by the simple repository API so we can figure out if there's a better, less problematic, answer we can add to support them.