Document data format for upload API

[brainwane]

Followup to #2666 (comment) . As @di said,

Metadata is specified at https://packaging.python.org/specifications/core-metadata/ but the conversion between this and "what fields PyPI accepts" is not documented. Perhaps we need more information here: https://warehouse.readthedocs.io/api-reference/legacy/#upload-api

[alanbato]

Working on it 👍

[brainwane]

Thanks! This'll especially be helpful in light of Dustin's work on PEP 566.

[alanbato]

While discussing the proposed additions to the documentation with @brainwane, I came up with some questions that maybe others can help answer.

I'm not sure if all the fields defined in the Core Metadata spec and inside forklift relevant. Like the dependency and file information (digests), for example.

Also, regarding dependecy information, in the form there are "legacy dependency information" and "newer dependency information" fields, which ones should appear in these docs?
Since we're talking about Legacy API Docs I'm assuming the former, but I'm not sure.

Any opinions, ideas or directives on how to proceed are welcome :)

[brainwane]

@alanbato We talked a little in IRC and now I'm not sure whether you have a WIP to share?

I'm sorry that you're stuck on this -- last week @di was working on another project, but this week he returns to concentrating on Warehouse. Do you think you could join us at one of our livechats to talk about it synchronously with him?

[di]

I'm not sure if all the fields defined in the Core Metadata spec and inside forklift relevant. Like the dependency and file information (digests), for example.

The dependency information is definitely relevant (when it is able to be provided). The digests are optional (Warehouse can and will compute them itself) but when provided, it will verify that they match.

Also, regarding dependecy information, in the form there are "legacy dependency information" and "newer dependency information" fields, which ones should appear in these docs?
Since we're talking about Legacy API Docs I'm assuming the former, but I'm not sure.

Let's just document the newer ones for now, these are the fields clients should be using.

[alanbato]

@brainwane Here's the PR #3520 (WIP). I still need to continue working on the info now that @di has cleared some things for me. And I've already added tomorrow's livechat to my calendar so I don't forget. 👍

[di]

#9085 pointed out that for "multiple use" classifier fields, the metadata field is singular (Classifier) while the form-field is plural (classifiers), a distinction that this documentation should make.

[pradyunsg]

One of the things that's come up in pypa/twine#743: how are incoming names be normalised and checked?

[MrMino]

It would be nice to standardize the error output of the servers too. Currently each index implementation serves different error codes / messages for very common snafus: lack of privileges during upload, overwrites, authentication failures, data validation errors, etc.

It's impossible to implement common checks for these in the tooling, and many of the error messages presented are confusing otherwise (e.g. Artifactory's laconic "403 Forbidden" when someone forgets to update the version in their setup.py).

[ewjoachim]

I'd say specifications on the errors return by the index server would belong in a PEP.

[dstufft]

There is now PEP 694: Upload 2.0 API for Python Package Repositories, which has discussions on discuss.python.org which is relevant to this issue.