anaconda-client doesn't know how to read valid metadata from non-setuptools build backends

[matthewfeickert]

As part of scientific-python/upload-nightly-action#15 I'm building a test package to upload following https://packaging.python.org/en/latest/tutorials/packaging-projects/. This minimal package

$ tree tests/debug_project 
tests/debug_project
├── pyproject.toml
├── README.md
└── src
    └── debug
        └── __init__.py

2 directories, 3 files

contains sufficient build information to be built with hatchling and be validated by twine

$ python -m build . && twine check --strict dist/*
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting build dependencies for wheel...
* Building wheel...
Successfully built debug-0.0.1.tar.gz and debug-0.0.1-py3-none-any.whl
Checking dist/debug-0.0.1-py3-none-any.whl: PASSED
Checking dist/debug-0.0.1.tar.gz: PASSED

However, anaconda-client v1.12.0 does not view it as having valid metadata

...
Processing "dist/debug-0.0.1-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Error:  Trouble reading metadata from "dist/debug-0.0.1-py3-none-any.whl". Is this a valid standard python package?

raising (I think) the exception block here

https://github.com/Anaconda-Platform/anaconda-client/blob/bb2e957b588a07ddd806b791ac367fa88b49ccb1/binstar_client/commands/upload.py#L253-L266

As anaconda-client apparently doesn't agree with twine on what is required metadata can you please specify what the required metadata is?

[matthewfeickert]

@vshevchenko-anaconda Feel free to close given that I'm somehow unable to replicate this (in the same Docker image as yesterday(?)) from the state of scientific-python/upload-nightly-action#27 as things are now properly failing out with

[ERROR] ('Authorization token is no longer valid', 401)

for an invalid token (as I don't want to upload for these issue discussions)

$ docker run --rm -ti -v $PWD:/work:ro mambaorg/micromamba:1.4.9-bullseye-slim /bin/bash
(base) mambauser@2a53f5d1c960:/tmp$ micromamba --quiet install --channel conda-forge --yes build twine anaconda-client
(base) mambauser@2a53f5d1c960:/tmp$ python -m build --outdir ./dist /work/tests/test_package/ && twine check --strict dist/*
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling)
* Getting dependencies for wheel...
* Building wheel...
Successfully built test_package-0.0.1.tar.gz and test_package-0.0.1-py3-none-any.whl
Checking dist/test_package-0.0.1-py3-none-any.whl: PASSED
Checking dist/test_package-0.0.1.tar.gz: PASSED
(base) mambauser@2a53f5d1c960:/tmp$ anaconda --token fail upload --user scientific-python-nightly-wheels ./dist/*.whl
Using Anaconda API: https://api.anaconda.org
Using "scientific-python-nightly-wheels" as upload username
Processing "./dist/test_package-0.0.1-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "test-package"
[ERROR] ('Authorization token is no longer valid', 401)
(base) mambauser@2a53f5d1c960:/tmp$ 

it would perhaps be interesting to still know what the required metadata is for anaconda-client, but that's maybe a discussion for another issue.

[matthewfeickert]

@vshevchenko-anaconda Hm, apparently there is a problem though as when the GitHub Action runs with a valid token (c.f. scientific-python/upload-nightly-action#27 (comment)) we are getting this failure I mentioned later on in the logs.

(this example has --verbose turned on)

...
+ anaconda --verbose --token *** upload --force --user scientific-python-nightly-wheels dist/test_package-0.0.1-py3-none-any.whl
Uploading wheels to anaconda.org...
Using Anaconda API: https://api.anaconda.org/
[DEBUG] Using token from command line args
Using "scientific-python-nightly-wheels" as upload username
Processing "dist/test_package-0.0.1-py3-none-any.whl"
Detecting file type...
[DEBUG] Testing if dist/test_package-0.0.1-py3-none-any.whl is a conda package ..
[DEBUG] Not a conda package (should be a .tar.bz2 or .conda file)
[DEBUG] Testing if dist/test_package-0.0.1-py3-none-any.whl is a Standard Python package ..
[DEBUG] This is a Standard Python wheel package
File type is "Standard Python"
Extracting standard python attributes for upload
Error:  Trouble reading metadata from "dist/test_package-0.0.1-py3-none-any.whl". Is this a valid standard python package?
Error:  Trouble reading metadata from "dist/test_package-0.0.1-py3-none-any.whl". Is this a valid standard python package?

This is strange as the GitHub Action uses the same Docker image that I used in the example in #676 (comment).

[matthewfeickert]

@vshevchenko-anaconda The problem is apparently however anaconda-client is validating metadata is assuming it was built with setuptools and not any other build backend that adheres to PEP 518.

I tested this by keeping everything else the same and switching build backends from hatchling to setuptools which then worked

--- a/tests/test_package/pyproject.toml
+++ b/tests/test_package/pyproject.toml
@@ -1,6 +1,6 @@
 [build-system]
-requires = ["hatchling"]
-build-backend = "hatchling.build"
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
 
 [project]
 name = "test-package"

This is pretty serious as this means that you're requiring a specific build backend to have build a Python package.

[matthewfeickert]

@ofek, while this isn't your problem to fix, I thought I might tag you here in the event that you can provide information that might help move this along.

[ofek]

Are you able to modify the code that checks validity so that we may see the actual traceback?

[matthewfeickert]

Are you able to modify the code that checks validity so that we may see the actual traceback?

I would need to build and install anaconda-client locally to do that. That isn't impossible (obviously :)), but that will need to wait until next week at the earliest, unless an Anaconda engineer responds here.

[matthewfeickert]

@csoja @vshevchenko-anaconda any chance that this could get looked at in the near future to figure out why anaconda-client rejects metadata that is validated by twine? This is still happening as of anacodna-client v1.12.3.