Publishing fails with "Invalid attestations" when triggered by tag push but works when run manually

[object-Object]

I have a somewhat convoluted release process for the Python package associated with my project HexDebug, where the Java portion of the project is built and published by a Jenkins job, and then a tag and GitHub release are created by Jenkins using a PAT scoped to the GitHub repo. The tag push then triggers a GitHub Actions workflow (here) which builds and publishes the Python package to PyPI.

I last used this release process on May 28, 2024, and it worked fine then. However, today I attempted to publish a new version and encountered this error message:

Uploading distributions to https://upload.pypi.org/legacy/
Uploading hexdoc_hexdebug-0.3.0.1.20.1.1.0-py3-none-any.whl
WARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Invalid attestations supplied during upload: Could not verify the      
         uploaded artifact using the included attestation: Verification failed: 
         Certificate's Build Config URI                                         
         (<Extension(oid=<ObjectIdentifier(oid=1.3.6.1.4.1.57264.1.18,          
         name=Unknown OID)>, critical=False,                                    
         value=<UnrecognizedExtension(oid=<ObjectIdentifier(oid=1.3.6.1.4.1.5726
         4.1.18, name=Unknown OID)>,                                            
         value=b'\x0cchttps://github.com/object-Object/HexDebug/.github/workflow
         s/release.yml@refs/tags/v0.3.0&#43;1.20.1')>)>) does not match expected
         Trusted Publisher (release.yml @ object-Object/HexDebug) 

Logs (failed run): https://github.com/object-Object/HexDebug/actions/runs/14603021555/job/40965782334

I tried re-running the failed job a few times, which produced the same error. Changing nothing else, I then manually triggered the same workflow on the same commit through the GitHub UI using workflow_dispatch, after which the package was successfully published.

Logs (successful run): https://github.com/object-Object/HexDebug/actions/runs/14603157823/job/40965990286

Here's the publishing configuration on PyPI, if it helps:

[webknjaz]

value=b'\x0cchttps://github.com/object-Object/HexDebug/.github/workflow
s/release.yml@refs/tags/v0.3.0+1.20.1')>)>) does not match expected
Trusted Publisher (release.yml @ object-Object/HexDebug)

The + bit is weird here. It seems like the version string is 0.3.0.1.20.1, but maybe it's parsed wrong by something… The number of segments is unusually high, but it doesn't technically violate PEP 440.

A workaround could probably be disabling attestations, since that's where I suspect the error is happening.

@woodruffw could you help me inspect what's going on here? It looks like a bug/signed data corruption at glance.

[webknjaz]

Oh.. Wait! The tag is v0.3.0+1.20.1. That's where the plus is coming from! (+ is an HTML-encoded +)

Still, I don't fully understand what's happening. Perhaps, either Twine or Warehouse misinterpret it in URLs. A bare plus in URLs is often converted to a whitespace. But if there's a series of normalizations somewhere, I can imagine that being lost.

[object-Object]

Yeah - the version string is 0.3.0.1.20.1.1.0 (PyPI), while the tag is v0.3.0+1.20.1 (GitHub) to match the mod version.
(The high number of version segments is because it contains the mod version (0.3.0+1.20.1) converted to PEP 440 (0.3.0.1.20.1) as well as a version number for the Python package (1.0).)

[webknjaz]

It's likely that something in the attestations/digital signing flow isn't working correctly. But I can't say for sure if it's the signing part of the validation. We'll have to wait to hear from William.

[woodruffw]

Thanks for the ping @webknjaz! Looking now.

Edit: Triage notes:

• Log entry here: https://search.sigstore.dev/?logIndex=200966990

[woodruffw]

Okay, I see what's happening here.

As part of pypi-attestations (i.e. the attestation verification logic on PyPI's side), we check the Build Config URI extension. This extension is normally a URL like:

https://github.com/OWNER/REPO/.github/workflows/WORKFLOW@REF

We need to check this extension since (unfortunately, IMO) it's the only place GitHub encodes the workflow filename reliably in the OIDC claims, and that's the trust root for Trusted Publishing itself. To check the extension we make sure that it matches the above format by comparing it against various REF forms that GitHub emits:

https://github.com/trailofbits/pypi-attestations/blob/69036f5be096d07bc9a1703761dec7dd0f8dfc25/src/pypi_attestations/_impl.py#L517-L523

That comparison assumes that REF is not transformed between the Source Repository Ref extension and its appearance in the Build Config URI extension, but that assumption is incorrect: the latter appears to HTML-entity-encode the former, which is how we end up with +.

TL;DR: This is failing because of a failing comparison that looks like:

"https://github.com/object-Object/HexDebug/.github/workflows/release.yml@refs/tags/v0.3.0+1.20.1" == "https://github.com/object-Object/HexDebug/.github/workflows/release.yml@refs/tags/v0.3.0+1.20.1"

In terms of fixing this, I need to think a bit (and consult some other Sigstore folks) -- the "simple" thing to do would be to perform the encoding before comparison, but that (1) adds malleability in a critical location and (2) isn't an injective mapping, since there are multiple equivalent HTML entity encodings for a codepoint.

STL;DR: @object-Object if you either disable attestations or publish with a tag that doesn't contain +, you can work around this for now. But I'm also looking into a more permanent fix 🙂

[woodruffw]

CC @di @haydentherapper for viz 🙂

[di]

I'm surprised that the Build Config URI is getting HTML-encoded. I understand it's out of our control, but should that be happening?

[haydentherapper]

We can add some tests under https://github.com/sigstore/fulcio/blob/main/pkg/identity/ciprovider/principal_test.go to figure out where this is happening, if Fulcio is doing HTML encoding or if it's over the wire. Off the top of my head, I can't think of anything within Fulcio that would be HTML encoding strings. We do use the Go template library but I don't think that's the culprit as there's a separate HTML template library.

[woodruffw]

Yeah, I was surprised to see it on the Fulcio extension side: FWICT Fulcio uses JoinPath to manipulate these URLs, but that API shouldn't be doing any entity escaping on its own.

[haydentherapper]

We've found the root cause, it is because we're using html/template rather than text/template. Certain characters are being escaped. I'll have a fix deployed shortly.

[haydentherapper]

Hey all, we have rolled out an updated version of Fulcio. Please try to re-sign.

[webknjaz]

Sounds like this has been solved. Closing.

[object-Object]

Apologies for the late update, but I can confirm that the issue appears to be solved - I just ran a release of my project IoticBlocks (which uses the same versioning scheme and release process as HexDebug), and the PyPI release job completed successfully when triggered by the tag v1.0.1+1.20.1.

https://github.com/object-Object/IoticBlocks/actions/runs/14933557836/job/41955613911