ci: migrate PyPI deployment to OIDC trusted publishing and fix x86 cross-compilation

[gluonparticle]

Summary

Migrate the CI/CD deployment pipeline to OIDC Trusted Publishing, resolve cross-compilation linker errors, and remove legacy authentication bottlenecks.

By migrating to OIDC, this PR removes the 2FA/token-scoping bottlenecks discussed in #375, fully enabling the automated Maturin deployments that were previously disabled, and fixing #384.

1. Fix x86 cross-compilation linker error

The linux (ubuntu-latest, x86) matrix job was failing with:
rust-lld: error: cannot open crti.o: No such file or directory
The ubuntu-latest runner lacks 32-bit glibc headers by default. Fix by installing gcc-multilib conditionally for the x86 target only.

2. Migrate to OIDC Trusted Publishing

Replace hardcoded API token auth (PYPI_API_TOKEN, PYPI_TOKEN) with keyless OIDC trusted publishing via pypa/gh-action-pypi-publish.

• Add permissions: id-token: write at the workflow level.
• Fix actions/download-artifact@v4 syntax (nested pattern and merge-multiple under the with: block).
• Rewrite pypi.yml as a standalone job (reusable workflows block OIDC token propagation).
• Note: crates.yml is intentionally left untouched, as Crates.io does not yet support OIDC and still requires the repository-level CARGO_REGISTRY_TOKEN.

Testing & Verification

To verify the OIDC handshake without disrupting production or triggering false releases, I ran an isolated shadow-deployment of this exact configuration to TestPyPI on my fork. All tests and deployment matrices pass flawlessly:-

End-to-End Verification Links:
(These tests were run on another branch before I pulled the latest changes and re-added Crates.yml.

• Deploy Maturin wheels - (Success)
• Deploy Poetry - (Success)
• Rust Unit Tests - (Success)
• Python Unit Tests - (Success)
• Isolated Benchmarks - (Success)

Action Required by Maintainers

Before the next release, a repository admin must register the Trusted Publishers on PyPI. Note that crates.yml has been restored to its original state from master; since OIDC is not yet supported for Crates.io, this workflow continues to rely on the existing CARGO_REGISTRY_TOKEN secret.

(https://pypi.org/manage/account/publishing/):

1. For Maturin: PyPI Project = eko, Workflow = maturin.yml
2. For Poetry: PyPI Project = eko, Workflow = pypi.yml
   (This will allow the legacy secrets PYPI_API_TOKEN and PYPI_TOKEN to be safely deleted) ;

[scarlehoff]

Hi @gluonparticle thanks for the PR.

I think the fix to maturin and a possible migration to OIDC are two separated issues and should be fixed separately.

(also, I wouldn't do the migration right now at all (I'm not sure what toking scoping bottleneck you mean) as due to the idiosyncrasies of scientific collaborations and academic jobs we wouldn't know who to assign as eternal trusted publisher 🥲)

[gluonparticle]

Hi @gluonparticle thanks for the PR.

I think the fix to maturin and a possible migration to OIDC are two separated issues and should be fixed separately.

(also, I wouldn't do the migration right now at all (I'm not sure what toking scoping bottleneck you mean) as due to the idiosyncrasies of scientific collaborations and academic jobs we wouldn't know who to assign as eternal trusted publisher 🥲)

Hi @scarlehoff , Thanks for your feedback!

I completely agree, raising these as separate PRs should be preferred for a cleaner state and easier reversion.

Regarding the academia uncertanty and the eternal publisher concern :- My intention to propose OIDC was actually to address that specifically, as it decouples the credentials from a single person (unlike legacy API tokens).

If we register our NNPDF/eko repository as a trusted pubisher, OIDC establishes trust between the repository and the PyPI (from the pypi.yml workflow).

So as long as the NNPDF Org admins control the repository, the handshake remains valid. Given the critical nature of these changes, I'm happy to help test this setup on a temporary organization first? ;

Please let me know if you have any other specific concerns! ;

[scarlehoff]

Oh, I didn't realise trusted publishers were organisations and not individuals. Good to know.
In any case, I would still separate both things (also, fixing maturin is the important bit, we don't really have a problem with the current publishing method)