Skip to content

feat(run-environment): mint CircleCI OIDC tokens for upload auth - #482

Open
fargito wants to merge 4 commits into
feat/circleci-run-environmentfrom
feat/circleci-oidc-upload-auth
Open

feat(run-environment): mint CircleCI OIDC tokens for upload auth#482
fargito wants to merge 4 commits into
feat/circleci-run-environmentfrom
feat/circleci-oidc-upload-auth

Conversation

@fargito

@fargito fargito commented Jul 28, 2026

Copy link
Copy Markdown
Member

Mint an OIDC token so CircleCI jobs need no CODSPEED_TOKEN secret.

The token CircleCI exposes in CIRCLE_OIDC_TOKEN and CIRCLE_OIDC_TOKEN_V2 cannot be used: its
audience is fixed to the id of the CircleCI organization, while CodSpeed requires its own. Only
the circleci CLI can request a custom audience — CircleCI publishes no endpoint for it, unlike
the ACTIONS_ID_TOKEN_REQUEST_URL GitHub Actions provides — so the runner shells out to
circleci run oidc get, once per upload, as a token expires an hour after it is minted.

Whether a job can mint at all is settled by minting a token and throwing it away, before the
benchmarks run. A CLI can be installed and still predate run oidc get, so probing for the binary
would pass and the mint would fail once the benchmarks had run for nothing.

Two cases keep needing a static token, and both are refused upfront: pull requests opened from a
fork, whose token names the fork, and jobs that cannot mint at all. A third is only visible when
the upload is refused: a pipeline triggered by a custom webhook gets a token naming no repository.

The error messages link integrations/ci/circleci/configuration, anchors #authentication and
#oidc-recommended. That page does not exist yet, and owes the three cases above.

Stacked on #473. Server-side support is not released yet, so a real CircleCI job cannot be
verified end to end.

Closes COD-3257

CircleCI jobs could only authenticate their uploads with a `CODSPEED_TOKEN` secret.
Mint an OIDC token instead when no static token is set, so a job needs no secret.

The token CircleCI exposes in `CIRCLE_OIDC_TOKEN_V2` cannot be used: its audience is
the id of the CircleCI organization, while CodSpeed requires its own. Only the
`circleci` CLI can request a custom audience, so the runner shells out to it, once per
upload since a minted token expires after an hour.

Two cases keep needing the static token, and both are refused before the benchmarks
run rather than at upload time: pull requests opened from a fork, whose token names
the fork, and jobs whose image does not ship the CLI. A pipeline triggered by a custom
webhook gets a token that names no repository, which only the upload can tell, so the
message shown when it is refused mentions it.

Closes COD-3257
Comment thread src/run_environment/circleci/oidc.rs Outdated
@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown

Greptile Summary

Adds native CircleCI OIDC authentication for benchmark uploads.

  • Mints a custom-audience token during configuration to detect unsupported jobs before benchmarks run.
  • Refreshes the token immediately before each upload.
  • Retains static-token authentication for forked pull requests and other unsupported pipeline types.
  • Improves CircleCI-specific authentication guidance for rejected uploads.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported compatibility-probe defect is fixed because missing or outdated CircleCI CLIs now fail during provider initialization, before benchmark execution, while preserving the CLI's diagnostic output.

Important Files Changed

Filename Overview
src/run_environment/circleci/oidc.rs Adds custom-audience token minting with exit-status, UTF-8, empty-output, and stderr handling while keeping token contents out of logs.
src/run_environment/circleci/provider.rs Replaces the inadequate CLI version probe with an actual preflight mint and refreshes authentication before every upload.
src/upload/uploader.rs Adds CircleCI-specific guidance for OIDC authentication failures caused by custom-webhook pipelines.
Cargo.toml Advances the runner package to the corresponding alpha release version.
Cargo.lock Synchronizes the locked root-package version with Cargo.toml.

Sequence Diagram

sequenceDiagram
    participant Runner
    participant Provider as CircleCI Provider
    participant CLI as circleci CLI
    participant Benchmarks
    participant Backend as CodSpeed Backend

    Runner->>Provider: check_oidc_configuration()
    Provider->>CLI: "run oidc get (aud=codspeed.io)"
    CLI-->>Provider: Preflight token
    Provider-->>Runner: OIDC available
    Runner->>Benchmarks: Execute benchmarks
    Benchmarks-->>Runner: Completed runs
    loop Each upload
        Runner->>Provider: set_oidc_token()
        Provider->>CLI: "run oidc get (aud=codspeed.io)"
        CLI-->>Provider: Fresh token
        Runner->>Backend: Upload with token
    end
Loading

Reviews (2): Last reviewed commit: "fix(run-environment): decide CircleCI OI..." | Re-trigger Greptile

@codspeed-hq

codspeed-hq Bot commented Jul 28, 2026

Copy link
Copy Markdown

Performance comparison unavailable

The base and head of this comparison were measured with different runner settings, so their benchmark values are not directly comparable.

What changed between base and head:

Re-run the base with the same settings to get a valid performance comparison.


Comparing feat/circleci-oidc-upload-auth (28253fc) with feat/circleci-run-environment (4e52cfa)

Open in CodSpeed

Comment thread src/upload/uploader.rs

fargito commented Jul 28, 2026

Copy link
Copy Markdown
Member Author

@greptileai review

fargito added 2 commits July 28, 2026 17:38
Spawning `circleci version` only proved that something answers to that name. A CLI
too old for `run oidc get`, or a job not allowed to issue tokens, passed the check
and failed at upload time, once the benchmarks had run for nothing.

Mint a token and discard it instead. It answers the same question about the CLI, plus
the ones only the CLI can answer, and it answers them before the benchmarks run. The
upload mints its own anyway, as a token expires an hour after it is minted.

Errors now carry what the CLI reported: only the first error of a chain is shown
outside of debug logging, so wrapping a mint failure in `context` was hiding the
reason it failed.

Refs COD-3257
The messages spelled out which cases OIDC cannot cover, which belongs in the
integration docs rather than in an error string, and left the CircleCI page
unreferenced while the sibling providers link theirs.

That page does not exist yet. It has to answer `#authentication` and
`#oidc-recommended`, and carry what the messages no longer say: a pipeline
triggered by a custom webhook gets a token naming no repository, and a pull
request opened from a fork gets one naming the fork.

Refs COD-3257
@fargito
fargito force-pushed the feat/circleci-oidc-upload-auth branch from 12d9f72 to 600b868 Compare July 28, 2026 15:39
Comment thread src/run_environment/circleci/provider.rs
Reading `CIRCLE_PR_NUMBER` as "this build is a forked pull request" rests on a
guarantee of CircleCI's, so quote it and point at where it is made.

Refs COD-3257
@fargito
fargito requested a review from adriencaccia July 28, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant