fix: repair touchstone benchmark CI#2688
Conversation
The two-workflow touchstone setup failed: the receive workflow skipped
every maintainer PR and the comment workflow then crashed on the missing
artifact.
- receive: gate `prepare` on the PR head being a same-repo branch instead
of `author_association`. The association comes from the webhook payload,
which reports CONTRIBUTOR/NONE (never MEMBER) for users with private org
membership, so legitimate maintainer PRs were silently skipped. The
same-repo check is the real security boundary (push access is required to
create a branch) and is immune to membership visibility.
- receive: replace the deprecated `::set-output` command with $GITHUB_OUTPUT,
compacting config.json via `jq -c` so no multi-line escaping is needed.
- comment: also require `workflow_run.conclusion == 'success'` so the job
no-ops instead of crashing ("Cannot read properties of undefined
(reading 'id')") when receive was skipped or failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The released touchstone tags (v1, v1.0.1) still call actions/upload-artifact@v2 and actions/download-artifact@v2, which GitHub now automatically fails. Only the upstream default branch has been updated to artifact@v4/v5 and github-script@v7, but no release includes that fix yet. Pin both the receive and comment actions to that default-branch commit until upstream tags a release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validation statusThe receive workflow runs from the PR branch and is fully green here: it ran the benchmarks and uploaded the The comment workflow is That red check is expected and cannot turn green until this PR is merged. The pinned comment action is a verified drop-in (same |
The touchstone benchmark CI added in #2685 failed on its first run (example). Three linked problems:
1. Receive workflow skipped every maintainer PR (root cause)
preparegated onauthor_association == OWNER/MEMBER/COLLABORATOR. That value comes from thepull_requestwebhook payload, which reportsCONTRIBUTOR/NONE(neverMEMBER) for maintainers whose org membership is private — so the gate was false, both jobs skipped, and no"pr"artifact was produced.→ Gate on the PR head being a same-repo branch instead. Creating a same-repo branch requires push access, so this is the real security boundary against untrusted fork code, and it's immune to membership visibility.
2. Comment workflow crashed instead of no-op'ing (the visible error)
It only checked
workflow_run.event == 'pull_request', so it ran after the skipped receive run and died downloading the missing artifact:Cannot read properties of undefined (reading 'id').→ Also require
workflow_run.conclusion == 'success'.3. Deprecated
::set-output(latent)Would have left
needs.prepare.outputs.configempty onceprepareactually ran, breaking thefromJsonbuild matrix.→ Switch to
$GITHUB_OUTPUT, compactingconfig.jsonwithjq -c(drops the multi-line escaping hack).This PR touches
.github/workflows/touchstone-*.yaml, which is in the receive trigger's path filter, so it exercises the fixed workflow itself.🤖 Generated with Claude Code