bench-runner: support org-level runner registration (default SCOPE=org) - #419
Merged
Conversation
The ephemeral bench runner was hard-coded to a repo-scoped registration on HarperFast/harper-pro, so only harper-pro workflows could use the harper-bench host. Register at org scope by default (org URL + org registration token) so a single host loop — and thus a single job at a time, preserving comparable perf numbers — serves every HarperFast repo's bench workflow (e.g. harper's new perf-benchmarks-nightly). SCOPE=repo restores the previous single-repo behavior. Requires the gh token to carry the admin:org scope for org-token minting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Contributor
|
Reviewed; no blockers found. |
kriszyp
added a commit
that referenced
this pull request
Jun 22, 2026
…upervisor (#423) #419 added org-scoped registration so the harper-bench host could serve both harper and harper-pro from one runner. That doesn't work: org/enterprise-level self-hosted runners are not routed jobs for these repos (the org's runner groups are inherited from the GitHub Enterprise level), so an org runner sits idle while a matching job queues indefinitely — verified empirically, a repo-scoped runner binds the same job in ~15s. Replace run-bench-runner.sh with bench-runner-supervisor.sh: it polls each repo for a queued job targeting the harper-bench label and, when it finds one, registers a throwaway repo-scoped --ephemeral runner that drains exactly that job and exits. Draining blocks, so at most one job runs at a time across all repos — strictly serial, no lock needed, and no always-idle runners. Adds an example systemd --user unit and rewrites the README for the multi-repo model. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The ephemeral bench-runner host loop (
benchmarks/ycsb/runner/run-bench-runner.sh) was hard-coded to a repo-scoped registration onHarperFast/harper-pro, so only harper-pro workflows could use theharper-benchself-hosted host. This adds aSCOPEswitch (defaultorg) that registers at org scope instead — org registration token + org URL — so a single host loop serves every HarperFast repo's bench workflow.Why
harper core's new
perf-benchmarks-nightlyworkflow (harper#1379) needs the sameharper-benchrunner. Org-scoping it (vs. standing up a second runner) keeps one loop running one container at a time, which preserves comparable night-to-night perf numbers — harper's and harper-pro's bench jobs simply serialize.SCOPE=repo ./run-bench-runner.shrestores the previous single-repo behavior.Notes for the reviewer
ghtoken to carry theadmin:orgscope (gh auth refresh -h github.com -s admin:org) — documented in the script header.harper-benchhost has been switched to org scope and verified (org runner online, old repo-scoped registration removed). This PR just brings the checked-in copy in sync.