Skip benchmark-ratchet on non-performance pull requests - #289
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
Summary
WalkthroughConfigure actionlint for the self-hosted runner. Add a ChangesCI benchmark workflow
Poem
Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error, 5 warnings)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![codescene-access[bot]](https://avatars.githubusercontent.com/in/3755875?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Reviewer's GuideAdds a lightweight changes classification job to CI and gates the paid benchmark-ratchet workflow on performance-relevant file changes, plus configures actionlint to recognize the Ubicloud runner label. Flow diagram for gating benchmark-ratchet on performance-relevant changesflowchart LR
GitHubEvent[GitHub event]
ChangesJob[changes job]
PathsFilter[dorny/paths-filter]
BenchOutput{needs.changes.outputs.bench}
BenchmarkRatchet[benchmark-ratchet job]
GitHubEvent --> ChangesJob
ChangesJob --> PathsFilter
PathsFilter --> BenchOutput
BenchOutput -- pull_request and bench == 'true' --> BenchmarkRatchet
GitHubEvent -- event_name != 'pull_request' --> BenchmarkRatchet
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/actionlint.yaml:
- Line 9: Update the config-variables setting in actionlint.yaml from null to an
allow-list containing only CODESCENE_CLI_SHA256, matching the sole vars.*
workflow variable.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 16a12af4-b20a-4299-906c-ee4719a4b012
📒 Files selected for processing (2)
.github/actionlint.yaml.github/workflows/ci.yml
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
leynos/shared-actions(auto-detected)leynos/pylint-pypy-shim(auto-detected)leynos/whitaker(auto-detected)
| # Configuration variables in array of strings defined in your repository or | ||
| # organization. `null` means disabling configuration variables check. | ||
| # Empty array means no configuration variable is allowed. | ||
| config-variables: null |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- actionlint configuration ---'
cat -n .github/actionlint.yaml
printf '%s\n' '--- vars references in workflows ---'
rg -n --glob '*.yml' --glob '*.yaml' \
'\bvars\.[A-Za-z_][A-Za-z0-9_]*\b' .github || true
printf '%s\n' '--- actionlint version/config references ---'
rg -n -i 'actionlint|config-variables' \
.github Makefile* pyproject.toml setup.cfg tox.ini .pre-commit-config.yaml 2>/dev/null || true
printf '%s\n' '--- workflow files ---'
git ls-files '.github/workflows/*.yml' '.github/workflows/*.yaml'Repository: leynos/cuprum
Length of output: 1234
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
root = Path(".github")
pattern = re.compile(r"\bvars\.([A-Za-z_][A-Za-z0-9_]*)\b")
references = {}
for path in sorted(root.rglob("*")):
if path.suffix not in {".yml", ".yaml"} or "workflows" not in path.parts:
continue
text = path.read_text()
names = sorted(set(pattern.findall(text)))
if names:
references[str(path)] = names
print("workflow vars names:", sorted({name for names in references.values() for name in names}))
for path, names in references.items():
print(f"{path}: {', '.join(names)}")
config = Path(".github/actionlint.yaml").read_text()
match = re.search(r"(?m)^config-variables:\s*(.*)$", config)
print("config-variables entry:", match.group(1).strip() if match else "<missing>")
PYRepository: leynos/cuprum
Length of output: 331
🌐 Web query:
actionlint config-variables null allow-list configuration semantics
💡 Result:
In actionlint, the config-variables configuration option is used to define an allow-list for configuration variables (vars context) used within GitHub Actions workflows [1][2]. The semantics of the config-variables setting are as follows: 1. When set to an array of strings, actionlint performs strict validation on the vars context [1][2]. Any variable referenced in the workflow that is not present in this list will be reported as an undefined configuration variable [3]. 2. When set to an empty array ([]), no configuration variables are permitted; any use of the vars context will be flagged [1][2]. 3. The default value is null, which disables the validation check entirely, meaning actionlint will not attempt to verify the names of configuration variables used in the workflows [1][3][2]. This configuration is typically defined within an actionlint.yaml or actionlint.yml file located in the.github directory of the repository [1].
Citations:
- 1: https://github.com/rhysd/actionlint/blob/main/docs/config.md
- 2: https://github.com/rhysd/actionlint/blob/v1.7.12/docs/config.md
- 3: https://github.com/rhysd/actionlint/blob/main/config.go
Set config-variables to an allow-list.
Replace null with CODESCENE_CLI_SHA256, the only vars.* name used by the workflows.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/actionlint.yaml at line 9, Update the config-variables setting in
actionlint.yaml from null to an allow-list containing only CODESCENE_CLI_SHA256,
matching the sole vars.* workflow variable.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
`actionlint` rejects `ubicloud-standard-4-ubuntu-2404` as an unknown runner label without a `self-hosted-runner` declaration. The other Ubicloud-using repositories in the estate (lille, wildside, nile-valley, chutoro) already carry `.github/actionlint.yaml` for this reason; bring cuprum in line so workflow changes can be linted.
`benchmark-ratchet` is the only paid Ubicloud job in this workflow, and a July 2026 usage audit found it running on every one of 420 CI runs — including docs edits and Dependabot github-actions batches that cannot change pipeline throughput. Add a `changes` job (GitHub-hosted, seconds) that classifies the diff with `dorny/paths-filter`, and gate `benchmark-ratchet` on it for pull requests. Performance-relevant paths — `cuprum/`, `rust/`, `benchmarks/`, `conftest.py`, the `Makefile`, `pyproject.toml`, `uv.lock`, and the workflow itself — still trigger the ratchet, so dependency bumps that could alter performance are still measured. Pushes to main are never gated: the ratchet must run there to refresh the `benchmark-ratchet-main-baseline` artifact that pull-request runs compare against. The `changes` job itself runs on every event so the `needs` edge cannot leave `benchmark-ratchet` skipped on main.
The estate-wide typos base no longer exempts inline code spans wholesale, so regenerating typos.toml drops the rule the committed copy still carried and three real identifiers start failing the spelling gate: a helper name, a third-party command-line flag, and the style guide's own example of a US-spelled API name. Exempt them one at a time in typos.local.toml rather than reinstating the blanket rule; prose inside backticks should still be spell-checked. Commit the regenerated typos.toml so the gate stops re-deriving the drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate in front of benchmark-ratchet lives entirely in declarative configuration, and every part of it fails silently in the direction that costs money or hides a regression: invert the condition and a genuine performance change merges unbenchmarked, drop the event clause and pushes to main stop refreshing the baseline that pull-request runs compare against. No ordinary test notices, so parse ci.yml and assert the contract — the bench output wiring, the needs edge, the gate expression verbatim, the exact filter path set, and the runner the detector uses. Property tests over sampled changed-path sets then check the rule those parts encode, against a path model a companion test stops from drifting away from the filter it claims to describe. Record the decision — event, filter verdict, ran or skipped — in the run summary of every run, because a skipped job and a broken gate look identical in the run list, and this gate exists to be audited against paid-runner spend. Add a per-ref concurrency policy while here. Superseded pull-request runs are cancelled, since a stale run only spends benchmark minutes on a diff nobody will merge; runs on main are not, because cancelling one abandons the baseline upload, and queueing them keeps two quick merges from publishing that artifact out of commit order. Narrow actionlint's config-variables to the one variable the workflows read, so a typo in a vars.* reference is an error rather than an empty string at run time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
22ac97a to
d80b30d
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
Two red checks on this branch are pre-existing and unrelated to the change; recording the evidence.
That 0.760 comes from the Python 3.15a fails four All local commit gates pass on the rebased branch: |
1 participant
Summary
This branch stops the
benchmark-ratchetjob — the only paid Ubicloud jobin this repository — from running on pull requests that cannot affect
performance. A July 2026 Ubicloud usage audit found the job firing on every
one of 420 CI runs that month, including docs-only edits and Dependabot
github-actions batches, making cuprum the estate's dominant premium-4
consumer.
A new
changesjob (GitHub-hosted, completes in seconds) classifies thediff with
dorny/paths-filter;benchmark-ratchetnow runs on pullrequests only when performance-relevant paths change. Pushes to
mainarenever gated, because the main run refreshes the
benchmark-ratchet-main-baselineartifact that pull-request runs compareagainst. Dependency bumps that touch
uv.lockorrust/still benchmarkby design, since a dependency change can legitimately alter throughput.
A companion commit adds
.github/actionlint.yaml
declaring the
ubicloud-standard-4-ubuntu-2404label, matching theconvention already present in lille, wildside, nile-valley, and chutoro,
so that
actionlintpasses on this repository's workflows.Review walkthrough
changesjob and the gatedneeds/ifonbenchmark-ratchetin.github/workflows/ci.yml
— the comments explain why
changesruns on every event (a skippedneedsdependency would otherwise skip the main-branch baseline run).for the runner-label declaration.
Validation
actionlint .github/workflows/ci.yml: clean (with the new labeldeclaration).
uv run pytest cuprum/unittests/test_extension_ci_contract.py cuprum/unittests/test_fetch_main_benchmark_baseline.py -q: 15 passed.Notes
dorny/paths-filteris pinned to the v3.0.2 commit(
de90cc6f…), consistent with the repository's SHA-pinning policy; thegrouped github-actions Dependabot ecosystem will keep it updated.
benchmark-ratchetis a required status check, a skipped run countsas satisfied, so gated pull requests still merge normally.
.github/actionlint.yamlnow listsCODESCENE_CLI_SHA256rather thandisabling the configuration-variable check, so a typo in a
vars.*reference is an actionlint error rather than an empty string at run time.
dictionary stopped exempting inline code spans, so regenerating
typos.tomlbroke the spelling gate on three pre-existing identifiers.They are exempted individually in
typos.local.tomlrather than byreinstating the blanket rule.
Review feedback addressed
cuprum/unittests/test_benchmark_gate_ci_contract.pyparsesci.ymland asserts the gate: thebenchoutput wiring, theneedsedge, the gate expression verbatim, the exact filter path set, the
GitHub-hosted runner for
changes, the summary step, and theconcurrency policy. Property tests (Hypothesis) over sampled changed-path
sets check the rule those parts encode — any watched path benchmarks
however it is mixed with docs, a diff touching nothing watched skips, and
a non-pull-request event always benchmarks. A companion test fails if a
filter pattern outside the two modelled forms is added, so the path model
cannot silently stop describing the filter. Verified non-vacuous by
inverting the gate and deleting a filter path: both mutations fail.
changesjob appends the decision — event, filterverdict, and whether the benchmark ran or was skipped — to
$GITHUB_STEP_SUMMARYon every run. All three fields are closed sets, sothe summaries stay countable; a skipped job and a broken gate are
otherwise indistinguishable in the run list.
concurrency: ci-${{ github.ref }}withcancel-in-progresstrue onlyfor pull requests. Superseded pull-request runs are cancelled; runs on
mainare not, because a cancelled run abandons the baseline upload, andqueueing them keeps two quick merges from publishing that artifact out of
commit order.
docs/developers-guide.mdrecords the path list, the runner label, themain-push behaviour, and why
changesis ungated;docs/users-guide.mdand
docs/cuprum-design.mdstate the gate where they describe theratchet, and ExecPlan 4.4.3 carries an amendment note.
Summary by Sourcery
Gate the paid benchmark-ratchet CI job behind a cheap path-change detector so it only runs when performance-relevant files change, and declare the Ubicloud self-hosted runner label for actionlint.
CI:
References