Skip to content

feat(sprint-gate): latency and idle-CPU limits are measured bands; the flat 10% is gone (c, 2/3) - #764

Closed
EtanHey wants to merge 6 commits into
wt/ratchet-c1-margins-modulefrom
wt/ratchet-c2-gate-consumer
Closed

feat(sprint-gate): latency and idle-CPU limits are measured bands; the flat 10% is gone (c, 2/3)#764
EtanHey wants to merge 6 commits into
wt/ratchet-c1-margins-modulefrom
wt/ratchet-c2-gate-consumer

Conversation

@EtanHey

@EtanHey EtanHey commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Ratchet bolt-down (c), slice 2 of 3 — sprint_gate.py uses the measured band; the flat 10% is gone

Size XS (+302/−17, stacked on #763). Base is wt/ratchet-c1-margins-module; merge that first.

Definition of done — tests that FAIL on 3ee7c279 and pass here (verified by running them against a git archive 3ee7c279 tree)

test on 3ee7c279 here
test_a_value_inside_measured_variance_is_no_longer_red294 ms (a value green main actually produced, w8-REPORT 2026-09-02) vs baseline 225 ms RED (limit 247.5 = 225 × 1.10) GREEN (limit 463.7, n=7)
test_a_two_fold_regression_no_longer_hides_behind_a_stale_baseline950 ms vs the corpus's 911.887 ms baseline GREEN (limit 1003) RED (limit 463.7)
test_fewer_than_five_attested_runs_is_unmeasured_never_a_verdict verdict from a round number UNMEASURED, values reported, no limits_ms
test_idle_cpu_goes_red_beyond_the_measured_band_even_under_the_ceiling — 20% idle, five green runs at 4.0–5.0% PASS (under 30) FAIL (limit 6.2%) — R3's worked example

What changed

  • check_search: p50/p95 compared to the band from attested runs (--attestations <dir|file>), never baseline × 1.10. Fewer than five attested runs for either percentile → check status UNMEASURED: rc 0, listed in payload.unmeasured (same rule as SKIPPED — release consumers reject it themselves; the first five main runs must be able to complete for a band to exist). details.margins carries the full sentence (mean, σ, k, n, limit) per percentile.
  • check_resource: the ratified 30% ceiling stays a hard budget; the measured band is added per process (cpu_over_measured_band, cpu_margins). Unmeasured band → ceiling alone decides, stated per process.
  • corpus.json: latency_regression_fraction removed. latency_baseline_ms stays as the calibration record (hostname/captured_at), which (b) also hashes.
  • Inline attestations are replay-only; a live config carrying them is refused ("the hand-editable baseline all over again"). Fixtures all_green / search_latency_red carry five inline runs so every check is still RED-provable.

Not done here, on purpose

  • No live gate run: mcp_roundtrip stores a probe chunk in the canonical DB, which this seat may not touch. Replay fixtures only.
  • ratchet.yml does not yet download N attestations for the collector — (b) is editing that workflow; wiring the download is a follow-up after (b) merges.

Tests

tests/test_sprint_gate.py + tests/test_ratchet_margins.py: 112 passed. ruff clean. Two pre-existing expectations updated (..._runs_latency_on_calibrated_host, ..._keg_built_from_this_sha_is_proof_eligible): a live run with no attestations is now UNMEASURED, not a PASS nobody measured.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1


Note

Medium Risk
Changes release-gate semantics: runs without attestations no longer get implicit PASS on latency/CPU bands, and overall gate can exit 0 with UNMEASURED checks—consumers must honor the new fields and attestation supply.

Overview
Sprint gate now judges search latency (p50/p95) and idle CPU per process using measured bands from scripts/ratchet_margins and attested green-main history, instead of baseline × 1.10 or ceiling-only CPU checks.

Live runs load history via --attestations; inline attestations in config are replay-only (live configs with inline attestations refuse). Payloads expose UNMEASURED when fewer than five attested runs exist for a metric, list checks in unmeasured, and treat overall PASS like SKIPPED (rc 0 unless something FAILs). FAIL-before-UNMEASURED ordering prevents a real regression from being masked by a sibling without a band.

resource_budget keeps the ratified CPU/RSS ceiling as a hard fail, and adds per-process idle-CPU ratchet verdicts (cpu_margins, cpu_verdicts, cpu_over_measured_band). corpus.json drops latency_regression_fraction. Fixtures and tests cover historical false RED/GREEN cases, attestation validation, and live --attestations wiring.

Reviewed by Cursor Bugbot for commit 10051b0. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Replace flat 10% latency threshold with attestation-based measured bands in sprint_gate

  • check_search and check_resource now judge p50/p95 latency and idle CPU against per-percentile, per-process margins derived from attestation histories instead of the removed latency_regression_fraction.
  • Adds an UNMEASURED check status for cases where no usable band exists (fewer than five attested runs). gate_status accepts UNMEASURED alongside PASS and SKIPPED for aggregate success, and result_payload lists unmeasured checks.
  • Adds a --attestations CLI argument for live runs; inline attestations are replay-only and validated (duplicate run IDs and malformed entries refuse the gate).
  • Behavioral Change: gate_status now reports overall PASS when checks are only PASS, SKIPPED, and/or UNMEASURED; a measured-band FAIL takes precedence over an UNMEASURED sibling. Calibrated live hosts without attestations now report search_latency and resource_budget as UNMEASURED rather than PASS.

Macroscope summarized 10051b0.

@EtanHey EtanHey added the XS Extra-small change (400 lines or fewer) label Sep 5, 2026
@EtanHey

EtanHey commented Sep 5, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_398a5ce8-136f-48c0-b304-024d28c20500)

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Team

Run ID: aba58a93-2b64-41f1-972b-e37cb6b66b70

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

@EtanHey I will review the changes.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@deepsource-io

deepsource-io Bot commented Sep 5, 2026

Copy link
Copy Markdown

DeepSource Code Review

We reviewed changes in e6f2219...10051b0 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

Important

Some issues found as part of this review are outside of the diff in this pull request and aren't shown in the inline review comments due to GitHub's API limitations. You can see those issues on the DeepSource dashboard.

PR Report Card

Overall Grade   Security  

Reliability  

Complexity  

Hygiene  

Code Review Summary

Analyzer Status Updated (UTC) Details
Python Sep 5, 2026 12:27p.m. Review ↗
Swift Sep 5, 2026 12:27p.m. Review ↗
JavaScript Sep 5, 2026 12:27p.m. Review ↗
Shell Sep 5, 2026 12:27p.m. Review ↗
Secrets Sep 5, 2026 12:27p.m. Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

Comment thread scripts/sprint_gate.py
Comment on lines +883 to +886
try:
config["attestations"] = margins.load_attestations(args.attestations)
except margins.AttestationError as error:
return str(error)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium scripts/sprint_gate.py:883

An inaccessible --attestations root causes load_attestations(args.attestations) to raise PermissionError (and other scan-time OSErrors), so this path escapes main as a traceback instead of returning the documented structured refusal with no checks. Catch OSError here as well, or normalize it inside load_attestations.

     try:
         config["attestations"] = margins.load_attestations(args.attestations)
-    except margins.AttestationError as error:
+    except (margins.AttestationError, OSError) as error:
         return str(error)
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @scripts/sprint_gate.py around lines 883-886:

An inaccessible `--attestations` root causes `load_attestations(args.attestations)` to raise `PermissionError` (and other scan-time `OSError`s), so this path escapes `main` as a traceback instead of returning the documented structured refusal with no checks. Catch `OSError` here as well, or normalize it inside `load_attestations`.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at the source in 096b078 (scripts/ratchet_margins.py, slice 1, which this PR is rebased on): load_attestations converts any OSError during the scan into AttestationError, so attestation_refusal returns the structured refusal. Investigating this found a second, quieter path: Path.rglob swallows PermissionError and would have returned zero files, reading an unreadable store as the bootstrap state. Replaced with os.walk(onerror=raise); test_a_root_that_cannot_be_scanned_is_a_refusal_not_a_traceback covers both.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

Comment thread scripts/sprint_gate.py
Comment thread tests/test_sprint_gate.py Outdated
history = real_history()
for document in history[3:]:
del document["measured"]["latency_baseline_ms.p95"]
returncode, payload = replay(tmp_path, capsys, latency_fixture([100.0] * 10, history))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused variable 'returncode'


An unused variable takes up space in the code, and can lead to confusion, and it should be removed. If this variable is necessary, name the variable _ to indicate that it will be unused, or start the name with unused or _unused.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asserted in 5e80279: returncode == 0 — the unmeasured verdict is not a failure, and the test now says so instead of discarding the value.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from ee9b324 to 5e80279 Compare September 5, 2026 11:38
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_b68fa876-45c2-4423-84c5-27fdbb02b2a8)

Comment thread scripts/sprint_gate.py
Comment on lines +885 to +908
return None
try:
config["attestations"] = margins.load_attestations(args.attestations)
except margins.AttestationError as error:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium scripts/sprint_gate.py:885

A malformed --attestations file with "run_id": [] aborts the gate with an unhandled TypeError instead of returning the required structured refusal with rc=1. load_attestations uses run_id as a set key before validate_attestation rejects its non-hashable type, so catch this malformed-input error here or validate run_id before deduplication.

Suggested change
except margins.AttestationError as error:
except (margins.AttestationError, TypeError) as error:
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @scripts/sprint_gate.py around line 885:

A malformed `--attestations` file with `"run_id": []` aborts the gate with an unhandled `TypeError` instead of returning the required structured refusal with `rc=1`. `load_attestations` uses `run_id` as a set key before `validate_attestation` rejects its non-hashable type, so catch this malformed-input error here or validate `run_id` before deduplication.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at the source in f3bc50b (slice 1, this PR rebased onto it): validate_attestation now requires run_id to be an int or a non-empty string — checked before load_attestations ever uses it as a key — so "run_id": [], true and " " are all AttestationError → the structured refusal, rc 1. I did not widen the except to TypeError: that would also swallow genuine programming errors in the gate as if they were bad input. Pinned by three new cases in test_a_malformed_attestation_is_refused_with_the_reason.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

EtanHey added a commit that referenced this pull request Sep 5, 2026
…before it is used as a key (Macroscope, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from 5e80279 to 83f41ac Compare September 5, 2026 11:42
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_6406ec2e-c654-4a4d-a930-98ea19eca181)

EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from 83f41ac to 83f362f Compare September 5, 2026 11:45
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_2fa7dc93-536b-41ea-8a66-2f2f146509ed)

EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from 83f362f to a3cf856 Compare September 5, 2026 11:53
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_f5af8d67-8150-471d-aa3e-5ef409fc4679)

@EtanHey

EtanHey commented Sep 5, 2026

Copy link
Copy Markdown
Owner Author

Lead review round 1 (Cursor pass) — disposition, head a3cf8562

finding disposition
high UNMEASURED masks a measured FAIL on the sibling percentile Real — the fail-open class this bolt exists to kill. Fixed in a3cf856. Order in check_search is now any False → FAIL; else any None → UNMEASURED; else PASS (scripts/sprint_gate.py:222). Test written first and failing: test_a_measured_fail_on_one_percentile_is_never_masked_by_an_unmeasured_sibling — 950 ms p50 against the 7-run band with p95 at 3 runs → rc 1, FAIL, verdicts={"p50": false, "p95": null}.
medium the per-percentile test never pins the fail-closed case Fixed by the test above (same commit).
medium inline fixture attestations skip the duplicate-run refusal Real, fixed in a3cf856: attestation_refusal calls margins.reject_duplicate_runs (:886), the same function the live store path uses (#763 abee735). test_duplicate_inline_attestations_refuse_the_replay.
low gate_status treats UNMEASURED like SKIPPED (rc 0) Refuted, intentional and stated: scripts/sprint_gate.py:962 and its docstring. Same consumer contract as SKIPPED (option (b), w6-REPORT.md): the payload names every check it could not judge in payload.unmeasured, and release consumers reject skipped/unmeasured themselves. It has to be rc-neutral for the first five attested main runs to be able to complete at all. What is NOT rc-neutral, after this round: a measured FAIL anywhere, which now wins over any unmeasured sibling.

DeepSource: Python was RED on this PR for check_resource cyclomatic 17 (PY-R1000); lifted the band step into cpu_bands() (83f362f, rebased) and it is green on this head.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from a3cf856 to c42bb2f Compare September 5, 2026 11:57
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_bf7f0351-e669-44e0-82f4-9ef6060b2e2a)

EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from c42bb2f to 9b684cd Compare September 5, 2026 11:59
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_a305c4ed-1514-4fdc-9ab6-e019bd77af19)

EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from 9b684cd to e66d280 Compare September 5, 2026 12:19
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_40b057ce-9d75-4421-883b-2f91fcc502bc)

EtanHey and others added 5 commits September 5, 2026 15:23
…d the flat 10% is gone (c, 2/3)

check_search compares p50/p95 to the band measured from attested green main runs (--attestations),
never to baseline × 1.10; fewer than five runs reports the check UNMEASURED (rc 0, listed in
payload.unmeasured, like SKIPPED) instead of a verdict nobody measured. check_resource keeps the
ratified 30% ceiling and adds the band per process. Inline attestations are replay-only.

Tests that FAIL on 3ee7c27: 294 ms (a real green-main value) was RED against 225 × 1.10; 950 ms
was GREEN against the corpus's 911.887 × 1.10 while every green-main run sits at 98–294 ms.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
…rce (DeepSource PY-R1000: cyclomatic 17)

Behaviour unchanged; the ceiling-plus-band reasoning moves into cpu_bands() with its docstring.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
…bling percentile; inline attestations obey the duplicate-run rule (lead review r1)

Order in check_search is now: any False -> FAIL; else any None -> UNMEASURED; else PASS. The payload
carries limits_ms (None where unmeasured) and per-percentile verdicts so a reader sees which side fired.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
…budget UNMEASURED, never PASS; verdicts are named (lead review r2)

Same order as check_search: any FAIL (ceiling, band, RSS, missing process) wins; else any process without a
band -> UNMEASURED, listed in payload.unmeasured; else PASS. Both checks now use margins.judge(), so a verdict
is PASS/FAIL/UNMEASURED by name and no 'is False' test can fold an unmeasured band into not-RED. all_green
carries idle_cpu_pct.* history for all four processes so the four-check green fixture proves the CPU band end
to end.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey force-pushed the wt/ratchet-c2-gate-consumer branch from e66d280 to 4ea9c57 Compare September 5, 2026 12:23
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_1c9ba750-5293-4a3b-9307-fce13e40dc44)

@EtanHey

EtanHey commented Sep 5, 2026

Copy link
Copy Markdown
Owner Author

Lead review round 2 — disposition, head 4ea9c574 (rebased on #763 e6f22198)

Lead must-answer 2 — a live calibrated run with no --attestations drops every latency limit while rc stays 0. It does not drop silently; it goes UNMEASURED, visibly, in two places: the check's own status (scripts/sprint_gate.py:226-227) and payload.unmeasured (:1009). Pinned by tests/test_sprint_gate.py:1384 test_live_gate_without_attestations_renders_latency_unmeasured, which asserts payload["unmeasured"] == ["search_latency", "resource_budget"] and the check status. What stays rc 0 is gate_status (:970), by the same contract SKIPPED already has: release consumers reject skipped/unmeasured themselves. The trade-off, stated so you can rule on it: making UNMEASURED rc 1 would fail every installed-Mac gate run from today until a Mac-side measurement enters (b)'s store through its workflow_dispatch writer — that plumbing does not exist yet, so the window is indefinite either way. I kept rc 0 + the named list and did not invent a fallback number; if you want rc 1, it is a one-line change to gate_status and the two fixture expectations. Note also that a measured FAIL anywhere now beats any unmeasured sibling (round 1), so the window only covers rows with no band at all, never a row that measured a regression.

Lead must-answer 3 — idle-CPU absence never surfaced as UNMEASURED. Real; fixed in 4ea9c57 with the same order as latency: any FAIL (ceiling, band, RSS, missing process) → FAIL; else any process without a band → UNMEASURED, listed (:469); else PASS. Both checks now use margins.judge() so every verdict is PASS/FAIL/UNMEASURED by name (cpu_verdicts, verdicts in the payload). Tests: test_idle_cpu_absence_of_a_band_is_unmeasured_not_pass (:1446), test_idle_cpu_fail_on_one_process_is_never_masked_by_an_unmeasured_sibling (:1460, the Cursor low). all_green.json now carries idle_cpu_pct.* history for all four processes so the four-check green fixture proves the CPU band end to end (the other Cursor low). Consequence, stated: a live run with no attestations now lists BOTH search_latency and resource_budget as unmeasured; three pre-existing test expectations updated to say so.

146 tests, ruff clean, DeepSource Python green.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

…k_resource (DeepSource PY-R1000: cyclomatic 18)

Behaviour unchanged; same tests pass.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_245d1258-e110-4fa9-9f38-82bd19a2c92a)

EtanHey added a commit that referenced this pull request Sep 5, 2026
…measured` — statistic + store reader (c, 1/3) (#763)

* feat(ratchet): measured margins from attested green main runs, or `unmeasured` — the statistic and the store reader (c, 1/3)

A row's limit is a one-sided 99% prediction limit for one run, mean + t(0.99, n-1)·s·√(1+1/n),
over ≥5 attested green main runs; fewer renders `unmeasured` and no number. Reads the
attestation.json artifacts (b) publishes from ratchet-attest.yml on main; nothing else, no
default path, no inline live values. No consumer yet: sprint_gate (2/3) and the table (3/3) follow.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): review round 1 on the margins module — finite-df t past the table, no assert, fail-closed store reads

- t_quantile past df=29: Cornish-Fisher expansion (|err| <= 1e-5 vs scipy for df >= 30) instead of the
  bare normal quantile, which is TIGHTER than every finite-df t (Macroscope, false-RED direction).
- describe(): explicit ValueError instead of an assert that -O would strip (DeepSource).
- load_attestations: OSError while scanning -> AttestationError, and os.walk(onerror=raise) instead of
  rglob, which swallows PermissionError and would have read an unreadable store as zero runs.
- Duplicate run_id across files is a malformed store, not two observations (Macroscope).
- Every measured value is validated at load (finite, non-negative, or null), so a loaded store can
  always be read and no row builder is the first place a bad value surfaces (Macroscope, #765).

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): a run_id must be an int or a non-empty string, refused before it is used as a key (Macroscope, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): a non-finite band is refused, never applied; integer and sum overflows become refusals (Macroscope round 2)

- limit=inf would have made within() True for every value and switched the row's gate off silently.
- mean via fsum(value/n); OverflowError anywhere in the band -> AttestationError.
- honest_value compares ints directly (math.isfinite(int) overflows past float range).
- minimum_runs below the t-table's floor is a ValueError up front, not a stdev/t_quantile failure later.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): measured_margin refuses non-finite inputs; duplicate-run rule is one function for store and inline paths (lead review r1)

Also adds the positive idle_cpu_pct coverage the review asked for (R3's soak numbers: 4.88% inside, 6.41% outside).

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): a boolean schema is not schema 1 (Macroscope)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* refactor(ratchet): split validate_attestation per field (DeepSource PY-R1000: cyclomatic 16)

No behaviour change; every existing refusal message and test is unchanged.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): validate before the unmeasured return; margin_for applies the duplicate-run rule itself (CodeRabbit)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): one run is one run whether its id is 1000 or "1000"; judge() names all three verdicts; a live non-measurement is refused (lead review r2)

- reject_duplicate_runs keys on str(run_id), so a writer's int and a fixture's string cannot count twice.
- judge(margin, value) -> PASS | FAIL | UNMEASURED: the fail-closed form of within() for consumers that
  render verdicts, so an unmeasured band cannot be collapsed into not-RED by an 'is False' test.
- within()/judge() refuse a nan/inf/negative live value (ValueError) instead of reporting a probe
  failure as a regression.
- Portable onerror coverage for find_attestations (the chmod test skips as root).

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

---------

Co-authored-by: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey
EtanHey deleted the branch wt/ratchet-c1-margins-module September 5, 2026 12:39
@EtanHey EtanHey closed this Sep 5, 2026
EtanHey added a commit that referenced this pull request Sep 5, 2026
…t unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
EtanHey added a commit that referenced this pull request Sep 5, 2026
…e flat 10% is gone (c, 2/3) (#770)

* feat(sprint-gate): latency and idle-CPU limits are measured bands, and the flat 10% is gone (c, 2/3)

check_search compares p50/p95 to the band measured from attested green main runs (--attestations),
never to baseline × 1.10; fewer than five runs reports the check UNMEASURED (rc 0, listed in
payload.unmeasured, like SKIPPED) instead of a verdict nobody measured. check_resource keeps the
ratified 30% ceiling and adds the band per process. Inline attestations are replay-only.

Tests that FAIL on 3ee7c27: 294 ms (a real green-main value) was RED against 225 × 1.10; 950 ms
was GREEN against the corpus's 911.887 × 1.10 while every green-main run sits at 98–294 ms.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* test(sprint-gate): assert the return code the per-percentile test left unused (DeepSource, #764)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* refactor(sprint-gate): lift the idle-CPU band step out of check_resource (DeepSource PY-R1000: cyclomatic 17)

Behaviour unchanged; the ceiling-plus-band reasoning moves into cpu_bands() with its docstring.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(sprint-gate): a measured FAIL is never masked by an unmeasured sibling percentile; inline attestations obey the duplicate-run rule (lead review r1)

Order in check_search is now: any False -> FAIL; else any None -> UNMEASURED; else PASS. The payload
carries limits_ms (None where unmeasured) and per-percentile verdicts so a reader sees which side fired.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(sprint-gate): an idle-CPU band that is unmeasured makes resource_budget UNMEASURED, never PASS; verdicts are named (lead review r2)

Same order as check_search: any FAIL (ceiling, band, RSS, missing process) wins; else any process without a
band -> UNMEASURED, listed in payload.unmeasured; else PASS. Both checks now use margins.judge(), so a verdict
is PASS/FAIL/UNMEASURED by name and no 'is False' test can fold an unmeasured band into not-RED. all_green
carries idle_cpu_pct.* history for all four processes so the four-check green fixture proves the CPU band end
to end.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* refactor(sprint-gate): lift sampling and the ceiling test out of check_resource (DeepSource PY-R1000: cyclomatic 18)

Behaviour unchanged; same tests pass.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

---------

Co-authored-by: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

XS Extra-small change (400 lines or fewer)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant