Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result

[JSONbored]

Context

The scoring primitives (#8083-#8088) and ground-truth wiring (#8101, #8104) are all merged, but nothing runs them against a real PR yet. .github/workflows/selfhost.yml is the closest existing CI-trigger pattern in this repo to mirror: pull_request trigger with types: [opened, synchronize, reopened, ready_for_review] (the explicit list matters — the default omits ready_for_review, so marking a draft PR ready wouldn't otherwise trigger a run), a paths: filter, a draft-PR skip guard (if: github.event.pull_request.draft != true), and a concurrency group keyed on github.ref so superseded pushes cancel in-progress runs.

BacktestCase (#8083) doesn't yet carry raw diff/issue content — #8129/#8130 (not yet merged) add that. Until they land, a classify function can only honestly backtest THRESHOLD/CONFIG changes to a small, enumerable set of known constants, not logic/regex rewrites. This issue is scoped to exactly what's honestly backtestable today:

• LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR (src/services/linked-issue-satisfaction.ts)
• DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE (src/rules/advisory.ts)

A classify function comparing a stored metadata.confidence value against a NEW threshold value needs no raw content — the corpus already has everything it needs for this specific case. Logic/regex-change backtesting is a separate follow-up issue, blocked on #8129/#8130.

Requirements

• Add a new CI workflow mirroring selfhost.yml's trigger shape: pull_request (types: [opened, synchronize, reopened, ready_for_review]), paths: filtered to exactly the two files above, the same draft-skip guard, and a concurrency group keyed on github.ref.
• Add a script (e.g. scripts/backtest-threshold-check.ts, split pure-core/thin-IO the same way scripts/export-d1-core.ts/scripts/export-d1-data.ts are) that:
  1. Parses the PR diff for a changed numeric value of LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR or DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE — a bounded regex extraction against these two known constant names specifically, not a general TS-diff parser.
  2. For each changed constant, exports the current corpus for its ruleId (scripts: export the rule-precision backtest corpus as a versioned, checksummed JSON snapshot #8084's CLI, reading from the live D1), builds a classify function comparing each case's stored metadata.confidence against the NEW threshold value, and runs scoreBacktest/compareBacktestScores (calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus #8085/calibration: Pareto-floor comparator between two BacktestScoreReports #8086) against a baseline built the same way from the OLD (pre-PR) threshold value.
  3. Renders the result via renderBacktestComparison (calibration: render a backtest score/comparison report as Markdown #8088) and posts it as a PR comment.
  4. Persists the BacktestComparison result (the structured object, not just the rendered comment) as a durable record — via recordAuditEvent with a new event type — so a future track-record tool has real structured history to aggregate from, not rendered Markdown to re-parse.
• Never blocks merge — comment/report only.
• If neither known constant actually changed in the diff (the PR touched the file for an unrelated reason), the workflow completes cleanly with no comment posted — this is the expected common case, not an error.

Deliverables

• The CI workflow file.
• The threshold-detection + backtest + comment + persistence script (pure core + thin IO wrapper).
• Tests for the pure-core logic: constant-diff parsing (both constants, both changed and unchanged cases), classify-function construction, the persisted-result shape. The thin IO wrapper (D1 export, gh pr comment/API call) is exempt from the same coverage bar under this repo's established pure-core/thin-IO precedent (scripts/export-d1-data.ts), provided it contains no logic beyond argument handling, the calls, and IO.

Test Coverage Requirements

99%+ patch coverage (branch-counted) on the pure-core script logic.

Expected Outcome

A PR that changes either known confidence threshold gets an automatic, honest backtest comparison comment — the first real, running piece of this epic's advisory check. Every run's result is durably recorded, giving the eventual Phase-2 gating decision (#8105) real data to decide from instead of speculation.

Links & Resources

• Parent epic: Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082
• Builds on calibration: pure BacktestCase corpus builder from RuleFiredEvent/HumanOverrideEvent pairs #8083-calibration: render a backtest score/comparison report as Markdown #8088, calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101, calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104 (all merged)
• .github/workflows/selfhost.yml (the CI trigger/concurrency/draft-skip pattern to mirror)
• src/services/linked-issue-satisfaction.ts (LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR), src/rules/advisory.ts (DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE) — the two thresholds in scope
• scripts/export-d1-core.ts/scripts/export-d1-data.ts (the pure-core/thin-IO pattern to mirror)

Boundaries

No merge-blocking (see #8105 for that future decision). No logic/regex-change backtesting in this issue — a separate follow-up issue covers that, blocked on #8129/#8130 landing first. Scoped to exactly the two named threshold constants — do not attempt to generically detect "any numeric constant changed," which risks false positives on unrelated numeric literals elsewhere in these files.

maintainer-only — CI wiring, not a build task for contributors.

[linear-code]

JSO-5609