docs(ci): record where the macos-x86 wedge actually is (B8b of #110) - #129
Conversation
B8b ran the full suite three times on macos-x86 and all three legs wedged in provisioning.test.ts after nine files in ~12.5 min. Two facts worth having in the scoped doc rather than only in an issue comment: - completed_at runs ~50 min behind the wedge on this platform, so the useful diagnostic boundary for #76 is ~13 min, not an hour. Read last_checkpoint_ts, never job_elapsed_s, as the wedge time. - The runner stops reporting inside the wedged file's own watchdog window, which is the concrete demonstration that nothing in-job can bound a lost runner. Also records #128 against the ledger section: build is not idempotent, so a concurrent leg's entry can land with zero counts and no last_checkpoint_ts while its check run still holds the real numbers. Anyone reading attempted-legs.json before that fix needs to know a missing timestamp means instrument loss, not an early death. Refs #110, #76, #128 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe CI instructions add macOS x86 runner-loss findings and document ledger defect ChangesCI diagnostics
Estimated code review effort: 1 (Trivial) | ~3 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR updates the scoped CI instructions doc to capture two durable, evidence-backed findings from #110 (B8b) about the macos-x86 · stable runner-loss wedge, so future debugging uses the correct timing boundary and avoids misreading incomplete ledger entries.
Changes:
- Document that
macos-x86full-suite runner loss wedges inprovisioning.test.tsafter ~12.5 minutes, and that GitHub’scompleted_atcan lag the actual wedge by ~50 minutes on this pool—solast_checkpoint_tsis the meaningful wedge timestamp. - Add a “known defect” note (#128) explaining how
ci-leg-ledger buildcan zero out an incomplete leg entry (e.g.,files_planned: 0, missinglast_checkpoint_ts) under refold-retry, and how to interpret such entries until fixed.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/instructions/ci.instructions.md:
- Around line 162-165: Update the paragraph around the macos-x86 stable legs to
identify 772.9 s and 745.4 s as the two intact measurements, and explicitly note
that the third record was zeroed by `#128`. Preserve the existing
provisioning.test.ts and completed_at timing analysis.
- Around line 240-242: Revise the `lands` guidance to remove the claim that a
missing `last_checkpoint_ts` means the instrument lost the payload or that the
leg died before its first file. Preserve the check-run fallback for zero-count
`runner-lost` entries, and state that the timestamp remains unavailable until
the check-run and jobs APIs can distinguish corruption from a leg ending before
its first checkpoint.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 51472835-8857-4ad3-892e-e09435ab4c09
📒 Files selected for processing (1)
.github/instructions/ci.instructions.md
📜 Review details
⏰ Context from checks skipped due to timeout. (6)
- GitHub Check: copilot-pull-request-reviewer
- GitHub Check: Unit Tests & Coverage
- GitHub Check: Unit Tests (windows-latest)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (actions)
… not at its start A live capture of the streaming job log (the blob is destroyed on runner loss, so this is the only copy) shows each leg ran 3.5-7.6 min further into provisioning.test.ts, freezing at its 5th/6th test. The previous text quoted the file boundary from last_checkpoint_ts as if it were the wedge. - boundary is ~16-20 min into the test step, not ~13 - completed_at lags the freeze by ~40 min, not ~50 - last_checkpoint_ts is the file boundary BEFORE the wedge, not the wedge Also records the instrument gap the capture exposed: the host snapshot samples at file boundaries only, so free disk, free memory and qemuCount have never been sampled inside the one file every known #76 leg dies in. Refs #110, #76 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he two missing-timestamp cases apart Both from CodeRabbit on #129. - 772.9 s / 745.4 s are the two legs whose records survived; the third was zeroed by #128. The previous wording read as if all three were measured. - "a missing last_checkpoint_ts means the instrument lost it" was too broad. It is separable, but by files_planned, not by the check-run/jobs APIs the review proposed: `open` writes the planned list before the first file runs, so a leg that genuinely died early reports files_planned: 12 / files_reported: 0, while the #128 corruption drops the payload and reports both as 0. Refs #110, #76, #128 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both findings addressed in 1. Which legs the figures come from — accepted as written. 772.9 s and 745.4 s are now explicitly the two legs whose records survived, with the third named as zeroed by #128. The old phrasing did read as if all three were measured. 2. Missing Docs-only change; no code touched. |
Evidence-only bite B8b of #110 produced two durable facts that belong in the scoped doc rather than only in an issue comment, plus a caveat against a defect found en route.
Three full-suite
macos-x86 · stablelegs (30767202175, 30767207011, 30767211555) all wedged inprovisioning.test.tsafter nine files in 772.9 s / 745.4 s (~12.5 min). Full report: #76 (comment)What this adds
Per-file watchdog section —
completed_atruns ~50 min behind the wedge on this platform (47.6 and 50.4 min measured, longer than B5's 44.6 min onlinux-x86), so the useful diagnostic boundary for #76 is ~13 min, not an hour. Readlast_checkpoint_ts, neverjob_elapsed_s, as the wedge time — and expect targeted #76 repros to be cheap now. Also records that the runner stops reporting inside the wedged file's own 1200 s watchdog window, which is the concrete demonstration that nothing in-job can bound a lost runner.Incomplete-leg ledger section — a caveat for #128:
ci-leg-ledger buildis not idempotent (finalizing a check run destroys the payload; the refold-retry path rebuilds from it), so a concurrent leg's entry can land withfiles_planned: 0and nolast_checkpoint_tswhile its check run still holds the real numbers. B8b lost one leg of three that way. Until #128 lands, a missing timestamp means the instrument lost it — not that the leg died before its first file.Docs only; no code or workflow changes. Deliberately does not touch
OBSERVED_MAX_S— these are still not full-suite figures, and the caps stay generous while #76 is open.Refs #110, #128. Not linked to close #76 — that issue's done-when is unsatisfied and the wedge is unexplained.
🤖 Generated with Claude Code
Summary by CodeRabbit