fix: preserve fork accuracy during bounded Codex cost catch-up

[xx205]

Summary

• Preserve accurate Codex fork and subagent accounting while large rollout files are scanned in bounded, resumable slices.
• Route fork-parent baseline work through the same persistent scanner path instead of requiring a full parent read inside the child resolver.
• Persist compact parent token snapshots, sparse checkpoints, append-validation anchors, partial JSONL state, and unresolved child events so later passes can resume without re-reading verified prefixes.
• Add background catch-up progress plus adaptive scheduling: 20% target duty cycle on AC power, 5% on battery, and automatic pauses for Low Power Mode or serious/critical thermal pressure.
• Add Spend Dashboard controls to finish indexing immediately, return it to background mode, or stop without discarding saved progress.
• Keep the last populated cost report visible and explicitly marked stale while an incompatible parser cache or explicit rebuild catches up.

Problem

#2452 correctly replaced a permanent giant-file skip with resumable bounded parsing, but fork baseline resolution still has a separate whole-parent admission path.

When a child references a parent larger than the 256 MiB per-file slice, the resolver cannot admit the full parent and returns an unresolved baseline. The unresolved fallback skips the first cumulative total and only counts later positive deltas, which can permanently omit child usage already present in that first total. Re-reading a smaller parent directly inside the resolver also bypasses the ordinary persistent catch-up path and can extend a foreground refresh.

Implementation

Accurate, resumable fork baselines

• Parent files discovered from children are queued into the same CodexScanBudget and cache path as ordinary rollout files.
• Each parent stores compact token snapshots plus sparse accumulator checkpoints.
• Timestamp-monotonic indexes use binary search and early stop; non-monotonic files retain the existing ordered fallback.
• A SHA-256 prefix anchor validates append-safe reuse. Replacement or prefix rewrite invalidates incompatible progress.
• Ordinary forks retain compact unresolved token lines and publish no inherited usage until the parent baseline is available.
• After a parent becomes complete, buffered children are replayed in the same refresh without re-reading their JSONL.
• Parents outside the requested history window remain dependency work and are retained until their children resolve.

Bounded scanner work

• The initial app refresh and every subsequent automatic catch-up pass install a 2-second JSONL parser deadline in addition to the existing 256 MiB per-file and 512 MiB per-refresh byte budgets. Direct library/CLI callers keep their existing behavior unless they explicitly configure a deadline.
• The reader checks deadline/cancellation at least every 256 KiB and persists state even when a slice ends inside an oversized JSONL record.
• Progress is stored as processed/total bytes and completed/total files.
• Newest rollouts remain first, so recent usage becomes visible before historical catch-up completes.

Adaptive catch-up and UI

• Automatic mode targets 20% active time on AC and 5% on battery.
• Low Power Mode and serious/critical thermal states pause new automatic passes.
• Accelerated mode uses one scanner task at utility priority and removes duty-cycle sleeps; it does not add parallel file parsing.
• Stop is graceful at the bounded-pass boundary and keeps the saved offset.
• Spend Dashboard shows byte/file progress and controls for accelerated, background, stop, and resume states.
• Parser-producer upgrades and explicit force rebuilds retain the last populated report with its original freshness timestamp until the bounded rebuild completes.
• The retained report is presentation-only: parser-dependent offsets and totals are never reused, and ordinary warm-cache incremental refreshes are not frozen.

Real-corpus validation

A privacy-safe validation selected a stable child/parent fork pair from a real local corpus. The parent was 488,985,173 bytes, so it could not fit in one 256 MiB slice. The copied files used anonymous names and the report emitted no paths, session IDs, dates, model names, token totals, prompts, responses, or session content.

Check	Unbounded reference	Bounded scanner
Files / total bytes	2 / 489,147,332	2 / 489,147,332
Passes	1	21
Scanner run time	53.860 s	119.172 s
Unresolved buffers after completion	—	0
Incomplete files after completion	—	0
CostUsageDailyReport.data	reference	exact match
CostUsageDailyReport.summary	reference	exact match

The bounded passes ran back-to-back without automatic scheduler sleeps. The 2-second deadline covers JSONL reading/parsing; cache load/save and aggregation remain outside that parser deadline.

Upgrade-cache convergence validation

A separate privacy-safe synthetic fixture seeded a populated cache under an incompatible parser producer, then rebuilt an oversized parent/child fork corpus with 1 KiB per-file and per-refresh slices.

• The prior daily report and summary remained visible on every pending pass.
• Catch-up status carried the prior cache timestamp so the UI can label the report as stale.
• No old parser offsets, file totals, or accumulator state were reused.
• The final bounded report exactly matched an unbounded reference.
• The stale report was removed as soon as catch-up completed.
• The existing warm-cache active/archive overlap regression passed in the full suite and in three repeated isolated runs, confirming ordinary incremental refreshes remain immediate.

Verification

• make test
  • 760 test selections in 64 groups;
  • 64/64 groups passed on the first attempt;
  • 0 failures, retries, or timeouts;
  • 936.2 seconds total.
• Focused cache/fetcher/catch-up run:
  • 53 tests across 5 suites passed.
• Upgrade transition regressions:
  • incompatible-cache bounded convergence passed;
  • ordinary warm-cache overlap passed in the full suite and 3/3 isolated repeats.
• Fresh Debug package build:
  • SwiftPM and Xcode compilation were both capped at 2 jobs;
  • the signed app bundle passed deep validation and opened the Usage & Spend settings pane.
• make check
  • parser hash, package checks, locale coverage, SwiftFormat, and strict SwiftLint passed;
  • 22 localized catalogs match English.
• Privacy-safe real-corpus fork validation:
  • passed in 173.866 seconds;
  • bounded output exactly matched the unbounded reference.
• Freshly built and signed Debug app:
  • automatic/background mode showed live byte progress with Finish now and Cancel;
  • accelerated mode showed live byte progress with Continue in background and Cancel.

Screenshots

Privacy-safe Spend Dashboard screenshots for automatic/background and accelerated indexing were captured from the freshly built app. They contain no local paths, account identity, session IDs, prompts, responses, or session content.

Automatic/background catch-up:

Accelerated catch-up:

Scope and limitations

• Account, rate-limit, and remote provider probing are unchanged.
• Automatic mode remains intentionally slower than accelerated mode.
• The 2-second parser deadline does not yet bound directory enumeration or full cache decode/encode/write.
• Progress totals can grow when a live file appends or a parent outside the original window is discovered.
• This validation covers a real 466 MiB oversized-parent case, not a full 200–276 GiB cold-corpus benchmark; the PR does not promise a fixed completion ETA.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 2, 2026, 6:40 PM ET / 22:40 UTC.

ClawSweeper review

What this changes

This PR makes bounded Codex cost scans persist and resume fork-parent discovery, preserves stale reports during cache rebuilds, and adds background or accelerated catch-up controls to the Spend Dashboard.

Merge readiness

⚠️ Ready for maintainer review - 4 items remain

Keep this PR open for maintainer review. Its current head contains the maintainer-authored missing-parent discovery follow-up, but that head is not on current main; the branch still carries a substantial, compatibility-sensitive scanner/cache/UI change that needs a deliberate merge decision rather than cleanup closure.

Priority: P2
Reviewed head: a0d9d52f5af1e3d15c913bf11e265ecbfb09a378
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	Strong real-behavior proof and broad regression coverage support the patch, while the persisted-cache and scheduler scope still warrants a maintainer upgrade-safety decision.
Proof confidence	🦞 diamond lobster (5/6) ✨ media proof bonus	Sufficient (screenshot): Prepared screenshots directly show automatic and accelerated dashboard catch-up with live byte progress and the matching controls, while the PR body supplies privacy-safe real-corpus convergence results; no sensitive data is visible in the inspected images.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (screenshot): Prepared screenshots directly show automatic and accelerated dashboard catch-up with live byte progress and the matching controls, while the PR body supplies privacy-safe real-corpus convergence results; no sensitive data is visible in the inspected images.
Evidence reviewed	6 items	Current main does not contain the PR head: Current checkout is main at 5b0b9fa3, while the reviewed PR head is merge commit a0d9d52f with parents 7fe05345 and 5b0b9fa3; the PR-specific follow-up is therefore still on this open branch rather than current main. Current-main scanner remains the pre-PR bounded-byte baseline: Current main has a per-file and per-refresh byte budget, but does not contain the branch's new persisted missing-parent discovery and time-sliced catch-up implementation. Maintainer follow-up is part of the open branch: The feature-history commit 7fe05345 is authored by Peter Steinberger, titled “fix: quiesce missing Codex fork parents,” and descends from the contributor's dashboard-cache binding commit; the supplied maintainer comment says it addresses the missing-parent refresh-amplification mechanism on top of this PR.
Findings	None	None.
Security	None	None.

How this fits together

CodexBar scans local Codex session JSONL files into a persisted cost cache that feeds token and spending reports. The changed path decides which session files are scanned, resolves fork-parent baselines, persists partial progress, and publishes progress to the native Spend Dashboard.

flowchart LR
  A[Local Codex session JSONL] --> B[Bounded cost scanner]
  B --> C[Persistent scan and fork cache]
  C --> D[Fork-parent discovery]
  C --> E[Catch-up scheduler]
  D --> F[Cost and token reports]
  E --> G[Spend Dashboard progress]
  F --> G

Loading

Decision needed

Question	Recommendation
Should this large persisted-cache and bounded-scanner change land as the integrated solution for fork-parent accounting and catch-up, after a final maintainer upgrade-path review?	Verify upgrade convergence, then merge: Confirm a prior v0.46.0 cache converges through parent discovery and bounded catch-up to stable totals without renewed CPU spin, then merge the current head.

Why: The contributor proof and owner follow-up are substantial, but the branch changes persisted state, accounting visibility during unresolved forks, background scheduling, and native UI controls. The remaining decision is release-risk acceptance, not a narrow mechanical repair.

Before merge

• Resolve merge risk (P1) - Merging changes persisted Codex scan/cache state and changes unresolved fork handling from partial attribution to fail-closed attribution; upgrades must converge without permanently hiding child usage.
• Resolve merge risk (P1) - The new background scheduler and parent-discovery work run on large local corpora, so a regression could reintroduce prolonged CPU use or restart discovery instead of resuming it.
• Resolve merge risk (P1) - The PR spans 52 files and includes a maintainer follow-up commit; the supplied successful checks are strong evidence, but a maintainer should still review the final merged-head upgrade path before landing.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	52 files; 4,696 additions; 366 deletions	The change crosses persisted scanner state, cache migration behavior, scheduling, UI, localization, and regression suites.
Integrated history	7 commits, including 1 maintainer follow-up	The final head incorporates a direct follow-up for missing-parent discovery rather than only the original contributor implementation.

Merge-risk options

Maintainer options:

1. Prove the released-cache upgrade path (recommended)
   Before merge, maintainers should verify that a v0.46.0 cache resumes and converges through the new persisted parent-discovery path without retaining stale offsets or omitting child usage.
2. Accept the bounded-scan operational risk
   Merge with the supplied corpus proof as sufficient, accepting that the new scheduler and discovery index are a broad operational change for unusually large local corpora.
3. Pause for a narrower release vehicle
   Keep the branch open if the combined cache, scanner, and dashboard scope is too large to accept in one release decision.

Technical review

Best possible solution:

Merge the reviewed head only after a maintainer confirms that a cache created by the latest shipped version resumes through the new parent-discovery and catch-up paths, reaches stable final totals, and does not regress the bounded CPU behavior on a many-file corpus.

Do we have a high-confidence way to reproduce the issue?

Yes, source and the supplied real-corpus validation establish the bounded parent-scan path: current main has byte-budgeted scanning, while this branch adds resumable parent discovery and regression coverage for the unresolved-parent behavior. This read-only review did not execute the corpus or tests.

Is this the best way to solve the issue?

Yes, the branch’s persistent scanner path is the maintainable direction because it avoids a separate full-parent resolver read and keeps parent work subject to the same bounded cache path. The final merge decision still needs upgrade-safety review because the implementation changes durable cache and visibility semantics.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 5b0b9fa3e37c.

Labels

Label justifications:

• P2: This is a meaningful local cost-accounting and responsiveness fix, but the supplied evidence does not show an emergency outage or data loss.
• merge-risk: 🚨 compatibility: The PR changes persisted cost-cache fields and unresolved-fork accounting behavior across upgrades.
• merge-risk: 🚨 availability: The PR changes bounded scan scheduling and discovery work for large local corpora, with CPU and refresh responsiveness implications.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (screenshot): Prepared screenshots directly show automatic and accelerated dashboard catch-up with live byte progress and the matching controls, while the PR body supplies privacy-safe real-corpus convergence results; no sensitive data is visible in the inspected images.
• proof: sufficient: Contributor real behavior proof is sufficient. Prepared screenshots directly show automatic and accelerated dashboard catch-up with live byte progress and the matching controls, while the PR body supplies privacy-safe real-corpus convergence results; no sensitive data is visible in the inspected images.
• proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. Prepared screenshots directly show automatic and accelerated dashboard catch-up with live byte progress and the matching controls, while the PR body supplies privacy-safe real-corpus convergence results; no sensitive data is visible in the inspected images.

Evidence

What I checked:

• Current main does not contain the PR head: Current checkout is main at 5b0b9fa3, while the reviewed PR head is merge commit a0d9d52f with parents 7fe05345 and 5b0b9fa3; the PR-specific follow-up is therefore still on this open branch rather than current main. (a0d9d52f5af1)
• Current-main scanner remains the pre-PR bounded-byte baseline: Current main has a per-file and per-refresh byte budget, but does not contain the branch's new persisted missing-parent discovery and time-sliced catch-up implementation. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:71, 5b0b9fa3e37c)
• Maintainer follow-up is part of the open branch: The feature-history commit 7fe05345 is authored by Peter Steinberger, titled “fix: quiesce missing Codex fork parents,” and descends from the contributor's dashboard-cache binding commit; the supplied maintainer comment says it addresses the missing-parent refresh-amplification mechanism on top of this PR. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:3456, 7fe053452770)
• Prepared real-behavior proof is directly visible: The supplied automatic and accelerated Spend Dashboard screenshots visibly show live byte progress and the expected mode controls: “Finish now” in automatic mode and “Continue in background” in accelerated mode. (/home/runner/_work/clawsweeper/clawsweeper/artifacts/event/codex/proof-scratch/2525/proof-image-1.png, a0d9d52f5af1)
• Release status is not established for this branch: The supplied repository state identifies v0.46.0 at b1170d83 as the latest release, while this PR head is later and remains open; no shipped release provenance for the branch was established. (b1170d83963d)
• Partial-clone limitation: The checkout could read current working-tree source and commit metadata, but required historical PR blobs were unavailable locally and lazy fetch failed because the remote hostname could not resolve; this review therefore does not claim a complete independent tree-level diff audit. (a0d9d52f5af1)

Likely related people:

• steipete: Authored the open-branch follow-up for persisted missing-parent discovery and appears repeatedly in the current scanner history. (role: recent area contributor; confidence: high; commits: 7fe053452770, 1a302e74363c; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsageCache.swift)
• Milan Mijatovic: Recent current-main history shows several cost-usage cache and date-invalidation fixes in the same scanner surface, making this a relevant upgrade-safety reviewer. (role: recent adjacent contributor; confidence: medium; commits: cdab5d32b533, 505bd82e947c, 64fd710c59fc; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Perform one final released-cache-to-current-head convergence review before merge.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (8 earlier review cycles)

• reviewed 2026-07-31T04:23:08.875Z sha 274f6af :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T04:34:25.555Z sha 274f6af :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T05:23:42.566Z sha 274f6af :: found issues before merge. :: [P1] Preserve a visible upgrade path for unresolved fork usage
• reviewed 2026-07-31T08:21:24.137Z sha ab34bd0 :: needs maintainer review before merge. :: none
• reviewed 2026-07-31T13:30:47.904Z sha ab34bd0 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T04:06:13.859Z sha ebf02b4 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T20:34:15.028Z sha 7fe0534 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T22:20:49.319Z sha a0d9d52 :: needs maintainer review before merge. :: none

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab34bd0cca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bind catch-up controls to the dashboard account cache

When an account-scoped dashboard scan remains pending—for example, a Codex archive exceeding the 512 MiB per-refresh limit—this panel reports and controls UsageStore.codexCostCatchUpActivity, but dashboard rows are loaded from separate caches under accounts/<cacheIdentity> in SpendDashboardSource.load (SpendDashboardController.swift:273-283). The buttons therefore advance the store's default selected-account cache rather than the cache supplying the displayed row, and completing that task does not invalidate Codex dashboard data because Codex is excluded from sourceRevisions; users can click “Finish now” yet keep seeing the stale/partial dashboard result. Track catch-up per dashboard account cache or avoid presenting these controls as dashboard progress.

Useful? React with 👍 / 👎.

[steipete]

Completed the #2538 CPU-spin catch-up on top of @xx205's PR in 7fe0534527.

The cost cache now owns a persisted, resumable session-discovery index. Missing parent IDs are stored with an inventory generation derived from the relevant roots/partitions, so a cached unresolved child reuses the same negative dependency key and remains idle until that inventory changes. Root traversal and session-head reads reserve and consume the existing CodexScanBudget; partial head reads and directory traversal resume from the on-disk state instead of restarting. Creating the parent or changing the root/partition inventory rotates the generation and replays the child's buffered events once.

I also included the bounded mechanism-1 follow-up from #2538/#2558: newest-first sorting snapshots each file's metadata once, eliminating fstatat calls from the comparator.

Regression proof:

• 250-file synthetic corpus: the cold absent-parent lookup scans once; the second warm refresh performs 0 session-head parses.
• 32-byte discovery budget: head parsing resumes across cache round trips and eventually records the negative lookup.
• Creating the parent invalidates the miss, publishes buffered child usage, and a following refresh remains bit-for-bit stable.
• Adding a partition file rotates the discovery generation while reparsing at most the new head.
• CostUsagePerformanceGateTests: 24/24 passed.
• CostUsageScannerBreakdownTests: 95/95 passed, including out-of-window symlink parents.
• Focused fork/cache/JSONL suites passed.
• make check: clean.
• make test: all 761 selections passed in 64/64 groups; zero retries or timeouts.
• Autoreview: --mode local, clean with no accepted/actionable findings.

The 0.46.1 changelog credits @xx205 and the #2538 investigation by @Helmi and @kiranmagic7. This now covers both identified refresh-amplification mechanisms without merging the PR.

[steipete]

Refreshed PR #2525 onto current main in merge commit a0d9d52f5a (parents 7fe0534527 and 5b0b9fa3e3).

• Reconciled the only textual conflict by regenerating CodexParserHash from the combined source (aa0b0865c496e548); Fix Codex Fast mode accounting from current trace records #2519's Fast-mode Submission SQLite filters/parser and fix: preserve fork accuracy during bounded Codex cost catch-up #2525's persisted fork-parent discovery/budget path are both intact. The changelog entry now sits in the existing 0.46.1 ### Fixed section.
• The earlier Linux arm64 failure was not the SDK exit-11 flake and not scanner-related: CLICardsClaudeSwapTests failed to launch an atomically-written temporary executable. Current main contains the portable non-atomic Data.write fixture fix, and that test passed after the merge.
• Proof: make check clean; 180 focused scanner/performance/fork/cache/JSONL/Fix Codex Fast mode accounting from current trace records #2519 tests passed; make test groups 1–35 passed, group 36 hit only the explicitly waived test: scale wall-clock timing budgets on loaded runners #2509 Kiro wall-clock race, and groups 37–66 were run individually with zero failures.
• Autoreview initially refused unrelated PNGs inherited from main; reviewing the exact merged tree against origin/main via review-only commit b493ee77aad1 produced one 319 KB pass and finished clean with no accepted/actionable findings.

No merge of the PR was performed.