feat(provider): add Muse local token usage - #3371
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c74100a70
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case .grok, .muse: | ||
| return self.grokLocalTokenSnapshot(from: snapshot, historyDays: windowDays) |
There was a problem hiding this comment.
When the Muse presentation snapshot is temporarily nil while a Grok token snapshot is cached, routing .muse through grokLocalTokenSnapshot returns self.tokenSnapshots[.grok]; the Muse card can therefore display Grok token and request history. Pass the provider into the helper or use a Muse-specific projection so a missing Muse snapshot remains empty.
AGENTS.md reference: AGENTS.md:L46-L46
Useful? React with 👍 / 👎.
| if dateDirs.isEmpty { | ||
| // Prune old data even when no date dirs exist | ||
| self.pruneExpired(cache: &cache, scanSinceKey: scanSinceKey, scanUntilKey: scanUntilKey, calendar: calendar) | ||
| // Only save if not cancelled | ||
| try checkCancellation?() | ||
| cache.lastScanUnixMs = Int64(now.timeIntervalSince1970 * 1000) | ||
| MuseSessionCostCacheIO.save(cache: cache, cacheRoot: cacheRoot, calendar: calendar) | ||
| let summary = self.summaryFromCache(cache: cache, calendar: calendar, sinceKey: scanSinceKey, untilKey: scanUntilKey, now: now, fileCount: 0) |
There was a problem hiding this comment.
If the last relevant Muse date directory is deleted, dateDirs becomes empty and this early return only prunes out-of-window entries; cached files and contributions within the current window are never checked for deletion. The provider will continue reporting the deleted sessions, potentially for the remainder of the lookback window, so the empty-directory path must also invalidate missing cached files.
Useful? React with 👍 / 👎.
| let costUsage = summary?.toCostUsageTokenSnapshot( | ||
| historyDays: MuseLocalSessionScanner.defaultLookbackDays) |
There was a problem hiding this comment.
For any non-default history setting, the fetch strategy scans context.costUsageHistoryDays but this conversion always labels the result as 30 days. In particular, a 90-day scan is subsequently projected by grokLocalTokenSnapshot with historyCoverageIsEstablished == false because the published historyDays is only 30, preventing downstream dashboard coverage calculations from recognizing the complete history. Carry the requested history length into MuseUsageSnapshot instead of restoring the default.
Useful? React with 👍 / 👎.
| } | ||
| private static func parseMuseSessionFileDelta(fileURL: URL, startOffset: Int64, calendar: Calendar, lookbackCutoff: Date) -> MuseParseResult { | ||
| var result = MuseParseResult() | ||
| guard let fullData = try? Data(contentsOf: fileURL) else { return result } |
There was a problem hiding this comment.
When startOffset > 0, this still loads the entire session file before taking a suffix; the append path also reads the whole file for fingerprints before and after this call. Consequently, every append to a large active session performs multiple full-file reads and allocations rather than work proportional to the new JSONL content, which can cause large memory and I/O spikes. Seek to startOffset and hash/read bounded ranges instead.
Useful? React with 👍 / 👎.
clawsweeper
Bot
added
P2
|
Codex review: needs real behavior proof before merge. Reviewed September 1, 2026, 8:41 PM ET / September 2, 2026, 00:41 UTC. ClawSweeper reviewWhat this changesAdds an opt-in Muse provider that scans local Muse session JSONL files, caches token/request history, and displays token-only usage in CodexBar. Merge readiness⛔ Blocked until real behavior proof is added - 12 items remain Keep open: this is a useful opt-in provider feature, but it currently can show Grok data as Muse data and has three additional correctness/performance defects; it also needs real Muse-session proof and maintainer product approval. Priority: P2 Review scores
Verification
How this fits togetherCodexBar provider descriptors fetch usage data and publish snapshots to the shared Usage & Spend UI. This PR adds a local Muse-log scanner and cache that feed Muse token history into that snapshot pipeline. flowchart LR
A[Muse session JSONL files] --> B[Local Muse scanner]
B --> C[Incremental usage cache]
C --> D[Muse usage snapshot]
D --> E[Usage and Spend projection]
E --> F[CodexBar provider card]
Decision needed
Why: This adds a durable provider, environment-variable configuration, local cache, and support surface; code repair alone cannot establish whether that product commitment is desired. Before merge
Findings
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Retain an opt-in, token-only Muse provider only after its projection is provider-isolated, its cache handles complete directory removal with bounded append reads, configured history is preserved, and a real Muse run verifies the resulting card/history. Do we have a high-confidence way to reproduce the issue? Yes—source-reproducible: a nil Muse snapshot plus a cached Grok snapshot takes the introduced branch into the Grok fallback, and complete removal of the only date directory takes the cache early-return path. Is this the best way to solve the issue? No—the current reuse of a Grok-specific projection violates provider data isolation; Muse needs its own snapshot projection while preserving the caller-selected history window and cache invariants. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 2cac84440c42. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
|
|
Thank you for the Muse work. Closing this overlapping implementation in favor of #3340’s better-proven local-token approach. This patch retains cross-provider fallback, deleted-cache invalidation, history-coverage and append-I/O defects. One scanner/cache owner is preferable; useful independent improvements can be incorporated with your credit. |
2 participants
Summary
Adds Muse as a first-class CodexBar provider with local token and request usage history derived from Muse session telemetry.
Muse does not currently expose subscription quota, reset-window, or billing/cost data suitable for CodexBar's normal quota model, so this integration intentionally reports token history only rather than fabricating quota or spend information.
How it works
Muse stores local session telemetry under its session directory as JSONL records. The provider scans
runtime.session/model_completedevents and aggregates:model_completedis treated as the authoritative usage event.goal_usage_attributionrecords are deliberately ignored to avoid double-counting the same completion.Both the direct JSONL format and Muse's wrapped
retained_frame/record_jsonformat are supported.The scanner only extracts usage telemetry required for aggregation; prompt, source, and tool-output contents are not surfaced through the provider.
Incremental session cache
Muse session histories can become quite large, so rescanning every session file on each CodexBar refresh is impractical.
This adds a persistent incremental cache for Muse session usage. It tracks per-file state and daily/model contributions so subsequent refreshes can:
File replacement detection includes a lightweight prefix fingerprint so a larger file replacing an existing session at the same path is not mistaken for an append.
JSONL boundary handling also preserves incomplete trailing writes for the next scan while consuming complete final records correctly.
On a local Muse history containing ~9,500 session files / ~5.8 GB of session data, an initial scan is necessarily expensive, but unchanged warm refreshes drop to roughly 3–4 seconds with zero session files reopened.
Provider behavior
Muse intentionally exposes:
Muse intentionally does not expose:
This follows CodexBar's existing token-history path for providers where local token telemetry is available without corresponding quota/cost information.
Tests
Adds coverage for:
model_completedusage extractiongoal_usage_attributiondouble-count preventionrecord_jsonrecordsMuseLocalSessionScannerTests: 13/13 passing.The incremental cache regression tests also pass individually; one cancellation test can encounter a
swiftpm-testing-helpersignal 5 issue in the local Swift Testing environment, despite the underlying synchronous cancellation path completing correctly.Configuration
Session root resolution supports:
MUSE_HOMECODEXBAR_MUSE_HOMEMuse CLI version detection is also included for provider metadata.
Scope
This PR is limited to the Muse provider and its local usage/cache implementation.
Changes needed to support packaging CodexBar forks with non-upstream Developer ID identities are intentionally kept in a separate PR.