Skip to content

Feat/token usage stats#126

Merged
Qsnh merged 104 commits intomainfrom
feat/token-usage-stats
Apr 25, 2026
Merged

Feat/token usage stats#126
Qsnh merged 104 commits intomainfrom
feat/token-usage-stats

Conversation

@Qsnh
Copy link
Copy Markdown
Contributor

@Qsnh Qsnh commented Apr 24, 2026

No description provided.

Qsnh and others added 30 commits April 23, 2026 19:51
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace hand-rolled comma-split with utils.SplitAndTrim in claudeBaseDirs
- Use config.DefaultCodexSessionsDir() in NewSyncer instead of inline path
- Use ai.EngineClaude/Codex/Pi constants throughout tokenstat package
- Use engine constants as SQL parameters in collectSessions query
- Extract scannerBufSize constant, unify 4MB/16MB buffer inconsistency
- Fix TOCTOU stat-then-open in piParser.Parse (open directly)
- Use strings.Cut instead of strings.Index in pi.go
- Wrap storeUsages in a single transaction (fewer SQLite write cycles)
- Simplify syncSession: remove redundant notFound boolean
- Remove what-not-why comment from launchRuntime in manager.go

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove TOCTOU pre-stat in findSessionFile; detect missing dir via WalkDir error
- Eliminate parserList from Syncer struct; use package-level defaultParserOrder
- Guard storeUsages against no-op transaction when usages slice is empty
- Trim scannerBufSize comment to just the non-obvious WHY

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move getOrCreate/mapValues from claude.go to session_files.go (shared helpers)
- Replace flat os.ReadDir loop in piParser with findSessionFile (consistent with claude/codex)
- Use exec.Engine bind variable in CreateBeeExecution SQL instead of hardcoded empty string
- Remove redundant two-line comment in syncSession; condense to inline
- Add clarifying comment on codexTotals.set superseding advance
- Remove redundant doc comment on resolveEngine

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix MAX(e.engine) SQL to use COALESCE(MAX(NULLIF(...)), '') so empty strings
  don't suppress non-empty engine hints from the same session
- Make storeUsages return error so syncSession propagates storage failures
  instead of silently succeeding
- Replace COUNT(*) with SELECT EXISTS in IsEmpty for early-exit efficiency
- Remove redundant Engine:"" zero-value assignment in CreateBeeExecution

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract scanJSONLFile and findWithLegacyFast helpers to eliminate
  duplicated scanner boilerplate and legacy-path lookup pattern across
  all three parsers
- Merge codexTotals into codexTokenUsage (methods advance/deltaAndSet)
  to remove a shadow type with identical fields
- Replace piModelPrefix string literal with a named constant
- Flatten pi.Parse nested conditionals; drop redundant ErrSessionDataNotFound
  re-wrap since findSessionFile already wraps it
- Replace syncer collectSessions correlated NOT EXISTS with a LEFT JOIN
  HAVING change-detection predicate so sessions are only re-synced when
  a new execution has completed since the last sync

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…identifies the source

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…OT NULL filter

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reads the last StatusUpdate record from ~/.kimi/sessions/*/{session_id}/wire.jsonl
via a forward scan that tracks only the final match (O(1) memory). Maps
input_other/output/input_cache_read/input_cache_creation to the existing
bee_token_stats schema with fixed model="kimi".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lback guard

- Replace os.Stat existence check with os.Open in findWithLegacyFast to
  eliminate the TOCTOU race window and redundant syscall
- Add defer tx.Rollback() guard in storeUsages following codebase idiom;
  remove now-redundant explicit Rollback call in the error path

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Qsnh and others added 28 commits April 24, 2026 05:47
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Legacy sessions (engine='') and sessions where all parsers return
ErrSessionDataNotFound now write a tombstone record (model=unknown,
tokens=0) to bee_token_stats. This sets synced_at so collectSessions
stops re-querying them on every sync cycle.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add tombstoneModel constant to replace magic "unknown" string in syncer
- Wrap pi parser error with session ID context, consistent with other parsers
- Replace os.Open existence check with os.Stat in findWithLegacyFast
- Add token_stats to filtered executions list response (was missing)
- Remove no-op identity map in TokenTrendChart
- Improve codex comment to explain why TotalTokenUsage resets prev state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Filter tombstone rows from API-facing store queries so sessions with no
  token data return null instead of {model:"unknown", total_tokens:0}
- Unify duplicate gin.H{} response blocks in ExecutionHandler.List into a
  single response write after branching
- Merge near-identical newTestServerWithExecutions/Sessions into shared
  newTestServer helper; collapse insertTestExecution into a one-liner
- Extract shared Tooltip+Info+TokenStatsTooltip pattern into
  TokenStatsInfoButton component used in sessions and session-detail pages
- Add LIMIT 500 to collectSessions query to bound result set on large DBs
- Add migration 44: index on bee_executions(engine) for the collectSessions
  WHERE clause (previously doing a full table scan every 10 min)
- Trim GetTokenTrend doc comment to keep only the non-obvious cross-day note

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Export TombstoneModel from store to eliminate duplicate const in syncer
- Build collectSessions IN clause dynamically from defaultParserOrder
- Fix TOCTOU in findWithLegacyFast (use Open instead of Stat)
- Use lazy scanner buffer (nil start, 16MiB cap) in scanJSONLFile

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use paginatedResponse helper in execution List handler instead of inlining fields
- Add config.DefaultKimiSessionsDir() and use it in NewKimiParser (consistent with codex/pi)
- Replace defaultParserOrder slice with ai.AllEngines() via Syncer.engines field
- Cache collectSessions SQL and args in Syncer at construction time (built once, not per tick)
- Use os.Stat instead of os.Open/Close for legacy path existence check in findWithLegacyFast
- Extract collectTokenStats helper to remove duplicate scan loops in GetBySessionID/GetBySessionIDs
- Separate DB error from empty-result case in buildSessionTokenStats and log the error

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…gation logic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…card UX

- formatTokenCount: add B suffix for >= 1B tokens
- formatNumber: new helper for locale-formatted integers
- Total Messages uses formatNumber (e.g. 10,000)
- Execution Duration secondary row now matches Active Workers / Tokens Today pattern (yesterday value + trending icon + %)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…omparison

Add yesterday message count and day-over-day change indicator to the Messages
Today card; add yesterday execution count and change indicator to the Executions
Today card. Remove received/sent message split and success/failed execution
breakdown from both the API response and backend queries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ExecStats

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ens card labels

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The y-axis labels (e.g. "120.0M", "180.0M", "240.0M") were getting
their leading digit clipped by the chart's -20px left margin, causing
them to display as "20.0M", "80.0M", "40.0M". Added formatTokenCountAxis
which drops the decimal for 3-digit M/K/B values, keeping labels short
enough to avoid clipping while preserving full precision in tooltips.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The engine WHERE filter in syncer.go covers almost all rows
(empty engine + all known engines), so the index is never used
by the query planner. Drop the migration to avoid unnecessary
write overhead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
resolveEngine can never fail: startup ensures the default engine is
enabled, and ValidateEngine guards any runtime change to the default.
Remove the error return, drop the 12-line dead-code block in
ExecuteWorker, and delete the test that exercised the unreachable path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a legend below the TOKEN USAGE TREND chart, matching the style of
the WORKER ACTIVITY & WORK DURATION chart legend.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…eview

- Add missing error log in buildTokenStatsMap for DB failures
- Fix scanner.Bytes() aliasing: copy bytes before passing to callback
- Extract buildDailySlice generic helper to eliminate triplicated
  daily-fill loops in GetTrend, GetExecutionDurationTrend, GetTokenTrend
- Extract changeIndicator helper in dashboard to eliminate four repeated
  ternary chains for icon/color/label computation
- Tighten TrendLineCard props with discriminated union: dataKey and
  tooltipLabel are required when children is not provided
- Trim redundant WHAT comment in execution handler List endpoint

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dpoints

- Rename execution_handler.go → session_handler.go
- Merge GET /workers/:id/executions + GET /executions into GET /sessions with optional worker_id param
- Rename GET /executions/:id/logs → GET /sessions/:id/logs
- Remove unused GET /executions/:id endpoint
- Update frontend api client and hooks accordingly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
TokensTotalInput/Output, TokensTodayInput/Output, and TokensYestInput/Output
were fetched from the DB and returned in the API response but never consumed
by the frontend (not in the TS type, no JS references). Remove the dead fields,
simplify the SQL to SELECT only total_tokens, and drop the stale test assertions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Qsnh Qsnh merged commit b9f1a71 into main Apr 25, 2026
@Qsnh Qsnh deleted the feat/token-usage-stats branch April 27, 2026 01:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant