Skip to content

Fix firewall token usage reporting to preserve raw counts and remove cache-rate transforms#31581

Merged
pelikhan merged 5 commits into
mainfrom
copilot/fix-token-usage-counting
May 11, 2026
Merged

Fix firewall token usage reporting to preserve raw counts and remove cache-rate transforms#31581
pelikhan merged 5 commits into
mainfrom
copilot/fix-token-usage-counting

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 11, 2026

  • Inspect branch, main, and merge requirements
  • Run baseline build/tests before changes
  • Merge origin/main into this branch and resolve conflicts if any
  • Recompile generated artifacts and commit only required updates
  • Run targeted validation and reply to PR comment

Copilot AI and others added 2 commits May 11, 2026 21:02
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix double-counting of cached input tokens in audit Fix firewall token usage reporting to preserve raw counts and remove cache-rate transforms May 11, 2026
Copilot AI requested a review from pelikhan May 11, 2026 21:12
@pelikhan pelikhan marked this pull request as ready for review May 11, 2026 21:39
Copilot AI review requested due to automatic review settings May 11, 2026 21:39
@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot merge main and recompile.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates gh aw audit (and the setup action’s step summary) to stop deriving cache-efficiency / transformed totals from firewall token logs, and instead report raw token bucket counts while keeping Effective Tokens (ET) as the only computed metric.

Changes:

  • Removes cache-efficiency calculation from Go and JS token-usage parsing paths.
  • Updates audit rendering to print explicit raw token buckets (input/output/cache read/cache write) and removes “Total” / “Cache hit” lines.
  • Updates Go and JS tests to assert the new “raw buckets + ET-only computed” behavior.
Show a summary per file
File Description
pkg/cli/token_usage.go Removes cache-efficiency computation during token usage parsing while keeping ET computation.
pkg/cli/token_usage_test.go Updates expectations to ensure cache-efficiency is no longer computed.
pkg/cli/audit_report_render_findings.go Changes audit token usage output to render raw token buckets and removes cache hit reporting.
pkg/cli/audit_report_test.go Adds coverage to ensure audit rendering no longer prints transformed totals/cache hit lines.
actions/setup/js/parse_mcp_gateway_log.cjs Removes cacheEfficiency from the parsed summary model and markdown footer.
actions/setup/js/parse_mcp_gateway_log.test.cjs Updates tests to assert cacheEfficiency is not computed or rendered.
pkg/workflow/data/action_pins.json Pins github/gh-aw-actions/setup@v0.71.5.
pkg/actionpins/data/action_pins.json Pins github/gh-aw-actions/setup@v0.71.5 (duplicate pin source).
.github/workflows/issue-arborist.lock.yml Updates locked workflow content (including codex harness model flag invocation).

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

pkg/cli/token_usage.go:176

  • CacheEfficiency is no longer computed, but TokenUsageSummary still carries a non-omitempty cache_efficiency float field. That means any JSON marshaling of TokenUsageSummary will now report cache_efficiency: 0 even when there are cache reads, which is misleading (and looks like an actual 0% metric rather than “not reported”). Consider removing the field from the summary entirely (and any downstream diff/rendering that uses it), or changing it to an optional/pointer field with omitempty so it’s absent when not computed/reported.
	tokenUsageLog.Printf("Parsed %d entries: %d input, %d output, %d cache_read, %d cache_write, %d requests",
		lineNum, summary.TotalInputTokens, summary.TotalOutputTokens,
		summary.TotalCacheReadTokens, summary.TotalCacheWriteTokens, summary.TotalRequests)

	// Compute effective tokens using per-model multipliers (with optional custom overrides)
	populateEffectiveTokensWithCustomWeights(summary, customWeights)
  • Files reviewed: 9/9 changed files
  • Comments generated: 1

Comment on lines +1059 to +1066
oldStderr := os.Stderr
r, w, err := os.Pipe()
require.NoError(t, err)
os.Stderr = w

renderTokenUsage(summary)
require.NoError(t, w.Close())
os.Stderr = oldStderr
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 11, 2026

@copilot merge main and recompile.

Done in commit 3256b81: merged origin/main into this branch and recompiled workflow artifacts.

@pelikhan pelikhan merged commit 21926a0 into main May 11, 2026
23 of 24 checks passed
@pelikhan pelikhan deleted the copilot/fix-token-usage-counting branch May 11, 2026 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gh aw audit may double-count cached input tokens and underreport cache hit rate in “Token Usage (Firewall Proxy)”

3 participants