Update token usage step summary to report AI credits#37374
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #37374 does not have the 'implementation' label (has_implementation_label=false) and has 0 new lines of code in business logic directories (default_business_additions=0, threshold=100). No custom .design-gate.yml config present. Neither enforcement condition is met. |
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ |
|
|
There was a problem hiding this comment.
Pull request overview
This PR updates the GitHub Actions step summary token-usage reporting to write directly to GITHUB_STEP_SUMMARY (when available) and to present cost using the “AI credits” pricing model, including a simplified table layout and updated legend text.
Changes:
- Adds helpers to build and append a token-usage section to
GITHUB_STEP_SUMMARY, with a compact###heading and<details>wrapper. - Updates token usage markdown to remove effective-token columns and rename cost columns to AI credits, including a new legend.
- Extends/adjusts JS tests to validate step-summary wiring, the new table shape, and updated wording.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/js/parse_token_usage.cjs | Adds step-summary section builder + writer that appends to GITHUB_STEP_SUMMARY or falls back to the Actions summary API. |
| actions/setup/js/parse_token_usage.test.cjs | Updates summary assertions and adds coverage for the GITHUB_STEP_SUMMARY append path and section wrapper. |
| actions/setup/js/parse_mcp_gateway_log.cjs | Updates token usage markdown generation to AI credits columns and adds an AI credits legend. |
| actions/setup/js/parse_mcp_gateway_log.test.cjs | Adjusts expectations to match the new AI credits-focused table/legend output. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 3
| function buildStepSummarySection(title, markdown) { | ||
| return `### ${title}\n\n<details>\n<summary>Per-request AI credits and token totals</summary>\n\n${markdown}</details>\n\n`; | ||
| } |
| * Renders one row per request in chronological order with per-request AI credits, | ||
| * a running AI credits total, followed by an aggregate totals row and legend. |
| ); | ||
|
|
||
| lines.push(""); | ||
| lines.push("Legend: `Alias` shows the model shorthand used in the table. `ΔAI Credits` is the per-request cost, and `AI Credits` is the running total computed with the current AI credits pricing model."); |
|
🧪 Test Quality Sentinel completed test quality analysis. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd and /zoom-out — approving with minor suggestions on test coverage and naming.
📋 Key Themes & Highlights
Key Themes
- Test naming drift: One test in
parse_mcp_gateway_log.test.cjskept its old ET-column name after the ET columns were removed; its assertions no longer match what the name implies. - Direct unit test gap:
appendStepSummarySectionis exported but only tested throughmain(); both its code paths (file write vs.core.summaryfallback) merit a focused unit test. - Assertion completeness: Two
addRawassertions both match the same single call; addingtoHaveBeenCalledTimes(1)would guard against accidental double-writes.
Non-diff note
The module-level JSDoc on line 11 of parse_token_usage.cjs still reads "via core.summary.addDetails" — worth updating to reflect the new direct file-append + addRaw path.
Positive Highlights
- ✅ Clean extraction of
buildStepSummarySection/appendStepSummarySectionseparates formatting from I/O concerns - ✅ New end-to-end GITHUB_STEP_SUMMARY integration test is thorough and uses a real temp file
- ✅ Terminology rename ("effective tokens" → "AI credits") is consistent across code, comments, and tests
- ✅ Column removal is clean —
formatETimport dropped,compoundedETvariable removed, table header/total-row aligned correctly
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 1.2M · 366.1 AIC · ⌖ 13.4 AIC
| expect(md).not.toContain("effective token"); | ||
| }); | ||
|
|
||
| test("compounded ET equals sum of per-turn delta ET values", () => { |
There was a problem hiding this comment.
[/tdd] Stale test name — the ET columns were removed, so this test no longer checks ET values in the markdown.
The current assertions only verify **Total** exists and that totalEffectiveTokens > 0 on the summary object — neither of which is specific to the old ET-column behaviour. The name implies a table-content invariant that is no longer asserted.
💡 Suggested rename
Rename to something that reflects what is actually being asserted:
test("totalEffectiveTokens is positive (AIC-only table)", () => {Also remove or update the inline comment at line 1697 — // The last entry's compounded ET equals totalEffectiveTokens so must appear in the table — since ET no longer appears in the table.
| extractRequestId, | ||
| readDedupedTokenUsage, | ||
| getSummaryTitle, | ||
| buildStepSummarySection, |
There was a problem hiding this comment.
[/tdd] appendStepSummarySection is exported but not imported or directly unit-tested.
buildStepSummarySection correctly gets its own unit test (line ~546), but the companion function appendStepSummarySection is only exercised indirectly through main(). Since it has two distinct code paths — direct file write vs. core.summary fallback — it's worth adding a focused test for each branch.
💡 Suggested approach
Import the function:
const {
buildStepSummarySection,
appendStepSummarySection,
...
} = require("./parse_token_usage.cjs");Then add:
test("appendStepSummarySection writes to GITHUB_STEP_SUMMARY when set", async () => {
const summaryPath = path.join(tmpDir, "summary.md");
process.env.GITHUB_STEP_SUMMARY = summaryPath;
await appendStepSummarySection("Title", "| col |");
expect(fs.readFileSync(summaryPath, "utf8")).toContain("### Title");
});
test("appendStepSummarySection falls back to core.summary when path is empty", async () => {
process.env.GITHUB_STEP_SUMMARY = "";
await appendStepSummarySection("Title", "| col |");
expect(mockCore.summary.addRaw).toHaveBeenCalled();
expect(mockCore.summary.write).toHaveBeenCalled();
});| @@ -159,7 +175,8 @@ describe("parse_token_usage", () => { | |||
|
|
|||
| await main(); | |||
|
|
|||
There was a problem hiding this comment.
[/tdd] Both assertions match the same single addRaw call, so neither guards against an accidental double-write regression.
Since appendStepSummarySection calls addRaw exactly once with the full section, adding a toHaveBeenCalledTimes(1) check would catch any future refactor that accidentally splits the section into multiple calls.
💡 Suggested addition
expect(mockCore.summary.addRaw).toHaveBeenCalledTimes(1);
expect(mockCore.summary.addRaw).toHaveBeenCalledWith(
expect.stringContaining("### Token Usage"),
true
);The second assertion alone is sufficient to verify content; the toHaveBeenCalledTimes(1) closes the gap.
🧪 Test Quality Sentinel Report
📊 Metrics & Test Classification (12 tests analyzed)
Test Classification Details
Language SupportTests analyzed:
|
There was a problem hiding this comment.
✅ Test Quality Sentinel: 73/100. Test quality is acceptable — 0% of new tests are implementation tests (threshold: 30%). All 12 analyzed tests verify observable behavioral contracts. Minor suggestions: consider error-path coverage for the new GITHUB_STEP_SUMMARY code path.
The JavaScript-generated token usage section now lands in
GITHUB_STEP_SUMMARYand presents cost in AI credits rather than effective tokens. The summary table drops the effective-token columns and updates its legend to match the current pricing model.Step summary wiring
GITHUB_STEP_SUMMARYwhen the path is available.###heading and collapsible<details>block so it reads cleanly in workflow runs.Cost model terminology
Token usage table
ΔETandETcolumns.ΔAI CreditsandAI Credits.Coverage
GITHUB_STEP_SUMMARY