docs: metrics.json is token-usage telemetry, not aggregate eval rates — no rename

[Copilot]

Debated renaming metrics.json → aggregate_scores.jsonl. Verdict: no rename. The file contains pipeline token-usage telemetry (calls, token counts, cache stats, timing), not aggregated evaluation rates. Renaming would be wrong on format (single JSON object, not JSONL) and semantics, with high breaking-change surface for no accuracy gain.

Separately, docs had a pervasive misdescription of metrics.json as "aggregate rates by dimension and category" — those rates are computed on-demand from scores.jsonl by the CLI/viewer and never persisted.

Fixes

• docs/guides/results.md — correct metrics.json description; reorder fast-inspection to scores.jsonl first
• docs/concepts.md — remove metrics.json from judge-stage outputs (it's written by the runner after all stages complete)
• docs/getting-started.md — fix pipeline table attribution of metrics.json
• AGENTS.md — debug inspection order: scores.jsonl first, metrics.json last
• assert_ai/library/judges/README.md — fix weight field description (was pointing to metrics.json incorrectly)

Original prompt

first debate whether scores.jsonl/metrics.jsonl is better renamed /refactored to scores.jsonl/aggregate_scores.jsonl. only when the benefit is much larger, open a pr to refactor the code and all doc surfaces.

**Chronological Review:** This conversation is a late-night fleet-mode session for Chang (PM working on the responsibleai/ASSERT OSS eval framework) preparing for v1 public-preview launch. The prior summary captured 7 previously merged PRs tonight (#177, #182, #181, #185, #186, #187, #188, #189) and one open PR (#180) with viewer table UX improvements.

The current segment continues from where the prior summary left off: restarting vite for PR #180 audit. Sequence:

1. Inspected port 5173 — found PID 28068 still owning it (stale vite from earlier)
2. Killed PID 28068
3. Verified port no longer LISTEN (only FinWait2 lingering)
4. Pulled latest PR viewer/compare: pin baseline left + show only populated dim columns #180 commits into p2m-wt-bank-primary worktree
5. Restarted vite detached with logs piped to session folder
6. Vite chose 5174 instead of 5173 (FinWait2 socket still lingering)
7. Verified HTTP 200 on 5174
8. Reported viewer URL + audit checklist to Chang

Intent Mapping:

• Last user message: "kill exising ports (why still exsits 5173) and refire a new one for 180 audit." — Chang wants the viewer restarted to inspect the latest PR viewer/compare: pin baseline left + show only populated dim columns #180 commits.

Technical Inventory:

• Vite v7.3.2 dev server
• SvelteKit viewer at viewer/src/routes/suite/[suite_id]/+page.svelte
• PowerShell Get-NetTCPConnection / Stop-Process / Start-Process for port management
• git worktrees pattern (p2m-wt-bank-primary is Chang's primary demo worktree on PR viewer/compare: pin baseline left + show only populated dim columns #180 branch)
• responsibleai remote (= responsibleai/ASSERT, where work pushes)
• TCP FinWait2 state explanation (lingering socket doesn't prevent rebind unless LISTEN)

Code Archaeology:

• PR viewer/compare: pin baseline left + show only populated dim columns #180 branch changliu2/viewer-compare-baseline-pin-and-populated-dims now at commit d74e009
• 6 commits total: 7dea023, a5ae45c, 0e3559d, a2658eb, afd95bf, d74e009
• Fast-forward pulled 99 lines (80 insertions / 19 deletions) into the single file viewer/src/routes/suite/[suite_id]/+page.svelte

Progress Assessment:

• All 19 todos from prior summary marked done
• Viewer successfully restarted; audit ready
• fix-test-runtime-safety agent still running in background
• PR viewer/compare: pin baseline left + show only populated dim columns #180 awaiting Chang's eyeball merge

Context Validation:
All loose ends from prior summary still relevant (CODEOWNERS placeholder, Discussions, PR #88 flip, stash cleanup).

Recent Commands Analysis:

1. Get-NetTCPConnection -LocalPort 5173 → PID 28068 owning (Listen + Established + FinWait2)
2. Stop-Process -Id 28068 -Force + recheck → FinWait2 remains (process gone)
3. Get-NetTCPConnection ... -State Listen → "Port 5173 has no LISTEN — safe to rebind"
4. cd p2m-wt-bank-primary; git status; git log → clean, HEAD at a2658eb
5. git fetch responsibleai; git pull ... --ff-only → fast-forwarded a2658eb..d74e009
6. Start-Process npm.cmd run dev with redirected logs → returned PID 21864
7. Get-Content viewer-stdout.log → "VITE v7.3.2 ready in 2624 ms; Local: http://127.0.0.1:5174/"
8. Invoke-WebRequest http://127.0.0.1:5174/ → StatusCode 200
9. Final report to Chang with audit checklist

1. Conversation Overview: - Primary Objectives: Late-night fleet-mode session for Chang (PM, responsibleai/ASSERT OSS eval framework v1 public-preview prep). Original user requests this segment included merging Yeming's PR #182, fixing PR #180 dimension display, rebasing PR #181, plus 4 follow-up agents (M-3 jargon, CHANGELOG, TS rename, Bundle D hygiene), then clean-slate doc sweep, viewer table UX improvements stacked on #180, CI test fix + CI/GHA developer UX design memo, and finally — the current segment trigger — "kill exising ports (why still exsits 5173) and refire a new one for 180 audit." - Session Context: 9 PRs merged tonight (#182, #181, #185, #186, #187, #188, #189 in this segment; #177 earlier), PR #180 still open with 6 commits awaiting Chang's eyeball merge. Currently restarting vite for that audit. - User Intent Evolution: Started broad (fleet of 5 follow-up agents), narrowed to clean-slate doc sweep (rejecting migration callouts for clean public-preview), then specific viewer UX fixes, then specific test fix + design memo, now operational (restart viewer to inspect).

2. Technical Foundation:
   • Vite v7.3.2 dev server: SvelteKit viewer, normally on 5173 but tonight bound to 5174 due to lingering TCP FinWait2 socket
   • PowerShell port management: Get-NetTCPConnection, Stop-Process, Start-Process with -RedirectStandardOutput/Error -WindowStyle Hidden -PassThru
   • Git worktree-per-agent pattern: p2m-wt-bank-primary is Chang's primary demo worktree, on PR viewer/compare: pin baseline left + show only populated dim columns #180 branch
   • Remote convention: responsibleai remote = responsibleai/ASSERT (push target); origin = microsoft/ASSERT (never push)
   • TCP FinWait2 explanation: lingering socket doesn't block rebind unless State=List...

Created from Copilot CLI via the copilot delegate command.

[changliu2]

@minthigpen , fyi on metrics.jsonl actually storing token telemetry as opposed to aggregate scores. No rename needed.

[Copilot]

Pull request overview

Docs-only correction clarifying that metrics.json is pipeline token-usage telemetry rather than aggregate evaluation rates, with no file rename. Adjusts inspection order and stage attribution accordingly.

Changes:

• Correct metrics.json descriptions across user-facing docs and the judges README.
• Re-attribute metrics.json from the judge stage to the runner (post-stage).
• Reorder debug/inspection lists to put scores.jsonl first and metrics.json last.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
docs/guides/results.md	Fix metrics.json description; reorder fast-inspection list.
docs/getting-started.md	Note metrics.json is written by the runner, not the judge.
docs/concepts.md	Remove metrics.json from judge-stage outputs; add runner clarification.
assert_ai/library/judges/README.md	Fix weight field description to reference scores.jsonl aggregation.
AGENTS.md	Reorder debug inspection order: scores.jsonl first, metrics.json last.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.