feat(benchmark): task-level routing stats for Harbor runs

[ryan-lempka]

Harbor runs previously captured routing stats only as one run-global snapshot. This adds task-level routing stats to every Switchyard benchmark run.

• serve --routing-log-file appends one JSONL record per request: task, session, selected model, tier, token usage
• the closed-book proxy sidecar stamps x-switchyard-intake-task (baked per task at dataset prep) and a per-trial proxy_x_session_id on model-bound requests
• run-baseline.sh enables the log and finalize rolls it up into routing_stats_by_task.json next to routing_stats_final.json

Tested with the unit suite, a live serve driven by headless claude and codex plus all three inbound formats (records attribute correctly, streaming included), and a 2-task concurrent Harbor smoke where both task ids appear in the log and the rollup matches the job results.

Linear: SWITCH-1033

Summary by CodeRabbit

• New Features

  • Added optional routing request logs for switchyard serve, including model, routing tier, task, session, and token usage details.
  • Benchmark runs now produce per-task routing statistics and include these artifacts in run manifests.
  • Proxy requests can now carry task and session identifiers for improved routing attribution.

• Documentation

  • Documented routing log options, generated artifacts, and configuration restrictions.

• Tests

  • Added coverage for routing logs, token usage, streaming responses, task attribution, and manifest summaries.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-116/
Built to branch gh-pages at 2026-07-22 19:57 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[coderabbitai]

Walkthrough

Switchyard now tags proxy requests with task/session metadata, optionally logs routing details for completed responses, and summarizes those logs into per-task benchmark artifacts during manifest finalization. CLI behavior, benchmark wiring, documentation, and tests cover the new flow.

Changes

Routing observability

Layer / File(s)	Summary
Proxy task and session identity benchmark/closed_book_proxy/proxy/rewriter.py, benchmark/prepare_harbor_dataset.py, tests/test_prepare_harbor_dataset.py	Proxy containers receive task IDs and attach task/session headers to matching requests.
Routing log emission and serve integration switchyard/lib/processors/routing_log_response_processor.py, switchyard/cli/switchyard_cli.py, tests/test_routing_log_response_processor.py, docs/cli_reference.md	Serve optionally appends JSONL routing records for streaming and non-streaming responses, with model, tier, metadata, and token usage fields.
Benchmark artifact finalization benchmark/run-baseline.sh, benchmark/run_manifest.py, tests/test_run_manifest.py, benchmark/README.md	Benchmark runs pass routing paths through container execution and finalization, which aggregates records into per-task routing statistics and documents the outputs.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Poem

I’m a rabbit with logs in my paws,
Hopping through routers without flaws.
Task tags bloom bright,
Streams finish right,
Stats roll up softly because.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.34% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: task-level routing statistics for Harbor benchmark runs.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/run_manifest.py`:
- Around line 248-261: Update the aggregation logic around the model bucket in
the manifest processing function to key buckets by both model and tier,
preventing records with different tiers from being merged. Preserve the existing
call and token accumulation behavior within each model/tier bucket, and add
coverage in test_run_manifest.py for mixed-tier records.
- Around line 265-272: Add a concise triple-quoted docstring to the public
finalize_manifest function describing its artifact-copy behavior, optional
routing-log aggregation, and returned exit code. Keep the documentation focused
on the function’s existing behavior without changing its implementation.

In `@switchyard/lib/processors/routing_log_response_processor.py`:
- Around line 41-54: The public process method lacks the required documentation.
Add a concise triple-quoted docstring to RoutingLogResponseProcessor.process
stating that it returns the response unchanged and emits streaming records only
after normal drain; leave the callback and response-processing behavior
unchanged.
- Around line 46-69: Update the response completion callback `_emit` and
`_write_record` flow so the synchronous lock acquisition and file append run via
`asyncio.to_thread()` instead of blocking the event loop. Preserve the existing
record construction, serialization, and write behavior while awaiting the
offloaded operation from `_emit`.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6a4fdbfb-6276-43aa-9285-3bd1640c218b

📥 Commits

Reviewing files that changed from the base of the PR and between f91b61a and e192824.

📒 Files selected for processing (11)

• benchmark/README.md
• benchmark/closed_book_proxy/proxy/rewriter.py
• benchmark/prepare_harbor_dataset.py
• benchmark/run-baseline.sh
• benchmark/run_manifest.py
• docs/cli_reference.md
• switchyard/cli/switchyard_cli.py
• switchyard/lib/processors/routing_log_response_processor.py
• tests/test_prepare_harbor_dataset.py
• tests/test_routing_log_response_processor.py
• tests/test_run_manifest.py