[v0.5.0] benchmark tokens — Claude token-count integration after client rewrap (live-phase-gated)

[ayhammouda]

Refs #63.
Closes #89.

Summary

Adds the Claude token-count integration (issue #89, work package 6 of the
v0.5.0 benchmark): a guarded Anthropic count-tokens caller, envelope capture
after client-side rewrap with approximation: true marking, and the runner

• report wiring to fill and honestly aggregate client_wrapped_tokens /
  raw_payload_tokens.

• benchmarks/adapters/guard.py: registers "anthropic": "ANTHROPIC_API_KEY"
  in PROVIDER_API_KEY_ENV so the count-tokens caller reuses the existing
  BENCHMARK_LIVE_PROVIDERS_ENABLED + per-provider-key guard (same guard as
  [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #85), rather than inventing a second gate.
• benchmarks/adapters/claude_tokens.py (new): LiveClaudeTokenCounter
  calls require_live_environment("anthropic") on every count() call,
  then POSTs to the Anthropic count-tokens endpoint via urllib.request
  only -- no SDK, no new dependency (binding pre-flight amendment,
  2026-07-08). build_client_wrapped_envelope implements methodology
  "Token Measurement" steps 1-3. FakeTokenCounter is the only counter any
  test in this repo uses.
• benchmarks/runner.py: _execute_cell fills each cell's token record via
  a new _build_token_record helper. Outside a maintainer-run live phase
  (every CI/unit-test run), it falls back to the same honest None
  placeholder as before. When the guard passes, it fills
  client_wrapped_tokens / raw_payload_tokens under the exported
  METHODOLOGY_TOKEN_LABEL constant.
• benchmarks/report.py: additive change -- CellRecord gains
  token_approximation, and _basic_stats excludes approximation: true
  records from the headline-eligible client-wrapped-token median in both
  REPORT.md and the README-safe summary, while still surfacing the
  excluded count (never silently dropped).
• docs/benchmarks/model-matrix.yml: fixes a comment that claimed nothing
  under benchmarks/adapters/ calls the count-tokens API -- this module now
  does, strictly behind the guard.

D6b merged-test sanction (used, scoped exactly as granted)

Registering "anthropic" in PROVIDER_API_KEY_ENV invalidated the merged
test asserting "anthropic" is an unknown provider
(tests/benchmarks/test_adapters.py). Per the issue's 2026-07-08 pre-flight
amendment, re-pointed that one assertion at "not-a-registered-provider".
Nothing else about that test (or any other merged test) changed.

Dependency rule (binding pre-flight amendment, honored)

No SDK, no optional extra, no dev/benchmark dependency group was added.
pyproject.toml and uv.lock are absent from this diff -- proof below.

$ git diff --stat main...HEAD -- pyproject.toml uv.lock
$ echo "exit: $?"
exit: 0

(empty output: neither file appears in the diff)

Full diff stat:

$ git diff --stat main...HEAD
 benchmarks/adapters/__init__.py        |  26 +++-
 benchmarks/adapters/claude_tokens.py   | 277 +++++++++++++++++++++++++++++++++
 benchmarks/adapters/guard.py           |  26 +++-
 benchmarks/report.py                   |  48 +++++-
 benchmarks/runner.py                   | 103 ++++++++++--
 docs/benchmarks/model-matrix.yml       |  10 +-
 tests/benchmarks/test_adapters.py      |   7 +-
 tests/benchmarks/test_claude_tokens.py | 239 ++++++++++++++++++++++++++
 tests/benchmarks/test_report.py        | 131 ++++++++++++++++
 tests/benchmarks/test_runner.py        |  78 ++++++++++
 10 files changed, 915 insertions(+), 30 deletions(-)

Why this triggered supervisor review

Total diff is 915 insertions + 30 deletions = 945 changed lines, over
the 500-line threshold (precedent: #97 at 1510 insertions, #98 at 915 + 36 =
951). This PR is opened per the pipeline's supervisor-review protocol and
will not be merged by this worker. Labeled supervisor-review.

The size is mostly new, additive test coverage
(test_claude_tokens.py alone is 239 lines) plus one new, self-contained
module (claude_tokens.py, 277 lines) -- there is no single file here that
is large or hard to review in isolation.

No other supervisor-review trigger applies on its own merits: no new
third-party dependency, no pyproject.toml change, no async introduced into
a previously-sync path. Runtime network access is the one other
pipeline-flagged category and is addressed below.

Runtime network access -- proof of structural incapability without the gate

LiveClaudeTokenCounter.count() calls require_live_environment("anthropic")
as its first line, every call, regardless of caller. Tests prove this two
ways:

1. test_live_claude_token_counter_refuses_without_config_and_never_touches_network
   monkeypatches urllib.request.urlopen to raise AssertionError if ever
   called, then asserts count() raises LiveProviderDisabledError --
   i.e. the guard trips before any network call is even attempted.
2. test_token_record_is_the_honest_placeholder_when_live_guard_is_disabled
   proves the runner's default per-cell path (no env vars set, as in every
   CI run) never reaches LiveClaudeTokenCounter at all.

CI never sets BENCHMARK_LIVE_PROVIDERS_ENABLED or ANTHROPIC_API_KEY, so
this path is never exercised there. Every test that exercises the "guard
passes" branch substitutes a fake counter
(benchmarks.adapters.claude_tokens.FakeTokenCounter) for
LiveClaudeTokenCounter -- none call the real API.

"Why this approach" (unprescribed design calls)

• Approximation heuristic. The issue says a client that "cannot expose
  its exact wrapped message envelope" must be marked approximation: true.
  I read that as: cells produced by MockOpenAIAdapter /
  MockGoogleAdapter (and their live-stub counterparts) are approximations
  by construction, because those adapters never call a real
  openai-python / google-genai client -- there is no real wrapped
  request body to recover, only a synthetic mock payload. Cells produced by
  the no-MCP baseline or the offline python_docs_mcp_adapter retrieval
  adapter hand this module real prompt/tool-call data directly (no
  intermediary provider SDK claiming to have "wrapped" it), so this
  module's own fixed, documented wrapping of that data is exact, not a
  guess. build_client_wrapped_envelope takes an optional
  provider_mock_payload param to make this switch explicit and testable
  rather than inferring it from adapter identity implicitly. Note: the
  runner's per-cell dispatch does not currently invoke the OpenAI/Google
  adapters at all (out of scope per the [v0.5.0] benchmark adapter — python-docs-mcp-server tool runner (offline stdio) #86 composition decision), so in
  today's runner this branch is exercised by direct unit tests
  (test_claude_tokens.py) rather than by run_benchmark -- the mechanism
  is in place and tested for when that wiring lands.
• Token-counting failure isolation. If count_cell_tokens raises after
  the guard passes (e.g. a live-phase network error), _build_token_record
  catches it and records that cell's token status as "failed" rather than
  letting the exception propagate. A token-counting problem must never
  crash the whole benchmark run or mask that cell's actual
  answer/latency/scoring results, which is the same failure-isolation
  philosophy _execute_cell already applies to adapter dispatch.
• token_label field. Added "token_label": METHODOLOGY_TOKEN_LABEL
  to every token record (placeholder and counted) so the constant is
  visible at the per-cell artifact level, not only where the report
  generator renders it -- directly implements "labeled ... via the
  exported constant" from the acceptance criteria.
• _fmt_tokens exclusion surfacing. Rather than silently dropping
  approximation: true counts from the headline median, _fmt_tokens
  gained an approximated count parameter so both report tables show e.g.
  "100, 1 approximation:true excluded" -- consistent with the existing
  placeholder-count precedent (", N placeholder") already in that
  function.

Acceptance criteria

• Envelope capture: the counting path takes the same client-side
  wrapping used by the benchmark client (methodology steps 1-3); where
  a client cannot expose its exact wrapped envelope, the record is
  marked approximation: true and the report generator excludes it
  from headline-eligible tables.
• The count-tokens API call sits behind require_live_environment()
  (same guard as [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #85) plus an Anthropic key env var; CI and unit tests
  never call it -- tests use a fake counter.
• Completed token records fill client_wrapped_tokens and
  raw_payload_tokens (raw payload counted separately as diagnostic
  data), labeled Claude Tokens (Normalized Payload) via the exported
  constant -- never confusable with provider billing tokens.
• New third-party runtime dependency criterion: superseded by the
  2026-07-08 pre-flight amendment, which tightened this from
  "optional extra allowed" to "no dependency at all." No dependency was
  added; pyproject.toml/uv.lock are absent from the diff (proven
  above).
• Tests (additive, under tests/benchmarks/) cover fake-counter
  records, approximation marking, guard refusal without env config, and
  serialization-latency capture alongside counts (decision 5.8).
• No README/benchmark claims; Refs #63, never Closes #63.

Validation gate (full output)

$ uv run ruff check src/ tests/
All checks passed!

$ uv run ruff check benchmarks/
All checks passed!

$ uv run pyright src/
0 errors, 0 warnings, 0 informations

$ uv run pyright benchmarks/
0 errors, 0 warnings, 0 informations

$ uv run pytest --tb=short -q
........................................................................ [ 17%]
........................................................................ [ 34%]
........................................................................ [ 51%]
........................................................................ [ 68%]
........................................................................ [ 85%]
..............................................................           [100%]
422 passed, 1 warning in 9.80s

$ uv run pytest tests/benchmarks -q
........................................................................ [ 62%]
...........................................                              [100%]
115 passed in 1.66s

$ uv run python-docs-mcp-server doctor
python-docs-mcp-server doctor

  + PASS: Python version -- 3.13.10
  + PASS: SQLite FTS5 -- SQLite 3.50.4
  + PASS: Build venv support -- .venv/bin/python has venv and ensurepip available for build-index
  + PASS: Cache directory -- ~/Library/Caches/mcp-python-docs
  + PASS: Index database -- ~/Library/Caches/mcp-python-docs/index.db (0.4 MB)
  + PASS: Disk space -- 151.1 GB free

All checks passed.

uv lock --check was not run: no dependency/extras change is proposed in
this PR.

(The gate above was re-run in full, all 7 commands, after the CodeRabbit
fix in 417b330 -- output shown is from that latest run.)

CodeRabbit triage

Finding	Severity	Status
tests/benchmarks/test_claude_tokens.py L150-167: the two remaining guard-refusal tests (refuses_with_flag_but_no_key, refuses_with_key_but_no_flag) did not stub urlopen, unlike the no-config test at L135-148; a guard-order regression could have let them reach the network instead of failing deterministically	Minor / quick win	Addressed in 417b330: both tests now monkeypatch urllib.request.urlopen to raise AssertionError("network reached") before exercising the refusal -- the same fail-closed pattern as the existing test. Change is confined to this PR's own new test file (no merged test touched, no assertion weakened). Replied on the review thread.

No other CodeRabbit findings at time of writing.

Open / deferred items

None in scope. Spending API credits and any live benchmark run remain
maintainer-only, per the issue's scope boundaries.