What
full_tokens is the denominator of every efficiency claim Lumen makes, and token_source records whether that number came from the tokenizer or from a bytes/4 estimate. 96.6% of the reported saving rests on a real tokenizer count. The remaining 3.4% is rows written before the token_source column existed, so they cannot be classified retrospectively. This records the decision to leave those rows unclassified rather than backfill them, and closes as wontfix so the reasoning stays searchable.
Evidence
- Provenance share over the live ledger: 96.6% of the reported saving comes from a measured count, 3.4% does not. Printed and asserted by
crates/lumen-mcp/tests/efficiency.rs, test the_recorded_savings_are_priced_and_their_provenance_is_known — it prints the share, then asserts it stays above 80.0, so a regression that stops recording token_source fails instead of quietly eroding the figure.
- The unknown is disclosed, not inferred:
unverified_provenance_rows counts every row where token_source IS NULL OR token_source <> 'measured' — crates/lumen-stats/src/lib.rs:494.
- Per-provenance breakdown of the same ledger: the "Token provenance" section of
.claude/lumen_report.sh.
The ledger is append-only by design. Rewriting the pre-column rows would replace a disclosed unknown with an undisclosed guess — the same mistake as the bytes/4 estimates that made a screenshot read 32,000 tokens.
The cost of this choice is real. The disclosure is a row count, not a share of the saving, so a reader sees how many rows are unclassified without seeing what fraction of the headline they carry. And the test floor is 80.0 while the measured share is 96.6, so the figure can drift a long way before anything fails.
What would change this
How to re-measure
# From the repository root. Requires a metering ledger on this machine;
# set LUMEN_DB=/path/to/lumen.db if the app has never run here.
cargo test -p lumen-mcp --test efficiency \
the_recorded_savings_are_priced_and_their_provenance_is_known -- --nocapture
# Row and token counts per provenance value, same ledger
./.claude/lumen_report.sh
Expect a line reading of that saving, NN.N% rests on a tokenizer count rather than bytes/4, and a pass while NN.N exceeds 80.0. With no readable ledger the test returns early and passes without printing — that is a skip, not a confirmation. The report script prints one row per token_source value; unknown is the pre-column rows.
What
full_tokensis the denominator of every efficiency claim Lumen makes, andtoken_sourcerecords whether that number came from the tokenizer or from a bytes/4 estimate. 96.6% of the reported saving rests on a real tokenizer count. The remaining 3.4% is rows written before thetoken_sourcecolumn existed, so they cannot be classified retrospectively. This records the decision to leave those rows unclassified rather than backfill them, and closes as wontfix so the reasoning stays searchable.Evidence
crates/lumen-mcp/tests/efficiency.rs, testthe_recorded_savings_are_priced_and_their_provenance_is_known— it prints the share, then asserts it stays above 80.0, so a regression that stops recordingtoken_sourcefails instead of quietly eroding the figure.unverified_provenance_rowscounts every row wheretoken_source IS NULL OR token_source <> 'measured'—crates/lumen-stats/src/lib.rs:494..claude/lumen_report.sh.The ledger is append-only by design. Rewriting the pre-column rows would replace a disclosed unknown with an undisclosed guess — the same mistake as the bytes/4 estimates that made a screenshot read 32,000 tokens.
The cost of this choice is real. The disclosure is a row count, not a share of the saving, so a reader sees how many rows are unclassified without seeing what fraction of the headline they carry. And the test floor is 80.0 while the measured share is 96.6, so the figure can drift a long way before anything fails.
What would change this
unverified_provenance_rowsalongside it, making the unknown invisible.How to re-measure
Expect a line reading
of that saving, NN.N% rests on a tokenizer count rather than bytes/4, and a pass whileNN.Nexceeds 80.0. With no readable ledger the test returns early and passes without printing — that is a skip, not a confirmation. The report script prints one row pertoken_sourcevalue;unknownis the pre-column rows.