Skip to content

v1.108.291 — Counting each byte of source once

Choose a tag to compare

@jgravelle jgravelle released this 22 Aug 16:38
· 111 commits to main since this release

Two metrics summed byte_length over every symbol in a file, so a class span and its methods spans were both counted.

The inflation is not uniform, which is why it corrupted the metric rather than scaling it: it tracks how class-heavy a file is. Measured on this repo at 33.4% overall and up to 2.28x on a single file, with get_architecture_metrics byte mass reaching 2.85x the real size of the files it describes. A Gini coefficient compares across files, so an error that varies per file is bias.

Fixed

  • get_architecture_metricsbytes_per_file moves 0.5682 to 0.5519, and the top-concentrator ranking changes (sqlite_store.py 2nd to 3rd). The ranking is the output callers act on. Files whose spans cannot be trusted are now unmeasurable, not zero, disclosed as bytes_unmeasurable_files; with nothing measurable the axis is None rather than 0.0, which would read as "perfectly even".
  • Token-savings baseline, seven sites where the first pass found four, in two shapes: nested spans (5 tools), and file_sizes charged per symbol instead of per file (get_ranked_context, 12.3x at 40 symbols, 32.2x at 1000). The affected tools credited themselves with more than they had saved.

One definition each in tools/_utils: file_byte_mass, symbol_span_bytes, distinct_file_bytes. Both shapes are ratcheted over the whole of src/.

Scope: the affected sites are six analytical tools. The high-volume retrieval path is not among them and never was — search_symbols, search_text and get_symbol_source accumulate under a seen_files guard, and get_file_content measures one file.

Added

  • _savings.json gains by_tool — a lifetime token count per tool, reported by get_session_stats as lifetime_by_tool. Local only: the telemetry payload stays {delta, total, anon_id}, pinned by a test that reads the sender source. Disclosed in SECURITY.md.
  • History cannot be backfilled, so sum(by_tool) starts below the lifetime total. The difference is reported as lifetime_unattributed rather than dropped, and SAVINGS_BASIS_GENERATION stamps the meter so a total spanning the change reads mixed_basis. Nothing is recomputed — a recomputed history is a guess wearing a measurement clothes.

Also ships the route-benchmark work merged via #533/#534 and unreleased until now: H3 refuted, pair_availability, and the @3 baseline correction.