Skip to content

Intercept .js and .jsx — no new grammar needed #8

Description

@HackPoint

What

Reads of .js and .jsx are not intercepted, so a large JavaScript file is billed at full cost
even though Lumen already carries a grammar that can outline it. It is the cheapest item left on the
coverage backlog: extension plumbing, not new parsing.

Evidence

  • Live metering ledger: 97,678 tokens across 20 built-in reads of .js files at or above the
    300-line threshold, over 17 distinct files, the largest 1,232 lines. 57 .js reads in total;
    most fall below the threshold and are out of scope.
  • tree-sitter-typescript 0.23.2 is already a dependency of lumen-core. Its TypeScript grammar is
    a syntactic superset of JavaScript; its TSX grammar handles JSX.
  • crates/lumen-core/queries/typescript-tags.scm already captures class_declaration,
    function_declaration and method_definition — the node kinds JavaScript parses to under that
    grammar.
  • Lists to change: crates/lumen-core/src/coverage.rs:32 (SOURCE_EXTS) and :41
    (INTERCEPTED_SOURCE_EXTS), plus both copies of the hooks' own list —
    .claude/hooks/lumen_read_intercept.sh:118 and the INTERCEPT_TEMPLATE the installer writes,
    lumenator/src-tauri/src/setup.rs:1266. Leaving the second one out recreates the drift
    coverage.rs was written to end.
  • Interception is retroactive in the ledger metric, and this is the part that bites.
    coverage::classify tests today's INTERCEPTED_SOURCE_EXTS against every historical row, so the
    moment js enters that list the 20 above-threshold .js reads already recorded reclassify from
    UncoveredKind to Optimizable — which is precisely what leaked counts.
    crates/lumen-mcp/tests/efficiency.rs:585 asserts leaked == 0 across all rows with no era
    boundary, so that assertion fails on any machine carrying this ledger until the check is
    reconciled; SIGNED_SINCE at :439 is the existing precedent for drawing such a boundary.
    crates/lumen-core/src/coverage.rs:181 likewise asserts a.js at 500 lines is UncoveredKind,
    and has to be rewritten in the same commit.
  • The repository corpus holds no real .js file at or above 300 lines, so this must be validated
    with fixtures rather than the corpus.

Not free. The mts/cts precedent is supported by the tools but deliberately not intercepted,
because interception changes which reads get diverted; the 97,678 tokens only land if js/jsx
enter both lists, which accepts that. Both known parsing failure modes degrade rather than break: a
mis-parsed file yields ERROR nodes, which the outline builder skips and the ranked arm declines as
NoDefs; minified bundles are single-line and never reach the threshold. The ledger
reclassification above is not a degradation — it is a test that goes red by construction, and
deciding how to reconcile it is part of this change rather than a surprise to discover afterwards.

Acceptance criteria

  • lumen_core::structure::detect_lang maps js, mjs, cjs to TypeScript and jsx to Tsx.
  • lumen_core::ranked::TagLang::detect returns the same mapping for those extensions.
  • js and jsx appear in SOURCE_EXTS, in INTERCEPTED_SOURCE_EXTS and in both hook lists;
    the existing superset test still passes, and coverage.rs:181 is updated to assert the new
    classification instead of the old one.
  • A .js fixture with a class, a function and a method outlines all three names, with
    returned_tokens < full_tokens.
  • A .jsx fixture with a component outlines its name, with returned_tokens < full_tokens.
  • The leak check in crates/lumen-mcp/tests/efficiency.rs is reconciled deliberately —
    reads recorded before .js was covered are distinguished from reads after it — and only then
    does the efficiency test report leaked as 0.
  • .js leaves the uncovered-kind rows because it moved to the intercepted side, not because
    past reads were relabelled as defects.
  • No new entry is added to any Cargo.toml.

How to test

cd /path/to/lumen
cargo test --release -p lumen-core structure
cargo test --release -p lumen-core coverage
cargo test --release -p lumen-mcp --test efficiency -- --nocapture
cargo tree -p lumen-core | grep tree-sitter

All three test commands must pass with the change complete — fixtures included, and with
coverage.rs:181 and the leak check updated as part of it. Run against unmodified assertions on a
machine that carries the ledger above, the second and third commands fail by construction, so a red
run there is the expected starting point rather than a regression. The efficiency run prints its
table: after reconciliation leaked must read 0 and .js must be absent from the uncovered-kind
rows. cargo tree must list the same grammars as before.

Metadata

Metadata

Assignees

No one assigned

    Labels

    coverageFile kinds the optimizer does not interceptenhancementNew feature or request

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions