Skip to content

Outline Markdown by its headings #9

Description

@HackPoint

What

detect_lang in crates/lumen-core/src/structure.rs has no arm for .md, so Markdown falls to Lang::Unknown and outline returns a single whole-file item. The whole-file guard in crates/lumen-mcp/src/lib.rs then declines the file rather than dress that item as an outline, so the read falls back to the built-in and pays for the whole file — including repeated reads of docs that have not changed. This is the largest single opportunity in the backlog.

Evidence

  • 274,340 tokens across 40 built-in reads at or above 300 lines; the largest was 1,060 lines. 205 .md reads in total, over 64 distinct files. Source: the live metering ledger.
  • Concentrated in re-reads: README.md alone was read 25 times.
  • The .md row is visible in the coverage backlog printed by the_recorded_ledger_agrees_with_itself in crates/lumen-mcp/tests/efficiency.rs when run with --nocapture (the test returns early on a machine with no ledger).

Acceptance criteria

  • A heading scanner lives in crates/lumen-core/src/structure.rs and runs before the tree-sitter language match; the diff leaves classify and to_item untouched.
  • Each ATX heading #..###### yields a CodeItem whose name is the heading text and whose kind is h1..h6, ending at the next heading of equal or shallower depth (last heading runs to end of file).
  • A # line inside a fenced block yields no item, for both backtick and tilde fences.
  • Setext headings (===/---) are either emitted as h1/h2 or documented as unsupported, with a test asserting whichever was chosen.
  • recall_file(names=["<heading text>"]) returns only that section.
  • A .md file with no headings yields zero items, so the existing all-kind == "file" guard in crates/lumen-mcp/src/lib.rs declines the file instead of returning it whole.
  • A README-sized fixture asserts the outline costs materially less than the full file. The fraction is chosen in the test, not here: no target has been measured for Markdown.
  • The corpus sees Markdown at all before it can vouch for it. walk in crates/lumen-mcp/tests/efficiency.rs admits files via ranked::TagLang::detect, which has no md arm, so no .md file enters the corpus until that is widened. Once it does, no corpus file trips no_tool_returns_more_than_the_file_on_any_real_file.
  • The .md row in the coverage backlog is treated as its own decision rather than as a side effect. The backlog is keyed on INTERCEPTED_SOURCE_EXTS in crates/lumen-core/src/coverage.rs; adding md there reclassifies the 40 recorded built-in reads from UncoveredKind to Optimizable, which the_recorded_ledger_agrees_with_itself counts as leaks and asserts to be zero. A scanner alone does not clear the row, and clearing it means deciding what to do with that assertion.

Tradeoff worth stating: this would be the only non-tree-sitter producer. A heading is a weaker structural claim than a parse node — it names a section without bounding a definition — so ranges will be wrong where nesting is inconsistent. Isolating the scanner limits that risk rather than removing it.

How to test

From the repository root:

cargo test -p lumen-core --lib structure
cargo test -p lumen-mcp --test efficiency -- --nocapture

Expect the first command to pass the heading-nesting, fenced-#, and setext fixtures. Expect the second to print the coverage backlog and to pass the outline-fraction assertion on the README-sized fixture; the .md row stays in that backlog until the intercept list changes, which is the separate decision above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    coverageFile kinds the optimizer does not interceptenhancementNew feature or request

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions