feat(explore): markdown index (#361) with a section-first doc tier - #1699
Open
bompus wants to merge 2 commits into
Open
feat(explore): markdown index (#361) with a section-first doc tier#1699bompus wants to merge 2 commits into
bompus wants to merge 2 commits into
Conversation
… doc tier Ports QingNagi/codegraph#361 (markdown extractor, heading nodes, name-matcher and resolution hooks) onto experimental and adds the section-first doc tier from feature/md-section-first: a doc-shaped query renders the best headed sections of the markdown file it names, ranked by idf-weighted line hits with path tokens weighted zero, capped at DOC_FILE_CAP per file. Markdown reaches an answer only through that tier, generated-file detection ignores markdown bodies, and the budget tiers count code files only so a README-heavy repo keeps its code answers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #361 and #1439, with the measurement from this comment on #361.
What this carries. #361's markdown extractor as it stands (headings, sections, tables and links as nodes; name-matcher and resolution hooks; its five test files), rebased onto
main, plus a doc tier incodegraph_exploreand three fixes the index needs to leave code answers alone.The doc tier. A doc-shaped query that names a markdown file renders that file's best sections first and whole: the top three by idf-weighted line hits, where a term that also appears in the file's own path weighs zero (it located the file, not the line) and a heading the query covers word for word counts as a named section; 8k characters per file, spent in score order and rendered in file order. The blast-radius, relationships and "additional files" blocks stay off unless a code file rendered too. Code queries take the same path they did before.
Three fixes for a shared index.
detectGeneratedFileskips the header check for markdown: a README that quotes "generated by" was dropped from the index.The server instructions now say markdown is indexed; #361's text still listed docs under what codegraph does not index, and measured as merged the model never called explore for a doc question (24 cells, 0 calls).
Measured. Headless Claude Code, Opus, a repo with 109 markdown files, six doc-question tasks, three rounds of 12 fresh cells against the shipped build with the same repo rules:
Every explore call chose the right file and section; the misses are the model grepping a file the prompt already names, and one prompt it reads as being about its own scheduling tools. Cost per cell $0.30 against $0.54.
Code answers on the same repo, one fresh session per query, 25 code queries against
main: 11 byte-identical, 3 the same lines reordered, 10 swap a fourth- or fifth-ranked padding file (markdown documents shift FTS tie ranks), and one prose prompt with no identifier answers from two markdown files instead of five unrelated code files.Tests.
tscclean. #361's five test files plus the two explore tests that the raw patch broke onmain(explore-allocation-1500,explore-session-state): 797 pass; the failures on my Windows host are all EBUSY temp-dir teardown inafterAll, none an assertion.Re-index after upgrading: the markdown nodes are written while indexing.