feat(index): best-effort parse-coverage signal — flag not-fully-indexed files#968
Merged
Conversation
…iles
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Persist the parse-coverage signal outside the graph and make the misses
queryable both as a list and as a graph.
- store: new index_coverage table (project, rel_path, kind, detail) —
coverage is metadata ABOUT the graph and never mixes into the graph
tables; replace/get/free APIs with a deleted-file prune keyed off
file_hashes (the live-file set)
- store: materialize a derived miss graph under the shadow project
"<project>::coverage" (Project -> Folder chain -> File{kind,detail},
CONTAINS_FOLDER/CONTAINS_FILE edges), rebuilt from the table inside
the same transaction on every replace; invisible to list_projects
(which scans the cache directory, not the projects table)
- pipeline: coverage is written in the same persist step as every other
post-dump SQL write (hashes, ADR, FTS). Full runs replace the whole
set; incremental runs merge rows surviving from files not re-extracted
with this run's fresh entries, so a fixed file's flag clears and
deleted files prune automatically
- mcp: new get_index_coverage tool (per-project parse_partial + skipped
lists with the best-effort note); query_graph gains graph="coverage"
to run the same cypher against the miss graph; the File-node property
marker from the previous commit is dropped in favor of the separate
table — the code graph gains no coverage rows
- wording: constructs in flagged ranges MAY be missing — tree-sitter
error recovery still salvages some (verified: a function inside a
flagged #ifdef region and even a broken def were still extracted)
- tests: store round-trip/prune/shadow-graph unit test; e2e asserts the
response fields, the advertised coverage-graph query, code-graph
purity, and that the flag clears after the file is fixed (incremental
route)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…subtraction
Four refinements to the coverage signal:
- extraction: subtract DEFINITE recovery before flagging. Tree-sitter
error recovery plus the ERROR-descending def walker often re-extract
constructs inside a failed region (verified: `def broken(:` comes
back as a def); a region whose every line is covered by definitions
that START inside it is not a miss, and a fully recovered file is not
flagged at all. Container defs (Module/Package) don't count as
evidence, and partially covered regions stay flagged — the
#ifdef-split case keeps its flag because the first branch's function
is genuinely lost
- naming: the query_graph option is graph="missed" (the graph shows
ONLY misses — "coverage" was misleading); shadow project renamed to
"<name>::missed"; tool descriptions updated so agents discover both
the option and its semantics
- hook: the CLI-installed PreToolUse augmenter now also matches Read
and injects a coverage note when the file being read is listed as not
fully indexed ("line ranges X-Y could not be parsed — the file
content you are reading is ground truth"). Safe against the old
issue-362 hazard: the augmenter is structurally non-blocking (always
exit 0, additionalContext only), mirroring the Gemini matcher that
already includes read_file; matcher upgrade bookkeeping updated
- ui: "Missed files" toggle in the graph sidebar renders the miss
graph as a second graph option — /api/layout gains graph=missed
(same db file, shadow-project scoping; base project name validated
as before)
Tests: recovery-subtraction cases (recovered def unflagged, garbage
region flagged, trailing recovered defs keep the flag), CLI matcher
tests updated, e2e resilience fixtures switched to an unrecovered
miss; verified end-to-end: hook emits the note for a flagged file and
stays silent otherwise, /api/layout?graph=missed serves the miss graph
and the code graph is unchanged, frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Render the miss graph side by side with the code graph instead of
swapping layouts:
- /api/layout (code graph) now also attaches "missed_graph": {nodes,
edges, offset} — the shadow-project layout placed below the primary
cluster (same satellite pattern as linked_projects; -Y slot so
cross-repo satellites collide last). graph=missed stays the isolated
view for API users
- the UI paints the skeleton white and ghostly beside the galaxy,
auto-frames BOTH clusters on load, and navigates naturally: clicking
the skeleton flies the camera into it (file labels + detail panel),
clicking a code node flies back to the code side, and clicking empty
space while the skeleton has focus returns to the overview (the
galaxy can be entirely off-screen at that point)
- the sidebar toggle now shows/hides the skeleton (with a missed-file
count) rather than swapping the layout
Verified in the browser: overview composition, skeleton focus with
split.c labeled, and the empty-space return flight; ui/httpd suites and
frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Check 8d still locked the matcher to the pre-#963 'Grep|Glob' and failed the pr-smoke leg on all platforms. The matcher now includes Read (the augmenter injects the coverage note when a not-fully-indexed file is read; structurally non-blocking, so the issue-#362 gate hazard cannot recur). The guard now pins the exact new matcher and still rejects Search/catch-all creep. Refs #963 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…_snippet
Per review: no dedicated coverage tool.
- index_status now carries the coverage report (parse_partial + skipped
lists from the separate index_coverage table, plus the best-effort
coverage_note); get_index_coverage is removed. Tool description
updated so agents learn the coverage semantics from index_status;
index_repository and the response note now point there
- the read hook resolves coverage through index_status (same envelope
fields, unchanged note wording); verified end-to-end against a real
index (flagged file -> note with exact ranges, clean file -> silence)
- get_code_snippet responses from a parse_partial file now carry a
correlated coverage_note ("line range(s) X could not be parsed ...
the source above is ground truth") — the result names its file, so
the warning is precisely anchored. Entirely-skipped files cannot
appear here (no nodes), so no uncorrelated noise is possible; clean
files carry no note (asserted both ways in the e2e test)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…ected Clicking a node of the missed skeleton now opens a dedicated right-panel callout instead of the standard node panel (code snippet and callers/callees are meaningless for a not-fully-indexed file): - explains the gap in plain words (best-effort detection; the file content itself is ground truth) - asks the user to have their agent summarize what fails to parse and report it upstream so the edge case can be handled - two working actions: a prefilled upstream GitHub issue (title 'Indexing gap: <file>'; body carries ONLY the file path and project name, with an explicit add-snippets-only-if-shareable note) and a copy-to-clipboard agent prompt (index_status -> summarize flagged ranges -> file the issue) with visible copied feedback Verified in the browser: skeleton click opens the callout with both actions; frontend build + tests green. Refs #963 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI security audit (security-ui.sh, layer A1) forbids hardcoded external URLs in graph-ui source — the callout's GitHub issues link tripped it on all pr-smoke legs. Follow the established pattern for external targets (/api/repo-info deep-links): /api/ui-config now carries upstream_issues_url and the callout consumes it, rendering the issue button only when the backend provides an https URL. The protocol check uses a regex literal on purpose: a bare protocol string in source also aborts the audit's URL extraction. Verified: security-ui.sh passes locally; /api/ui-config serves the URL; the callout opens with the prefilled link in the browser; frontend build + tests green. Refs #963 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the best-effort indexing-coverage signal from #963 (Signal A): files whose parse tree contains tree-sitter ERROR/MISSING regions were silently indexed as if complete — constructs in those regions could be absent from the graph with no signal anywhere. Canonical trigger: the preprocessor-blind
#ifdef-split-brace pattern in C.What it does
Capture (
cbm_extract_file): setsparse_incompleteand records the 1-based line ranges of the top-most error regions (bounded at 64), serialized as"start-end,start-end".Report (
index_repositoryresponse): newparse_partialsummary — distinct fromskipped[](these files WERE indexed;skipped_countsemantics are unchanged). The per-run logfile lists both classes.Persist (separate from the graph): new
index_coveragetable (project, rel_path, kind, detail) written in the same persist step as all other post-dump SQL writes. Coverage is metadata about the graph — the code graph gains no rows. Full runs replace the set; incremental runs merge (fixed file → flag clears; deleted file → pruned againstfile_hashes).Query:
index_status: now carries the coverage report (per-projectparse_partial+skippedlists + best-effort note) — no dedicated tool.query_graph(graph="coverage"): the misses as a file-structure graph (Project → Folder → File{kind, detail}) materialized under a shadow project<name>::coverage— same cypher engine, zero engine changes, invisible tolist_projects.Best-effort framing (deliberate)
A flag means constructs in the listed ranges MAY be missing — prefer grep there. Absence of a flag is NOT a completeness guarantee. Verified empirically that tree-sitter error recovery still salvages some constructs inside flagged ranges (the def walker descends into ERROR subtrees), which is why the wording says MAY.
Tests
tests/test_parse_coverage.c(new, red-first): 7 unit cases incl. false-positive guards and the 64-region cap.tests/test_index_resilience.c: e2e — response fields, logfile, coverage-graph query, code-graph purity, flag-clears-on-fix via the incremental route.tests/test_store_nodes.c: store round-trip / deleted-file prune / shadow-graph materialization + wipe.Local: affected suites 764/764 green (ASan/UBSan),
lint-format+lint-cppcheckclean, prod build clean.Refs #963