fix(parsing): emit CALLS edges and code blocks; repair Neo4j 5 schema DDL - #50
Conversation
Four independent defects meant the call graph and block features produced
nothing, silently, in every language.
1. edge_extractor: `_extract_call_edges` built its candidate-caller list from
an empty list literal (`for n in []`) rather than the `nodes` its caller
already had, so `_find_enclosing_callable` never resolved a caller and no
CALLS edge was ever emitted. The sibling `_extract_containment_edges`
shows the intended pattern verbatim. Threads `nodes` through.
2. block_extractor: `_manual_extract_blocks` started its walk at the tree
root while pruning anything outside the target function's byte range. The
root spans the whole file, so it returned on the first call and no block
was ever extracted. Anchors the walk on the function's own subtree.
3. schema_manager: `CREATE CONSTRAINT` / `CREATE INDEX` emitted the name
after `IF NOT EXISTS`. Neo4j 5 rejects this with
`Invalid input '<name>': expected 'FOR' or 'ON'`, so every constraint and
index silently failed to be created while indexing reported success.
4. api: call traversals matched typed `:CALLS|VIRTUAL_CALL|...` relationships,
but `batch_upsert_edges` writes generic `:EDGE` carrying a `kind` property.
Even with extraction fixed, `find_callers`/`find_callees` returned nothing.
Matches `:EDGE` and filters on `kind`.
Measured on this repo (76 Python files), before -> after:
CALLS edges in graph 0 -> 498
blocks extracted 0 -> 1360
constraint creation errors many -> 0
Intra-file caller lookup scored against an independent oracle built from
Python's stdlib `ast` (a different parser from tree-sitter): precision 0.99,
recall 0.98, F1 0.98 over 15 symbols. Cross-file call resolution remains
unimplemented -- `name_to_id` is built per file -- and is out of scope here.
tests/test_python_parsing.py carried this bug as a strict xfail
("known gap: ... CALLS ... not extracted yet"); it now XPASSes, so the marker
is removed.
Also aligns requirements.txt (`tree-sitter>=0.22,<0.24`) with pyproject
(`>=0.24`), which were mutually exclusive, and makes the `watchdog` import in
services/__init__ lazy so that the documented `pip install -e .` yields a
working CLI instead of ModuleNotFoundError.
Suite: 3 failed, 184 passed (baseline on main: 3 failed, 174 passed).
The 3 failures are pre-existing and unrelated.
|
CI on this PR is red at the Ruff is unpinned and Because lint runs before #51 fixes that and is passing. Once it lands, this PR should go green on a re-run; I've verified all three CI steps locally in the meantime. |
Adds tree-sitter-go and a GO_QUERIES module covering structs, interfaces, functions, methods, struct fields, imports and calls. Go models types as `type_declaration -> type_spec` with the concrete shape on the `type` field, so structs and interfaces are matched on the type_spec rather than on a dedicated node. Methods carry a `receiver` and are a separate node type (method_declaration) from plain functions, so Area() lands as METHOD while describe()/main() land as FUNCTION. The calls query handles both shapes Go uses -- a bare identifier (describe()) and a selector (fmt.Println()) -- capturing the method name for the latter. Query names reuse the existing generic mapping in node_extractor, so no per-language dispatch was needed. Verified on a sample with an interface, struct, method, two functions and both call shapes: INTERFACE Shape / STRUCT Rect / FUNCTION describe, main / METHOD Area FIELD W, H / IMPORTS 1 Call edges additionally require the _extract_call_edges fix from #50; with that applied locally this sample yields exactly the two intra-file edges (describe -> Area, main -> describe) and correctly excludes external calls such as fmt.Println. This PR does not include that fix. test_unsupported_language.py used .go as its example of an unsupported extension, which is no longer true -- switched to .rb. Suite: 3 failed, 188 passed (baseline on main: 3 failed, 174 passed) -- same three pre-existing failures, fixed separately in #51.
|
@r0h1tb for others from |
|
Good pr! |
|
I re-verified it against The issue lays out the two options you'd have to choose between (align readers to |
2 participants
Fixes #49
Problem
The call-graph and block features returned nothing in every language, and no Neo4j 5 constraint or index was ever created. All of it failed silently —
ast-rag callersprintedNo callers found., indexing printed0 blocksandDone, nothing exited non-zero.Four independent defects, each covered in detail in #49.
Root cause and fix
1.
edge_extractor.py:260— candidate callers built from an empty listmethod_nodeswas unconditionally[], so_find_enclosing_callablealways returnedNoneand every call site hitcontinue.extract_edgesalready receivesnodes; it just wasn't passed down (line 84)._extract_containment_edgesat line 145 has the identical comprehension written correctly, so this fix restores the intended shape rather than inventing one.2.
block_extractor.py— walk started at the root it then pruned_manual_extract_blocksbegan attree.root_nodewhile rejecting any node outside the target function's byte range. The root spans the whole file, so it returned on the first call and never descended. Now anchors on the function's own subtree viadescendant_for_byte_range, which also removes the need for the range check inside the walk. Nesting depth is counted from the function, so a top-level block in a function is depth 1.3.
schema_manager.py— invalid Neo4j 5 DDLCREATE CONSTRAINT/CREATE INDEXemitted the name afterIF NOT EXISTS, which Neo4j 5 rejects withInvalid input '<name>': expected 'FOR' or 'ON'. Name now precedes it, in both helpers.4.
ast_rag_api.py— traversals matched relationship types that are never writtenbatch_upsert_edgeswrites(a)-[:EDGE {kind: ...}]->(b), but the call traversals matched:CALLS|VIRTUAL_CALL|LAMBDA_CALL|CROSS_FILE_CALL. Those now match:EDGEand filter onkindvia a newCALL_EDGE_KINDSconstant. Without this, fixing (1) alone still yields no callers.Plus two packaging fixes that block the documented install:
requirements.txtpinnedtree-sitter>=0.22,<0.24whilepyproject.tomlrequires>=0.24— mutually exclusive. Aligned to>=0.24.services/__init__.pyimportedwatcher_serviceeagerly, which importswatchdogat module scope.watchdogships only in themcp/devextras, so the README'spip install -e .produced a CLI that died on--help. The import is now lazy via module__getattr__, which keeps the dependency optional as the extras clearly intend — rather than promoting it to a base dependency.Scope — deliberately left out
name_to_idis per-file, so only same-file calls resolve. On this repo that's 507 intra-file vs 2521 cross-file call relationships, i.e. most of the call graph is still missing. That's a design change, not a bug fix.INHERITS,OVERRIDES,TYPES,CONTAINS_BLOCK,RELATES,CAPTURES) have the same:EDGEmismatch. I only touched the call path because it's the one I could verify end to end. Aligning the rest vs. changing the writer to emit typed relationships is your call — happy to do either.edge_extractor.py(a second copy of ten methods nested inside_add_type_relation_edges, including a second_extract_call_edgeswith the same bug). It's unreachable, so I left it rather than bury this fix in a large reformat. Worth deleting separately.Tests
Three new files, plus one marker removal.
tests/test_python_parsing.py::TestPythonCallEdges::test_method_call_edges_extractedalready encoded this bug as@pytest.mark.xfail(strict=True)— "known gap: ... CALLS ... not extracted yet". It now XPASSes, so the marker is removed. That test was written against the symptom; the new tests pin the cause.On this branch with the source changes stashed but the tests kept, 9 tests fail:
with the literal message on the first being:
The watchdog tests were also run with
watchdogactually uninstalled, not just mocked —ast-rag --helpworks, andservices.WorkspaceWatcherraises a clearModuleNotFoundErroronly when accessed.Verification
Full suite:
main, dev extras installed)Same three failures before and after, all pre-existing and unrelated:
test_update_project_dry_run.py::test_update_project_dry_runand::test_search_by_signature_format—ModuleNotFoundError: No module named 'mcp.server.fastmcp'(FastMCP moved in newermcpreleases)test_unsupported_language.py::test_supported_extensions_grouped_by_language— expectstypescript -> ['.ts', '.tsx'], gets['.ts']; a leftover from cd75313 moving.tsxto a dedicated grammarThe xfail count drops 2 → 1 because the previously-xfailing call-edge test now passes.
End to end, re-indexing this repo against Neo4j 5 + Qdrant:
CALLSedges in graphFor retrieval quality I built an independent oracle from Python's stdlib
ast— a different parser from tree-sitter — so the system is not graded against its own output, which is the flaw inbenchmarks/create_ground_truth.py. Scoring intra-file caller lookup over the 15 most-called single-definition symbols:Before the fix this is not 0.98 vs some lower number — there were no
CALLSedges in the database at all, so the feature had no output to score.ruff checkis clean on the files I touched (the repo has ~563 pre-existing findings elsewhere, which I left alone), andruff formatis clean on all changed files.One thing that looks odd, flagged deliberately
I removed a
@pytest.mark.xfail(strict=True)from an existing test. That is normally a smell — it can hide a regression. Here the marker's own reason states the gap this PR closes, andstrict=Truemeans the suite fails while the marker stays, so it has to go. The behaviour it asserts is additionally covered by the newtests/test_call_edges.py, in both Python and Java.Happy to split this into separate PRs per defect, or to drop the packaging changes into their own, if you'd prefer smaller reviews.