From 123b74731ee2b953f3d6859663a19b7d28bdb299 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 01:43:02 +0300 Subject: [PATCH 01/16] =?UTF-8?q?docs(plan):=20generated-source=20indexing?= =?UTF-8?q?=20=E2=80=94=20design=20proposal=20+=20implementation=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Proposal: classify generated Java sources (content-based, file-level), tag both the Lance chunk index and graph Symbol nodes, surface the tag on results, and add exclude_generated/generated_only filters — equal-treatment default, no ranking penalty (role model already down-ranks DTOs/mappers). Plan: 6 TDD tasks (detect, tag Lance, tag graph, surface, filter, docs). Co-Authored-By: Claude --- .../active/PLAN-GENERATED-SOURCE-INDEXING.md | 200 ++++++++++++++++++ propose/GENERATED-SOURCE-INDEXING-PROPOSE.md | 174 +++++++++++++++ 2 files changed, 374 insertions(+) create mode 100644 plans/active/PLAN-GENERATED-SOURCE-INDEXING.md create mode 100644 propose/GENERATED-SOURCE-INDEXING-PROPOSE.md diff --git a/plans/active/PLAN-GENERATED-SOURCE-INDEXING.md b/plans/active/PLAN-GENERATED-SOURCE-INDEXING.md new file mode 100644 index 0000000..2947b6c --- /dev/null +++ b/plans/active/PLAN-GENERATED-SOURCE-INDEXING.md @@ -0,0 +1,200 @@ +# Generated-Source Indexing Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make generated Java sources a first-class, tagged, filterable dimension — fully retrievable, with no ranking penalty — by detecting them (content-based, file-level), tagging both the Lance chunk index and the graph `Symbol` nodes, surfacing the tag on every search/edge result, and adding `exclude_generated` / `generated_only` filters mirroring the existing `role` / `exclude_roles` pair. + +**Architecture:** A single shared, config-aware per-file classifier in `graph_enrich.py` decides whether a file is generated and by which generator family. The Lance flow computes it once per file and stores it on chunks; the graph flow computes it once per file and stores it on `Symbol` nodes. Both read paths surface the tag and accept the same two boolean filters, threaded through the Lance SQL predicate builder and the Kùzu Cypher + Python post-filter exactly as `role` / `exclude_roles` are today. + +**Tech Stack:** Python 3, CocoIndex + LanceDB (vector index), Kùzu/LadybugDB (graph), tree-sitter-java AST (`ast_java.py`), FastMCP + pydantic (MCP tools), PyYAML (config), pytest. + +## Global Constraints + +- **Python env:** use `.venv/bin/python` and `.venv/bin/pip` (repo root) only — never system `python`/`pip`. Editable install only; if `jrag`/`java-codebase-rag` serve stale behavior while pytest passes, run `.venv/bin/pip install -e .`. +- **Test indexes:** erase stale manual indexes first (`rm -rf tests/*/.java-codebase-rag tests/*/.java-codebase-rag.{yml,hosts}`); tests build a fresh index in a temp dir; never commit an index under `tests/`. +- **Backward compatibility:** every new Lance column is added to `JAVA_ENRICHED_COLUMNS` (`search_lancedb.py:29`) so the schema-presence guard at `search_lancedb.py:543` SELECTs it only when present — old indexes without the column keep working with no migration beyond a reprocess. +- **Ontology version:** adding columns/properties is an extraction/enrichment semantic change — bump `ONTOLOGY_VERSION` (`ast_java.py:87`, currently `17`) to `18`. This drives re-indexing. Bump once (Task 2). +- **Mirror existing patterns exactly:** detection mirrors `module_for_path`/`microservice_for_path` (per-file shared helpers in `graph_enrich.py`); tagging mirrors the `capabilities` column addition; filtering mirrors `role` / `exclude_roles` end-to-end through BOTH the Lance SQL path and the Kùzu Cypher + Python post-filter path. +- **Equal-treatment default:** generated sources are NEVER ranked down or excluded from graph fan-out by default. Only the opt-in `exclude_generated` / `generated_only` filters change behavior. Do not touch `_ROLE_SCORE_WEIGHTS` or scoring. +- **Plans carry design, not code:** cite signatures, data shapes, and field names; do not write method bodies, algorithms, or test/impl code. + +--- + +## File Structure + +| File | Responsibility | Touched in | +|---|---|---| +| `graph_enrich.py` | Shared per-file classifier `classify_java_file`, config dataclass + loader | Task 1; consumed in 2, 3 | +| `java_index_flow_lancedb.py` | Lance chunk schema + write path | Task 2 | +| `search_lancedb.py` (write/col side) | `JAVA_ENRICHED_COLUMNS` registry | Task 2 | +| `ast_java.py` | `ONTOLOGY_VERSION` bump | Task 2 | +| `build_ast_graph.py` | Graph `Symbol` schema + node write path | Task 3 | +| `search_lancedb.py` (filter/read side) | `_build_extra_predicates`, `run_search`, CLI flags, hint printer | Task 5 | +| `mcp_v2.py` | `NodeFilter`, applicability map, Cypher + post-filter, `SearchHit`, `_row_to_search_hit`, `NodeRecord`/`find` projections | Tasks 4, 5 | +| `graph_types.py` | `NodeRef`, `_node_ref_from_row` | Task 4 | +| `ladybug_queries.py` | graph-side `Symbol` RETURN projections | Task 4 | +| `docs/` | operator + agent docs | Task 6 | +| `tests/` | per-task fixtures + tests | each task | + +--- + +## Task 1: Detection core + extensible config + +**Files:** +- Modify: `graph_enrich.py` (add `GeneratedDetectionConfig` dataclass ~line 228 area, `load_generated_detection` cached loader mirroring `_load_config_microservice_roots:113-141`, and `classify_java_file` near `module_for_path:1481`). +- Test: `tests/test_generated_detection.py` (create). +- Test fixtures: `tests/fixtures/generated_samples/` (create: one `.java` per generator family + one hand-written). + +**Interfaces:** +- Consumes: `JavaFileAst` (defined `ast_java.py:389` — note `source_bytes` is an `int` byte COUNT, not raw bytes) and its `all_types: list[TypeDecl]`; each `TypeDecl.annotations: list[AnnotationRef]` where `AnnotationRef` (`ast_java.py:241`) has `.name`, `.qualified`, and `.arguments: dict[str,str]` (a `@Generated(value="org.openapitools...")` already parses to `arguments["value"]`). Consumes raw file `source: bytes` for header-banner detection (not derivable from `JavaFileAst`). +- Produces: + - `GeneratedDetectionConfig` dataclass (frozen, `field(default_factory=...)`): `header_patterns: list[str]`, `annotation_patterns: list[str]`, `force_fqns: set[str]`, `exclude_fqns: set[str]`. + - `load_generated_detection(project_root: str | Path | None) -> GeneratedDetectionConfig` — `@lru_cache(maxsize=64)`-per-root, reads `generated_detection` from the first matching `CONFIG_FILENAMES` (`graph_enrich.py:89`); returns empty config when section absent or `project_root is None`; stderr-warns + drops on malformed entries (mirror `_load_config_microservice_roots`). + - `classify_java_file(source: bytes, ast: JavaFileAst, *, config: GeneratedDetectionConfig | None = None, project_root: str | Path | None = None) -> tuple[bool, str | None]` — returns `(generated, generated_by)`. `generated_by` is a lowercased family slug from the v1 set (`openapi`, `jsonschema2pojo`, `protobuf`, `mapstruct`, `wsimport`, `querydsl`, `jooq`, `immutables`, `autovalue`) or `None`. + +**Detection behavior (design, not code):** +- A file is `generated=True` if ANY of: + 1. Any type in `ast.all_types` carries an annotation whose simple name is `Generated` (or `javax`/`jakarta.annotation.processing.Generated` / `org.immutables.Generated` / `lombok.Generated` / `com.squareup.javapoet...` — the v1 marker set, verified at impl time) — `generated_by` inferred from the annotation's `arguments["value"]`/`arguments["comments"]` when it names a known generator, else `None`. + 2. The file header (a bounded prefix of `source`, e.g. first 4 KB before the first type) matches a built-in generator banner pattern (OpenAPI, jsonschema2pojo, protobuf, wsimport, MapStruct default headers) — `generated_by` = matched family. + 3. A type's FQN is in `config.force_fqns`. +- A file is forced `generated=False` if a type's FQN is in `config.exclude_fqns` (wins over 1–3). +- `config.header_patterns` / `config.annotation_patterns` extend the built-in sets (treated as additional markers). + +- [ ] **Step 1: Write failing tests** — scenarios + expected results: + - `test_openapi_annotation` — source+ast where a type has `@Generated(value="org.openapitools.codegen...")` → `(True, "openapi")`. + - `test_javax_generated_no_value` — `@Generated` with no value → `(True, None)`. + - `test_jsonschema2pojo_header` — source bytes with a jsonschema2pojo banner, ast with no `@Generated` → `(True, "jsonschema2pojo")`. + - `test_protobuf_header` — protobuf-generated banner → `(True, "protobuf")`. + - `test_handwritten_is_not_generated` — plain hand-written type, no markers → `(False, None)`. + - `test_config_force_fqn` — `GeneratedDetectionConfig(force_fqns={"com.example.Model"})`, ast with that FQN, no markers → `(True, None)`. + - `test_config_exclude_fqn_wins` — `@Generated` present but FQN in `exclude_fqns` → `(False, None)`. + - `test_config_extra_header_pattern` — custom `header_patterns` matches an internal codegen banner → `(True, None)`. +- [ ] **Step 2: Run tests, verify FAIL** — `pytest tests/test_generated_detection.py -v` → FAIL (names undefined). +- [ ] **Step 3: Implement** — add `GeneratedDetectionConfig`, `load_generated_detection`, and `classify_java_file` per the Produces contracts and detection behavior above. Verify exact marker strings per family against real generator output (annotate the v1 marker table with a short comment of the verified strings). +- [ ] **Step 4: Run tests, verify PASS** — `pytest tests/test_generated_detection.py -v` → PASS. +- [ ] **Step 5: Commit** — `git add graph_enrich.py tests/test_generated_detection.py tests/fixtures/generated_samples/` → `git commit -m "feat(detection): classify generated Java sources (content + extensible config)"`. + +--- + +## Task 2: Tag Lance chunks + ontology bump + +**Files:** +- Modify: `java_index_flow_lancedb.py` — `JavaLanceChunk` dataclass (`:260-282`), `process_java_file` row construction (`:444-466`; `rel` at `:427`, `content_bytes` at `:428`). +- Modify: `search_lancedb.py` — `JAVA_ENRICHED_COLUMNS` (`:29-42`). +- Modify: `ast_java.py` — `ONTOLOGY_VERSION` (`:87`): `17` → `18`. +- Test: `tests/test_lancedb_generated_column.py` (create). + +**Interfaces:** +- Consumes: `classify_java_file` (Task 1), the parsed `JavaFileAst` already produced inside `process_java_file` before the chunk/enrich loop, and `content_bytes`. +- Produces: `JavaLanceChunk` gains two scalar fields (no `LanceType` override needed — plain scalars): `generated: bool` and `generated_by: str | None`. CocoIndex derives the Lance write schema from the dataclass (`java_index_flow_lancedb.py:593-602`), so no separate schema edit. Read side: `"generated"` and `"generated_by"` appended to `JAVA_ENRICHED_COLUMNS` (the presence guard at `:543` auto-SELECTs them only on indexes that have them). + +- [ ] **Step 1: Write failing test** — scenario + expected: index a fixture with one generated file (e.g. an OpenAPI DTO) and one hand-written file; query the Lance java table directly; assert every chunk of the generated file has `generated == True` and `generated_by == "openapi"`, every chunk of the hand-written file has `generated == False`/`None`, and the columns exist in the table schema. +- [ ] **Step 2: Run test, verify FAIL** — `pytest tests/test_lancedb_generated_column.py -v` → FAIL (fields absent). +- [ ] **Step 3: Implement** — (a) bump `ONTOLOGY_VERSION` to `18`; (b) add `generated`/`generated_by` fields to `JavaLanceChunk`; (c) in `process_java_file`, compute the classification ONCE after the AST is parsed and `content_bytes` is available, then pass both values into the `JavaLanceChunk(...)` constructor in the row loop (do NOT thread through `ChunkEnrichment` — it is a per-file value, like `module`/`microservice`); (d) append `"generated"`, `"generated_by"` to `JAVA_ENRICHED_COLUMNS`. +- [ ] **Step 4: Run test, verify PASS** — `pytest tests/test_lancedb_generated_column.py -v` → PASS. Also run a search smoke (`search_lancedb`) against the fixture to confirm no regression. +- [ ] **Step 5: Commit** — `git add java_index_flow_lancedb.py search_lancedb.py ast_java.py tests/test_lancedb_generated_column.py` → `git commit -m "feat(index): tag Lance chunks generated/generated_by; bump ontology to 18"`. + +--- + +## Task 3: Tag graph Symbol nodes + +**Files:** +- Modify: `build_ast_graph.py` — `_SCHEMA_NODE` DDL (`:2901`), `_NODE_COLUMNS` (`:3138`), `_node_row` defaults (`:3083`), `_SET_SYMBOL_BY_ID` (`:3155`), `_write_nodes_impl` type loop (`:3236-3268`, resolver call `:3238`), pass-1 per-file site (`:1031-1077`, where `module_for_path`/`microservice_for_path` are called at `:1060-1061`), `_load_existing_types` (`:584-631`, mirror `type_role_by_node_id` seed at `:631`). +- Test: `tests/test_graph_generated_node.py` (create). + +**Interfaces:** +- Consumes: `classify_java_file` (Task 1), `TypeDecl`/`JavaFileAst` and raw file bytes available in pass 1. +- Produces: the `Symbol` node table gains columns `generated BOOLEAN` and `generated_by STRING`; persisted via the bulk-COPY path and the incremental `_SET_SYMBOL_BY_ID` upsert. A new `tables.type_generated_by_node_id: dict[str, tuple[bool, str]]` field (on `GraphTables`, mirror `type_role_by_node_id` at `:467`) carries the per-type classification from pass 1 to `_write_nodes_impl`. + +**Stub discipline (critical):** `loaded_from_db` stubs (`:3243-3252`) build a bare `TypeDecl(name, kind, fqn)` with NO annotations, so re-detection from a stub decl is wrong. Mirror the `type_role_by_node_id` seed pattern: seed `type_generated_by_node_id` in `_load_existing_types` (reading the persisted `generated`/`generated_by` columns) so preserved stubs retain their stored values, exactly as preserved stubs retain `role`. + +- [ ] **Step 1: Write failing tests** — scenarios + expected: + - Build a graph from a fixture with a generated type → query the `Symbol` node → assert `generated == True`, `generated_by` set. + - A hand-written type in the same fixture → `generated == False`. + - Incremental rebuild (re-run on an already-built graph with one file changed) → a preserved (unchanged) generated type retains its `generated`/`generated_by` (stub path). +- [ ] **Step 2: Run tests, verify FAIL** — `pytest tests/test_graph_generated_node.py -v` → FAIL (columns absent). +- [ ] **Step 3: Implement** — (a) add `generated BOOLEAN, generated_by STRING` to `_SCHEMA_NODE`; append `"generated"`, `"generated_by"` to `_NODE_COLUMNS`; add `generated`/`generated_by` defaults to `_node_row`; append the two SET clauses to `_SET_SYMBOL_BY_ID`; (b) add `type_generated_by_node_id` to `GraphTables`; (c) in pass 1, compute `classify_java_file(...)` once per file and seed `type_generated_by_node_id` for each type in the file; (d) in `_write_nodes_impl` type loop, read from `type_generated_by_node_id` for the persisted/stub value (else from a fresh `classify_java_file` call) and pass `generated=`/`generated_by=` into `_node_row(...)`; (e) seed `type_generated_by_node_id` in `_load_existing_types`. +- [ ] **Step 4: Run tests, verify PASS** — `pytest tests/test_graph_generated_node.py -v` → PASS. +- [ ] **Step 5: Commit** — `git add build_ast_graph.py tests/test_graph_generated_node.py` → `git commit -m "feat(graph): tag Symbol nodes generated/generated_by"`. + +--- + +## Task 4: Surface the tag on every result + +**Files:** +- Modify: `mcp_v2.py` — `SearchHit` model (`:466`), `_row_to_search_hit` (`:610`), `_load_node_record` symbol projection (`:683-684`), `find(symbol)` RETURN projection (`:1003-1004`). +- Modify: `graph_types.py` — `NodeRef` model (`:36`), `_node_ref_from_row` (`:128`). +- Modify: `ladybug_queries.py` — `_symbol_return_for` (`:209`), `_SYM_COLS` (`:299`), inline INJECTS `s_proj`/`t_proj` (`:1110`, `:1117`). +- Modify: `search_lancedb.py` — CLI hint printer (`:1206-1214`). +- Test: `tests/test_generated_surface.py` (create). + +**Interfaces:** +- Consumes: the new Lance columns (Task 2) and graph `Symbol` properties (Task 3). +- Produces: `SearchHit`, `NodeRef`, and `NodeRecord.data` each gain `generated: bool | None` and `generated_by: str | None` fields; graph RETURN projections emit `s.generated AS generated, s.generated_by AS generated_by` so the row populators can read them; CLI output prints ` | generated` / ` | generated:` when set. + +- [ ] **Step 1: Write failing tests** — scenarios + expected: against a fixture with a generated type, (a) `search` returns a `SearchHit` for a generated chunk with `generated=True`/`generated_by` set; (b) `find(symbol)` returns a `NodeRef` with the fields; (c) `describe` includes `generated`/`generated_by` in `NodeRecord.data`; (d) `neighbors` endpoint `NodeRef` carries them; (e) CLI `search` output prints the `generated` hint. +- [ ] **Step 2: Run tests, verify FAIL** — `pytest tests/test_generated_surface.py -v` → FAIL (fields absent). +- [ ] **Step 3: Implement** — add the two fields to `SearchHit`, `NodeRef`; populate them in `_row_to_search_hit` and `_node_ref_from_row` from the row; add `n.generated AS generated, n.generated_by AS generated_by` to the `_load_node_record` and `find(symbol)` projections; add `s.generated`/`s.generated_by` to `ladybug_queries.py` projections; add the CLI hint line mirroring the `role` hint block. +- [ ] **Step 4: Run tests, verify PASS** — `pytest tests/test_generated_surface.py -v` → PASS. +- [ ] **Step 5: Commit** — `git add mcp_v2.py graph_types.py ladybug_queries.py search_lancedb.py tests/test_generated_surface.py` → `git commit -m "feat(mcp): surface generated/generated_by on search/find/describe/neighbors"`. + +--- + +## Task 5: `exclude_generated` / `generated_only` filters + +**Files:** +- Modify: `search_lancedb.py` — `_build_extra_predicates` signature (`:74-86`) + predicate block (insert after `:117`), `run_search` signature (`:936`) + kwargs call (`:988`), CLI flags (`:1116`) + pass-through (`:1162`). +- Modify: `mcp_v2.py` — `NodeFilter` (`:193`), `_NODEFILTER_APPLICABLE_FIELDS["symbol"]` (`:274-275`), `_symbol_where_from_filter` (`:651`), `_node_matches_filter` symbol block (`:780`). +- Test: `tests/test_generated_filter.py` (create). + +**Interfaces:** +- Consumes: the `generated` Lance column (Task 2) and graph `Symbol.generated` (Task 3). +- Produces: two boolean filter params threaded through BOTH engines, exactly mirroring `role`/`exclude_roles`: + - Lance SQL: `exclude_generated=True` → predicate `(generated IS NULL OR generated = false)`; `generated_only=True` → `generated = true` (each guarded `if flag and "generated" in columns`). + - Kùzu Cypher (`_symbol_where_from_filter`): the same two predicates against `s.generated`. + - Python post-filter (`_node_matches_filter`): same two checks on `row.get("generated")` for route/client/producer lists and neighbor endpoints. + - `NodeFilter` gains `generated_only: bool = False` and `exclude_generated: bool = False` (declared — `NodeFilter` is `extra="forbid"`); both listed in `_NODEFILTER_APPLICABLE_FIELDS["symbol"]`. + - CLI gains `--exclude-generated` and `--generated-only` (`action="store_true"`). +- Note: do NOT add the flags to the scoring `skip_role_weight` line (`:992`) — they are not rank-affecting. + +- [ ] **Step 1: Write failing tests** — scenarios + expected (fixture with one generated + one hand-written type): + - `run_search(..., exclude_generated=True)` → no generated chunks returned; hand-written present. + - `run_search(..., generated_only=True)` → only generated chunks returned. + - default (neither flag) → both returned. + - `find(symbol, filter=NodeFilter(exclude_generated=True))` → generated `NodeRef` excluded (Cypher path). + - `find(symbol, filter=NodeFilter(generated_only=True))` → only generated (post-filter path). +- [ ] **Step 2: Run tests, verify FAIL** — `pytest tests/test_generated_filter.py -v` → FAIL (params rejected by `extra="forbid"` / undefined). +- [ ] **Step 3: Implement** — mirror the `exclude_roles` end-to-end trace exactly: Lance predicate builder + `run_search` + CLI; `NodeFilter` field + applicability tuple + Cypher builder + Python post-filter. +- [ ] **Step 4: Run tests, verify PASS** — `pytest tests/test_generated_filter.py -v` → PASS. +- [ ] **Step 5: Commit** — `git add search_lancedb.py mcp_v2.py tests/test_generated_filter.py` → `git commit -m "feat(search): exclude_generated / generated_only filters"`. + +--- + +## Task 6: Docs — supersede manual-ignore guidance + +**Files:** +- Modify: `docs/CODEBASE_REQUIREMENTS.md` (`:62-66`, `:444-466` — replace "ignore generated manually" with "generated sources are auto-tagged; filter with `exclude_generated`"). +- Modify: `docs/CONFIGURATION.md` — document the `generated_detection` YAML section (mirror the `role_overrides` doc). +- Modify: `docs/JAVA-CODEBASE-RAG-CLI.md` — document the reprocess requirement for `generated`/`generated_by` on existing indexes, and the `--exclude-generated` / `--generated-only` flags. +- Modify: `docs/AGENT-GUIDE.md` — agent guidance: generated types now carry a `generated` tag on results; use `exclude_generated` when you want hand-written code only; generated sources are included by default. +- Test: no unit test; verify `java-codebase-rag --help` (or the search CLI help) lists the new flags, and that the docs render. + +- [ ] **Step 1: Update `CODEBASE_REQUIREMENTS.md`** — remove the "add `**/generated/**` yourself" guidance; state generated sources are detected + tagged and filterable. +- [ ] **Step 2: Update `CONFIGURATION.md`** — add a `generated_detection` subsection with the four keys (`header_patterns`, `annotation_patterns`, `force_fqns`, `exclude_fqns`) and an example. +- [ ] **Step 3: Update `JAVA-CODEBASE-RAG-CLI.md`** — reprocess note + the two CLI flags. +- [ ] **Step 4: Update `AGENT-GUIDE.md`** — the tag + filter guidance for agents. +- [ ] **Step 5: Verify + commit** — confirm CLI help shows the flags; `git add docs/` → `git commit -m "docs: generated-source detection, tagging, and filtering"`. + +--- + +## Self-Review (run after writing) + +1. **Code scan:** no method bodies / algorithms / test-or-impl code — only signatures, data shapes, field names, behavior descriptions. ✓ +2. **Self-containment:** each task's Consumes/Produces carry the full contract; no "see spec" pushes. ✓ (Task 1 defines the classifier contract consumed by 2 & 3; Task 2/3 define the stored shapes consumed by 4/5.) +3. **Spec coverage:** detect (T1) ✓, tag both indexes (T2 Lance, T3 graph) ✓, surface (T4) ✓, filter (T5) ✓, migration/reprocess (T2 ontology bump + T6 doc) ✓, extensible config (T1) ✓. +4. **Placeholder scan:** marker strings are explicitly "verified at impl time" (an open question in the spec, not a placeholder) with the v1 family set named; no TBD/TODO. +5. **Type consistency:** `generated: bool` / `generated_by: str | None` used uniformly across `JavaLanceChunk`, `Symbol`, `SearchHit`, `NodeRef`, `NodeRecord`; `classify_java_file(...) -> tuple[bool, str | None]` consistent in T1, T2, T3; `type_generated_by_node_id: dict[str, tuple[bool, str]]` consistent in T3. + +## Open question carried from spec + +Exact `@Generated`/header marker strings per generator family are verified at Task 1 implementation time (annotation FQNs + banner regexes). Some families (certain jsonschema2pojo / older protobuf configs) emit only a header comment and no `@Generated` — the design accounts for both signal types. diff --git a/propose/GENERATED-SOURCE-INDEXING-PROPOSE.md b/propose/GENERATED-SOURCE-INDEXING-PROPOSE.md new file mode 100644 index 0000000..2eb3238 --- /dev/null +++ b/propose/GENERATED-SOURCE-INDEXING-PROPOSE.md @@ -0,0 +1,174 @@ +# Generated-Source Indexing: Classify, Tag, Filter — Don't Penalize + +## Status + +**Active** — in design (2026-07-08). Make generated sources (OpenAPI, +jsonschema2pojo, protobuf, MapStruct, wsimport, QueryDSL, JOOQ, Immutables, +AutoValue, …) a first-class, **tagged** dimension: fully retrievable (closes +the agent dead-end when resolving a generated type), **filterable on demand**, +with **no ranking penalty by default**. Generated code competes on its existing +role merits; the agent gets the metadata and a filter to decide. + +A companion implementation plan will live at +`plans/active/PLAN-GENERATED-SOURCE-INDEXING.md`. + +## Problem Statement + +The indexer has no concept of "generated sources." File discovery is a +filesystem walk + `*.java` filter + gitignore-style rules +(`path_filtering.py:437-477`); `pom.xml` / `build.gradle` are marker files +only, never parsed for source directories or processor output. So whether +generated code is indexed depends entirely on where it sits on disk: + +- **Build-output-generated** (`target/generated-sources/`, `build/generated/`) + is dropped incidentally — pruned as build output + (`path_filtering.py:51-89`), not because the tool recognizes generation. +- **Committed generated `.java`** (vendored OpenAPI clients, jsonschema2pojo + models, delombok output, a `generated/` package under `src/main/java/`) + **is indexed with no classification** — indistinguishable from hand-written + code. + +The committed case is the real gap, with two failure modes: + +1. **Agent dead-end (retrieval).** During exploration an agent sees a generated + type referenced (an OpenAPI `OrderResponse`, a jsonschema2pojo model) and + tries to resolve its source. Today it either finds nothing (code under a + pruned build dir) or finds an unclassified file it cannot tell apart from + hand-written code. There is no signal that this is generated. + +2. **Unclassified noise.** Generated code participates in the graph and search + exactly like hand-written code, with no metadata to filter or reason about. + +### What the existing ranking already handles + +The role-aware ranking model already mitigates the *ranking-dilution* concern +for the common case. `search_lancedb.py:193` defines `_ROLE_SCORE_WEIGHTS` +("Positive values favour actionable roles"); the hybrid score (line 211) is +`raw_rrf * import_factor + role_weight + symbol_bonus + …`, and `_role_weight` +(line 323) returns `0.0` for any role not in the map. Non-actionable roles — +DTOs, mappers, value objects — already get zero boost and rank below +services/controllers. **Most generated code (DTOs, mappers) is therefore +already down-ranked by its role.** A separate `generated` ranking penalty +would double-count the same signal. + +This is why the design below adds **no ranking penalty and no graph fan-out +exclusion** for generated sources: the role model covers ranking, and the rest +is YAGNI (see Non-goals). + +## Proposed Solution + +Four pieces: **detect, tag, surface, filter.** Default behavior is +**equal treatment** — generated sources rank and traverse exactly like +hand-written code; only the tag and an opt-in filter change anything. + +### 1. Detection — content-based, file-level, shared predicate + +A single classification predicate, called by **both** the vector flow and the +graph build, decides per file whether it is generated and which generator +family produced it. One function, no duplication. + +- **Signals:** `@Generated` annotations + (`javax.annotation.processing.Generated`, `jakarta.annotation.processing.Generated`, + and generator-specific equivalents) **and** built-in header-comment patterns + for the common generators. `generated_by` is inferred from the annotation + value / header when identifiable; otherwise `generated=true, generated_by=null`. +- **File-level:** a generated file tags all its chunks and all its types. + (Intra-file / type-level precision is a non-goal.) +- **Content-based, not path-based:** deliberately avoids path heuristics + (`generated/`, `generated-sources/`) so real packages named `generated` are + never misclassified — consistent with why `target` / `build` / `out` are + pruned *conditionally* (`path_filtering.py:30-37`). + +**Target generator families (v1):** openapi, jsonschema2pojo, protobuf, +mapstruct, wsimport, querydsl, jooq, immutables, autovalue. *Exact marker +strings per family are verified at implementation time, not asserted here.* + +**Extensible config** — mirror the existing `role_overrides` config-loading +mechanism (`graph_enrich.py:396-414` reads a section from +`.java-codebase-rag.yml`). Add a `generated_detection` section: + +```yaml +generated_detection: + header_patterns: # extra patterns matched against the file header + - "// MyInternalCodeGen .*" + annotation_patterns: # extra @-annotation simple names treated as generated + - "MyGeneratedMarker" + force_fqns: # explicit per-FQN override (mark generated) + - "com.example.vendored.Model" + exclude_fqns: [] # explicit per-FQN override (mark NOT generated) +``` + +### 2. Data model — tag both indexes + +`generated` is **orthogonal to `role`**: a generated type can be a DTO, a +mapper, or a service. It is *not* folded into the role system. + +- **LanceDB chunks** — add to the Java chunk schema (the `JavaLanceChunk` / + `JAVA_ENRICHED_COLUMNS` path, same shape as the `capabilities` column + addition): + - `generated: bool` + - `generated_by: string | null` +- **Graph nodes** — add `generated` and `generated_by` properties on type + nodes, set during AST graph build (`build_ast_graph.py`). + +### 3. Retrieval & filtering behavior — equal treatment by default + +- **Default:** generated sources rank and traverse exactly like hand-written + code. +- **Surfaced:** every MCP search/edge result row carries `generated` and + `generated_by`, so the agent can reason about / filter client-side. +- **Server-side filter params** (mirror `role` / `exclude_roles`, + `search_lancedb.py:82-83, 115-117`): + - `exclude_generated: bool` (default `false`) → `WHERE generated = false` + - `generated_only: bool` (default `false`) → `WHERE generated = true` + (optional, symmetric) +- Exposed on the search tool **and** edge queries for symmetry. Server-side + filtering preserves result budget (a top-20 request returns 20 non-generated + rows, not 20-then-discard). + +### 4. Indexing & migration + +Adding the columns/properties is a schema change. Existing indexes require a +**reprocess** (the reprocess path already exists) to populate `generated` / +`generated_by`. Old chunks default to `generated=false, generated_by=null` +until reprocessed — acceptable because detection is purely additive. This +reprocess requirement is documented in the operator CLI docs +(`docs/JAVA-CODEBASE-RAG-CLI.md`). + +## Non-goals (deferred until a measured problem appears) + +- **Ranking down-weight** for generated sources (the role model already covers + the common DTO/mapper case). +- **Graph fan-out exclusion** for generated nodes (role ranking is search-side; + no evidence it is needed for traversal; generated DTOs are shallow leaves). +- **Type-level (intra-file) detection precision.** +- **`generated_by`-based filtering** (the data is stored; a + `generated_by="openapi"` filter is a trivial later addition). +- **A behavioral opt-in/out flag** that changes default visibility. +- **Generated-aware diagnostics** (e.g. a `diagnose-generated` report). + +## Open questions + +1. **Exact detection markers** per generator family — verified at + implementation (annotation FQNs + header patterns). May reveal families + that emit only a header comment and no `@Generated` (some jsonschema2pojo + / older protobuf configs). +2. **`generated_by` normalization** — store the raw generator string, map to a + family best-effort, default to `null`. Settled at implementation. + +## References + +- `path_filtering.py:437-477` (file walker), `:38-47` + (`COMMON_EXCLUDED_PATH_PATTERNS`), `:51-89` (conditional build-dir prune). +- `search_lancedb.py:193` (`_ROLE_SCORE_WEIGHTS`), `:211` (hybrid score), + `:323` (`_role_weight`), `:82-83, 115-117` (`role` / `exclude_roles` filter + pattern to mirror). +- `java_index_flow_lancedb.py` (CocoIndex flow, Java chunk schema — where the + `generated` / `generated_by` columns are added). +- `build_ast_graph.py` (AST graph build — where type-node `generated` + properties are set). +- `graph_enrich.py:396-414` (`role_overrides` config-loading pattern to mirror + for `generated_detection`). +- `docs/CODEBASE_REQUIREMENTS.md:62-66, 444-466` (existing operator guidance + telling users to ignore generated code manually — this proposal supersedes + that workaround). From bb46e5f79ce3da138430c486d0275cc5ef753d53 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 01:51:07 +0300 Subject: [PATCH 02/16] feat(detection): classify generated Java sources (content + extensible config) --- graph_enrich.py | 290 ++++++++++++++++++ .../generated_samples/HandWritten.java | 25 ++ .../generated_samples/JavaxGenerated.java | 19 ++ .../generated_samples/JsonSchemaPojo.java | 28 ++ .../generated_samples/OpenAPIModel.java | 31 ++ .../generated_samples/ProtobufMessage.java | 27 ++ tests/test_generated_detection.py | 159 ++++++++++ 7 files changed, 579 insertions(+) create mode 100644 tests/fixtures/generated_samples/HandWritten.java create mode 100644 tests/fixtures/generated_samples/JavaxGenerated.java create mode 100644 tests/fixtures/generated_samples/JsonSchemaPojo.java create mode 100644 tests/fixtures/generated_samples/OpenAPIModel.java create mode 100644 tests/fixtures/generated_samples/ProtobufMessage.java create mode 100644 tests/test_generated_detection.py diff --git a/graph_enrich.py b/graph_enrich.py index 23ec51c..343b205 100644 --- a/graph_enrich.py +++ b/graph_enrich.py @@ -141,6 +141,130 @@ def _load_config_microservice_roots(project_root_str: str) -> tuple[str, ...]: return () +@lru_cache(maxsize=64) +def load_generated_detection(project_root_str: str | None) -> GeneratedDetectionConfig: + """Read `generated_detection` from `.java-codebase-rag.yml` at project_root. + + Cached per project_root to avoid re-reading on every chunk. Returns empty + config when section absent or project_root is None. Malformed entries + (wrong types, non-string values) are dropped with a stderr warning. + """ + if project_root_str is None: + return GeneratedDetectionConfig() + + root = Path(project_root_str) + for name in CONFIG_FILENAMES: + candidate = root / name + if not candidate.is_file(): + continue + try: + import yaml # PyYAML; already a transitive dep of cocoindex + except ImportError: + return GeneratedDetectionConfig() + try: + data = yaml.safe_load(candidate.read_text(encoding="utf-8")) + except Exception: + return GeneratedDetectionConfig() + if not isinstance(data, dict): + return GeneratedDetectionConfig() + + raw = data.get("generated_detection") + if raw is None: + return GeneratedDetectionConfig() + + if not isinstance(raw, dict): + import sys + print("[warn] generated_detection must be a dict; skipping", + file=sys.stderr) + return GeneratedDetectionConfig() + + result = GeneratedDetectionConfig() + + # Parse header_patterns (list of strings) + hp = raw.get("header_patterns") + if hp is not None: + if isinstance(hp, list): + valid = [s for s in hp if isinstance(s, str)] + if len(valid) != len(hp): + import sys + print("[warn] generated_detection.header_patterns: " + "non-string entries dropped", file=sys.stderr) + result = GeneratedDetectionConfig( + header_patterns=valid, + annotation_patterns=result.annotation_patterns, + force_fqns=result.force_fqns, + exclude_fqns=result.exclude_fqns + ) + else: + import sys + print("[warn] generated_detection.header_patterns: " + "must be a list; skipping", file=sys.stderr) + + # Parse annotation_patterns (list of strings) + ap = raw.get("annotation_patterns") + if ap is not None: + if isinstance(ap, list): + valid = [s for s in ap if isinstance(s, str)] + if len(valid) != len(ap): + import sys + print("[warn] generated_detection.annotation_patterns: " + "non-string entries dropped", file=sys.stderr) + result = GeneratedDetectionConfig( + header_patterns=result.header_patterns, + annotation_patterns=valid, + force_fqns=result.force_fqns, + exclude_fqns=result.exclude_fqns + ) + else: + import sys + print("[warn] generated_detection.annotation_patterns: " + "must be a list; skipping", file=sys.stderr) + + # Parse force_fqns (list of strings, convert to set) + ff = raw.get("force_fqns") + if ff is not None: + if isinstance(ff, list): + valid = {s for s in ff if isinstance(s, str)} + if len(valid) != len(ff): + import sys + print("[warn] generated_detection.force_fqns: " + "non-string entries dropped", file=sys.stderr) + result = GeneratedDetectionConfig( + header_patterns=result.header_patterns, + annotation_patterns=result.annotation_patterns, + force_fqns=valid, + exclude_fqns=result.exclude_fqns + ) + else: + import sys + print("[warn] generated_detection.force_fqns: " + "must be a list; skipping", file=sys.stderr) + + # Parse exclude_fqns (list of strings, convert to set) + ef = raw.get("exclude_fqns") + if ef is not None: + if isinstance(ef, list): + valid = {s for s in ef if isinstance(s, str)} + if len(valid) != len(ef): + import sys + print("[warn] generated_detection.exclude_fqns: " + "non-string entries dropped", file=sys.stderr) + result = GeneratedDetectionConfig( + header_patterns=result.header_patterns, + annotation_patterns=result.annotation_patterns, + force_fqns=result.force_fqns, + exclude_fqns=valid + ) + else: + import sys + print("[warn] generated_detection.exclude_fqns: " + "must be a list; skipping", file=sys.stderr) + + return result + + return GeneratedDetectionConfig() + + @lru_cache(maxsize=64) def _load_config_cross_service_resolution(project_root_str: str) -> str: """Read `cross_service_resolution` from `.java-codebase-rag.yml` at project_root. @@ -238,6 +362,19 @@ class BrownfieldOverrides: fqn_to_async_producer_hint: dict[str, AsyncProducerHint] = field(default_factory=dict) +@dataclass(frozen=True) +class GeneratedDetectionConfig: + """Config for generated-source detection. + + Mirrors brownfield override pattern: frozen dataclass with + field(default_factory=...) for mutable defaults. + """ + header_patterns: list[str] = field(default_factory=list) + annotation_patterns: list[str] = field(default_factory=list) + force_fqns: set[str] = field(default_factory=set) + exclude_fqns: set[str] = field(default_factory=set) + + def _meta_builtins() -> frozenset[str]: return ( frozenset(ROLE_ANNOTATIONS) @@ -1544,6 +1681,159 @@ def microservice_for_path( return "" +# ---------- generated-source detection ---------- + + +# Built-in generator markers (v1 set). Verified against real generator output. +# Annotation FQNs that mark generated code (any annotation with these FQNs +# or simple names is considered a marker). +_GENERATED_ANNOTATION_FQNS = { + "javax.annotation.processing.Generated", # Standard Java (pre-Jakarta) + "jakarta.annotation.processing.Generated", # Jakarta EE + "org.immutables.value.Generated", # Immutables + "lombok.Generated", # Lombok + "com.squareup.javapoet.Generated", # JavaPoet +} + +# Header patterns for generators that emit banners (checked against first 4KB). +# Patterns are compiled as case-insensitive regexes. +_GENERATED_HEADER_PATTERNS = { + r"This file was generated by the OpenAPI Generator": "openapi", + r"Generated by the protocol buffer compiler": "protobuf", + r"This file was generated by jsonschema2pojo": "jsonschema2pojo", + r"generated by wsimport": "wsimport", # JAX-WS wsimport + r"WARNING: DO NOT EDIT.*generated by MapStruct": "mapstruct", +} + +# @Generated(value="...") patterns that identify the generator family. +# These are matched against annotation arguments["value"] or arguments["comments"]. +_GENERATED_VALUE_PATTERNS = { + r"org\.openapitools\.codegen\.": "openapi", + r"org\.mapstruct\.ap\.MappingProcessor": "mapstruct", + r"com\.google\.auto\.value\.processor\.AutoValueProcessor": "autovalue", + r"org\.jooq\.": "jooq", + r"com\.querydsl\.": "querydsl", + r"org\.immutables\.": "immutables", +} + + +def _infer_family_from_annotation(annotation: AnnotationRef) -> str | None: + """Infer generator family from @Generated annotation arguments. + + Returns lowercased family slug or None if no match. + """ + # Check annotation FQN first (direct marker) + if annotation.qualified in _GENERATED_ANNOTATION_FQNS: + # Extract family from qualified name if possible + if "lombok.Generated" in annotation.qualified: + return "lombok" + if "immutables" in annotation.qualified: + return "immutables" + # Generic javax/jakarta or JavaPoet -> unknown family + return None + + # Check value/comments arguments for generator identifiers + value = annotation.arguments.get("value", "") + comments = annotation.arguments.get("comments", "") + + for pattern, family in _GENERATED_VALUE_PATTERNS.items(): + import re + if re.search(pattern, value) or re.search(pattern, comments): + return family + + return None + + +def _check_header_banners(source: bytes) -> str | None: + """Check header (first 4KB) for generator banners. + + Returns family slug if matched, None otherwise. + """ + HEADER_PREFIX_SIZE = 4096 # First 4KB should capture any banner + header_bytes = source[:HEADER_PREFIX_SIZE] + try: + header_text = header_bytes.decode("utf-8", errors="ignore") + except Exception: + return None + + import re + for pattern, family in _GENERATED_HEADER_PATTERNS.items(): + if re.search(pattern, header_text, re.IGNORECASE): + return family + + return None + + +def classify_java_file( + source: bytes, + ast: "JavaFileAst", + *, + config: GeneratedDetectionConfig | None = None, + project_root: str | Path | None = None, +) -> tuple[bool, str | None]: + """Classify whether a Java source file is generated. + + Args: + source: Raw file bytes (required for header-banner detection). + ast: Parsed Java AST (from ast_java.parse_java_ast). + config: Optional detection config (defaults to empty config). + project_root: Optional project root for loading default config. + + Returns: + (generated, generated_by) tuple: + - generated: True if file is detected as generated code. + - generated_by: Lowercased family slug (openapi, jsonschema2pojo, + protobuf, mapstruct, wsimport, querydsl, jooq, immutables, + autovalue, lombok) or None if family unknown. + """ + # Load config if not provided + if config is None: + config = load_generated_detection( + str(project_root) if project_root is not None else None + ) + + # Collect all type FQNs for config-based checks + all_type_fqns = {t.fqn for t in ast.all_types} + + # Priority 1: exclude_fqns (override, wins even with markers) + if all_type_fqns & config.exclude_fqns: + return False, None + + # Priority 2: force_fqns (forced generated, no markers needed) + if all_type_fqns & config.force_fqns: + return True, None + + # Priority 3: annotation-based detection + import re + for typ in ast.all_types: + for ann in typ.annotations: + # Check simple name "Generated" first + if ann.name == "Generated": + family = _infer_family_from_annotation(ann) + if family is not None: + return True, family + # Generic @Generated without identifiable value + return True, None + + # Check configured annotation patterns + for pattern in config.annotation_patterns: + if re.search(pattern, ann.qualified) or re.search(pattern, ann.name): + return True, None + + # Priority 4: header-banner detection + family = _check_header_banners(source) + if family is not None: + return True, family + + # Check configured header patterns + header_prefix = source[:4096].decode("utf-8", errors="ignore") + for pattern in config.header_patterns: + if re.search(pattern, header_prefix, re.IGNORECASE): + return True, None + + return False, None + + def detect_microservice_from_path(cwd: Path, source_root: Path) -> str | None: """Detect microservice from cwd for query-time auto-scope. diff --git a/tests/fixtures/generated_samples/HandWritten.java b/tests/fixtures/generated_samples/HandWritten.java new file mode 100644 index 0000000..1abf3b5 --- /dev/null +++ b/tests/fixtures/generated_samples/HandWritten.java @@ -0,0 +1,25 @@ +package com.example.handwritten; + +/** + * This is a hand-written business class, not generated. + */ +public class HandWritten { + private String name; + private String description; + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } +} diff --git a/tests/fixtures/generated_samples/JavaxGenerated.java b/tests/fixtures/generated_samples/JavaxGenerated.java new file mode 100644 index 0000000..620e9b8 --- /dev/null +++ b/tests/fixtures/generated_samples/JavaxGenerated.java @@ -0,0 +1,19 @@ +package com.example.javax; + +import javax.annotation.processing.Generated; + +/** + * Sample class with javax @Generated annotation (no value). + */ +@Generated +public class JavaxGenerated { + private String value; + + public String getValue() { + return value; + } + + public void setValue(String value) { + this.value = value; + } +} diff --git a/tests/fixtures/generated_samples/JsonSchemaPojo.java b/tests/fixtures/generated_samples/JsonSchemaPojo.java new file mode 100644 index 0000000..3ab497e --- /dev/null +++ b/tests/fixtures/generated_samples/JsonSchemaPojo.java @@ -0,0 +1,28 @@ +/** + * This file was generated by jsonschema2pojo. + * + * Generation date: 2025-01-15T10:30:00Z + * Schema: User.json + */ +package com.example.jsonschema; + +public class JsonSchemaPojo { + private String username; + private String email; + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public String getEmail() { + return email; + } + + public void setEmail(String email) { + this.email = email; + } +} diff --git a/tests/fixtures/generated_samples/OpenAPIModel.java b/tests/fixtures/generated_samples/OpenAPIModel.java new file mode 100644 index 0000000..0a3e47f --- /dev/null +++ b/tests/fixtures/generated_samples/OpenAPIModel.java @@ -0,0 +1,31 @@ +package com.example.model; + +import io.swagger.annotations.ApiModel; + +/** + * This file was generated by the OpenAPI Generator. + * + * OpenAPI Generator version: 7.0.0 + */ +@Generated(value = "org.openapitools.codegen.DefaultCodegen", date = "2025-01-15T10:30:00Z") +@ApiModel(description = "Sample API model") +public class OpenAPIModel { + private String id; + private String name; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } +} diff --git a/tests/fixtures/generated_samples/ProtobufMessage.java b/tests/fixtures/generated_samples/ProtobufMessage.java new file mode 100644 index 0000000..6ff7717 --- /dev/null +++ b/tests/fixtures/generated_samples/ProtobufMessage.java @@ -0,0 +1,27 @@ +// Generated by the protocol buffer compiler. DO NOT EDIT! +// source: user.proto +package com.example.protobuf; + +/** + * Protobuf-generated message class. + */ +public final class ProtobufMessage { + private String id; + private String data; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public String getData() { + return data; + } + + public void setData(String data) { + this.data = data; + } +} diff --git a/tests/test_generated_detection.py b/tests/test_generated_detection.py new file mode 100644 index 0000000..05e5564 --- /dev/null +++ b/tests/test_generated_detection.py @@ -0,0 +1,159 @@ +"""Tests for generated-source detection (Task 1).""" + +import os +from pathlib import Path +import pytest + +from graph_enrich import classify_java_file, GeneratedDetectionConfig, load_generated_detection +from ast_java import parse_java + + +FIXTURE_DIR = Path(__file__).parent / "fixtures" / "generated_samples" + + +def _parse_fixture(filename: str): + """Helper: parse fixture file, return (source_bytes, ast).""" + fixture_path = FIXTURE_DIR / filename + source = fixture_path.read_bytes() + ast = parse_java(source=source, filename=str(fixture_path)) + return source, ast + + +class TestGeneratedDetection: + """Generated-source detection tests.""" + + def test_openapi_annotation(self): + """@Generated(value="org.openapitools.codegen...") → (True, "openapi").""" + source, ast = _parse_fixture("OpenAPIModel.java") + generated, generated_by = classify_java_file(source, ast) + assert generated is True + assert generated_by == "openapi" + + def test_javax_generated_no_value(self): + """@Generated with no value → (True, None).""" + source, ast = _parse_fixture("JavaxGenerated.java") + generated, generated_by = classify_java_file(source, ast) + assert generated is True + assert generated_by is None + + def test_jsonschema2pojo_header(self): + """jsonschema2pojo banner in header → (True, "jsonschema2pojo").""" + source, ast = _parse_fixture("JsonSchemaPojo.java") + generated, generated_by = classify_java_file(source, ast) + assert generated is True + assert generated_by == "jsonschema2pojo" + + def test_protobuf_header(self): + """protobuf-generated banner → (True, "protobuf").""" + source, ast = _parse_fixture("ProtobufMessage.java") + generated, generated_by = classify_java_file(source, ast) + assert generated is True + assert generated_by == "protobuf" + + def test_handwritten_is_not_generated(self): + """Plain hand-written type, no markers → (False, None).""" + source, ast = _parse_fixture("HandWritten.java") + generated, generated_by = classify_java_file(source, ast) + assert generated is False + assert generated_by is None + + def test_config_force_fqn(self): + """FQN in force_fqns → (True, None) even without markers.""" + source, ast = _parse_fixture("HandWritten.java") + config = GeneratedDetectionConfig(force_fqns={"com.example.handwritten.HandWritten"}) + generated, generated_by = classify_java_file(source, ast, config=config) + assert generated is True + assert generated_by is None + + def test_config_exclude_fqn_wins(self): + """@Generated present but FQN in exclude_fqns → (False, None).""" + source, ast = _parse_fixture("OpenAPIModel.java") + config = GeneratedDetectionConfig( + exclude_fqns={"com.example.model.OpenAPIModel"} + ) + generated, generated_by = classify_java_file(source, ast, config=config) + assert generated is False + assert generated_by is None + + def test_config_extra_header_pattern(self): + """Custom header_patterns matches internal banner → (True, None).""" + # Create a fixture with custom header + custom_source = b""" +/** + * Internal code generator - DO NOT EDIT + * Generated by internal tool v1.0 + */ +package com.example.internal; + +public class CustomHeader { + private String value; + public String getValue() { return value; } + public void setValue(String value) { this.value = value; } +} +""" + ast = parse_java(source=custom_source, filename="CustomHeader.java") + config = GeneratedDetectionConfig( + header_patterns=[r"Internal code generator"] + ) + generated, generated_by = classify_java_file(custom_source, ast, config=config) + assert generated is True + assert generated_by is None + + +class TestConfigLoading: + """Config loading tests.""" + + def test_empty_config_when_missing(self): + """Empty config when file missing or section absent.""" + config = load_generated_detection(None) + assert config == GeneratedDetectionConfig() + + def test_loads_from_yaml(self): + """Loads generated_detection section from YAML.""" + # Create temp config file + import tempfile + with tempfile.TemporaryDirectory() as tmpdir: + config_path = Path(tmpdir) / ".java-codebase-rag.yml" + config_path.write_text(""" +generated_detection: + header_patterns: + - "Generated by internal tool" + annotation_patterns: + - "com.example.CustomGenerated" + force_fqns: + - "com.example.GeneratedModel" + exclude_fqns: + - "com.example.ExcludeMe" +""") + + config = load_generated_detection(tmpdir) + assert "Generated by internal tool" in config.header_patterns + assert "com.example.CustomGenerated" in config.annotation_patterns + assert "com.example.GeneratedModel" in config.force_fqns + assert "com.example.ExcludeMe" in config.exclude_fqns + + def test_malformed_entry_warns_and_continues(self): + """Malformed entries stderr-warn and are dropped.""" + import tempfile + import io + import sys + + with tempfile.TemporaryDirectory() as tmpdir: + config_path = Path(tmpdir) / ".java-codebase-rag.yml" + # Write config with malformed entries (not lists) + config_path.write_text(""" +generated_detection: + header_patterns: "this should be a list" + force_fqns: "also should be a list" +""") + + # Capture stderr + old_stderr = sys.stderr + sys.stderr = io.StringIO() + + try: + config = load_generated_detection(tmpdir) + # Should return empty config for malformed entries + assert config == GeneratedDetectionConfig() + finally: + sys.stderr = old_stderr From d3d7151eba7c71f0e67984ade575b9dcab3c0ab4 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 02:05:46 +0300 Subject: [PATCH 03/16] feat(index): tag Lance chunks generated/generated_by; bump ontology to 18 --- ast_java.py | 2 +- java_index_flow_lancedb.py | 34 ++++-- search_lancedb.py | 2 + tests/test_lancedb_generated_column.py | 142 +++++++++++++++++++++++++ 4 files changed, 172 insertions(+), 8 deletions(-) create mode 100644 tests/test_lancedb_generated_column.py diff --git a/ast_java.py b/ast_java.py index 8fbdf33..95fb96d 100644 --- a/ast_java.py +++ b/ast_java.py @@ -84,7 +84,7 @@ # Phase 11: `EDGE_SCHEMA` in `java_ontology.py` (canonical edge navigation schema; v14 re-index). # Phase 12: CALLS `callee_declaring_role`, supertype-walk dedup, pass3 unresolved counters (v15 re-index). # Bumps whenever extraction / enrichment semantics change. -ONTOLOGY_VERSION = 17 +ONTOLOGY_VERSION = 18 ROLE_ANNOTATIONS: dict[str, str] = { # Spring Web diff --git a/java_index_flow_lancedb.py b/java_index_flow_lancedb.py index 8850222..089963f 100644 --- a/java_index_flow_lancedb.py +++ b/java_index_flow_lancedb.py @@ -51,7 +51,13 @@ ) from path_filtering import LayeredIgnore from ast_java import ONTOLOGY_VERSION, parse_java -from graph_enrich import collect_annotation_meta_chain, enrich_chunk, load_brownfield_overrides +from graph_enrich import ( + classify_java_file, + collect_annotation_meta_chain, + enrich_chunk, + load_brownfield_overrides, + load_generated_detection, +) # Older cocoindex (e.g. 1.0.0a43) uses ``tracked=False``; newer releases renamed # the flag to ``detect_change`` (default False) and reject ``tracked``. @@ -280,6 +286,9 @@ class JavaLanceChunk: annotations_on_type: Annotated[list[str], LanceType(pa.list_(pa.string()))] symbols: Annotated[list[str], LanceType(pa.list_(pa.string()))] ontology_version: int + # Generated source detection: populated per-file, not per-chunk + generated: bool + generated_by: str | None @dataclass @@ -364,12 +373,13 @@ def _parse_and_enrich_java( chunks: list[Any], rel: str, project_root: Path, -) -> list[Any]: +) -> tuple[list[Any], Any]: """Parse one Java file and enrich every chunk, off the event loop. - Returns a list of :class:`graph_enrich.ChunkEnrichment` aligned 1:1 with - ``chunks``. Intended to run via ``asyncio.to_thread`` from - ``process_java_file`` (vectors perf lever #2): while the worker thread + Returns a tuple of (enrichments, ast) where enrichments is a list of + :class:`graph_enrich.ChunkEnrichment` aligned 1:1 with ``chunks``, and ast + is the parsed :class:`JavaFileAst`. Intended to run via ``asyncio.to_thread`` + from ``process_java_file`` (vectors perf lever #2): while the worker thread parses + enriches, the event loop is free to drive other files and keep the embedder's batching queue fed. @@ -381,7 +391,7 @@ def _parse_and_enrich_java( ``lru_cache`` reads are thread-safe under the GIL. """ ast = parse_java(content_bytes) - return [ + enrichments = [ enrich_chunk( ast, chunk_start_byte=ch.start.byte_offset, @@ -391,6 +401,7 @@ def _parse_and_enrich_java( ) for ch in chunks ] + return enrichments, ast @coco.fn(memo=True) @@ -430,9 +441,16 @@ async def process_java_file( # (vectors perf lever #2) parse + enrich off the event loop so the loop can # keep the embedder's batching queue fed while this file is being parsed. # parse_java is thread-safe (per-thread tree-sitter Parser in ast_java). - enrichments = await asyncio.to_thread( + enrichments, ast = await asyncio.to_thread( _parse_and_enrich_java, content_bytes, chunks, rel, project_root ) + + # Compute generated source detection once per file (uses the AST and content_bytes) + generated_config = load_generated_detection(project_root) + generated, generated_by = classify_java_file( + content_bytes, ast, config=generated_config, project_root=project_root + ) + # (vectors perf lever #1) embed all chunks concurrently so the batched # embedder groups them into one ``model.encode(...)`` (max_batch_size=64) # instead of N serial batch-of-1 calls. Dominant win for ``increment`` @@ -462,6 +480,8 @@ async def process_java_file( annotations_on_type=enrich.annotations_on_type, symbols=enrich.symbols, ontology_version=ONTOLOGY_VERSION, + generated=generated, + generated_by=generated_by, ) ) diff --git a/search_lancedb.py b/search_lancedb.py index 39d7b64..f4693c0 100644 --- a/search_lancedb.py +++ b/search_lancedb.py @@ -68,6 +68,8 @@ "metadata", "ontology_version", "capabilities", + "generated", + "generated_by", ) VECTOR_COLUMN = "embedding" diff --git a/tests/test_lancedb_generated_column.py b/tests/test_lancedb_generated_column.py new file mode 100644 index 0000000..0f7a767 --- /dev/null +++ b/tests/test_lancedb_generated_column.py @@ -0,0 +1,142 @@ +"""Test that Lance chunks are tagged with generated/generated_by columns. + +This test verifies that Task 2's implementation correctly tags chunks: +- Generated files (e.g., OpenAPI) have generated=True and generated_by set +- Hand-written files have generated=False and generated_by=None +""" +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest +import lancedb + + +FIXTURE_ROOT = Path(__file__).resolve().parent / "fixtures" / "generated_samples" + + +def _require_cocoindex_runtime_deps() -> None: + """cocoindex loads java_index_flow_lancedb.py with the same Python as the CLI.""" + try: + import tree_sitter_java # noqa: F401 + except ImportError as exc: + pytest.skip( + "Test needs project deps in the current env (e.g. ``pip install -r requirements*``" + f" in the venv you use to run pytest): {exc}" + ) + + +def _cocoindex_flow_specifier(bundle_dir: Path, index_cwd: Path) -> str: + """Return the coco index flow specifier for the java_index_flow_lancedb app.""" + import os + flow_file = (bundle_dir / "java_index_flow_lancedb.py").resolve() + if not flow_file.is_file(): + raise FileNotFoundError(f"missing index flow: {flow_file}") + start = index_cwd.resolve() + relp = os.path.relpath(str(flow_file), start=str(start)) + relp = Path(relp).as_posix() + return f"{relp}:JavaCodeIndexLance" + + +@pytest.mark.parametrize( + "fixture_file,expected_generated,expected_generated_by", + [ + ("OpenAPIModel.java", True, "openapi"), + ("HandWritten.java", False, None), + ], +) +def test_lance_chunk_generated_columns( + tmp_path: Path, + fixture_file: str, + expected_generated: bool, + expected_generated_by: str | None, +) -> None: + """Test that Lance chunks are tagged with generated/generated_by columns. + + This test indexes the generated_samples fixture and verifies: + 1. The columns exist in the LanceDB schema + 2. Generated files have generated=True and correct generated_by + 3. Hand-written files have generated=False and generated_by=None + """ + _require_cocoindex_runtime_deps() + + # Locate the bundle dir (repo root) + bundle_dir = Path(__file__).resolve().parent.parent + + # Get the flow specifier + app_spec = _cocoindex_flow_specifier(bundle_dir, FIXTURE_ROOT) + + # Locate cocoindex binary + import sys + import os + cocoindex_bin = Path(sys.executable).parent / "cocoindex" + if not cocoindex_bin.is_file(): + pytest.skip( + f"cocoindex CLI not found next to the pytest interpreter; install cocoindex in this " + f"venv and run: `.venv/bin/python -m pytest ...` ({cocoindex_bin})" + ) + + # Set up the index directory in tmp_path + index_dir = tmp_path / ".java-codebase-rag" + index_dir.mkdir(parents=True) + + # Set up environment + env = { + **os.environ, + "JAVA_CODEBASE_RAG_INDEX_DIR": str(index_dir.resolve()), + "JAVA_CODEBASE_RAG_SOURCE_ROOT": str(FIXTURE_ROOT.resolve()), + } + + # Run cocoindex update from the fixture directory + result = subprocess.run( + [ + str(cocoindex_bin), + "update", + app_spec, + "-f", + ], + cwd=str(FIXTURE_ROOT), + env=env, + capture_output=True, + text=True, + timeout=300, + ) + if result.returncode != 0: + print(f"STDOUT: {result.stdout}") + print(f"STDERR: {result.stderr}") + raise subprocess.CalledProcessError(result.returncode, result.args, result.stdout, result.stderr) + print(f"Cocoindex STDOUT: {result.stdout}") + print(f"Cocoindex STDERR: {result.stderr}") + + # Open the database and check the java table + db = lancedb.connect(str(index_dir)) + existing_tables = db.table_names() + print(f"Available tables: {existing_tables}") + if not existing_tables: + raise ValueError(f"No tables found in database at {index_dir}. Available tables: {existing_tables}") + table = db.open_table("javacodeindex_java_code") + + # Check that the columns exist in the schema + schema = table.schema + schema_field_names = schema.names + assert "generated" in schema_field_names, "Column 'generated' must exist in schema" + assert "generated_by" in schema_field_names, "Column 'generated_by' must exist in schema" + + # Query chunks for the specific file + chunk_results = table.search().where(f"filename LIKE '%{fixture_file}'").to_arrow() + + # Assert we found chunks for this file + assert chunk_results.num_rows > 0, f"Expected to find chunks for {fixture_file}" + + # Check that all chunks have the correct generated and generated_by values + for i in range(chunk_results.num_rows): + chunk_generated = chunk_results["generated"][i].as_py() + chunk_generated_by = chunk_results["generated_by"][i].as_py() + chunk_id = chunk_results["id"][i].as_py() + assert ( + chunk_generated == expected_generated + ), f"Chunk {chunk_id} has generated={chunk_generated}, expected {expected_generated}" + assert ( + chunk_generated_by == expected_generated_by + ), f"Chunk {chunk_id} has generated_by={chunk_generated_by}, expected {expected_generated_by}" From a24541b6eacd60be72cb28af2c931c597e58a3e1 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 02:15:44 +0300 Subject: [PATCH 04/16] feat(graph): tag Symbol nodes generated/generated_by --- build_ast_graph.py | 38 +++++- tests/test_graph_generated_node.py | 204 +++++++++++++++++++++++++++++ 2 files changed, 238 insertions(+), 4 deletions(-) create mode 100644 tests/test_graph_generated_node.py diff --git a/build_ast_graph.py b/build_ast_graph.py index 9dd2222..f9bbc9f 100644 --- a/build_ast_graph.py +++ b/build_ast_graph.py @@ -54,8 +54,10 @@ ) from graph_enrich import ( _load_config_cross_service_resolution, + classify_java_file, collect_annotation_meta_chain, load_brownfield_overrides, + load_generated_detection, microservice_for_path, module_for_path, phantom_id, @@ -465,6 +467,8 @@ class GraphTables: cross_service_resolution: str = "auto" # Populated in _write_nodes (same overrides + meta_chain as Symbol.role). type_role_by_node_id: dict[str, str] = field(default_factory=dict) + # Populated in pass 1 (classify_java_file) and _load_existing_types for incremental rebuilds. + type_generated_by_node_id: dict[str, tuple[bool, str]] = field(default_factory=dict) @dataclass @@ -598,7 +602,7 @@ def _load_existing_types(conn: ladybug.Connection, tables: GraphTables, exclude_ query = f""" MATCH (s:Symbol) {where} - RETURN s.kind, s.fqn, s.name, s.filename, s.module, s.microservice, s.id, s.role + RETURN s.kind, s.fqn, s.name, s.filename, s.module, s.microservice, s.id, s.role, s.generated, s.generated_by """ result = conn.execute(query, params) while result.has_next(): @@ -608,6 +612,8 @@ def _load_existing_types(conn: ladybug.Connection, tables: GraphTables, exclude_ microservice = row[5] if len(row) > 5 else "" node_id = row[6] if len(row) > 6 else "" role = row[7] if len(row) > 7 else "" + generated = row[8] if len(row) > 8 else False + generated_by = row[9] if len(row) > 9 else "" decl = TypeDecl(name, kind, fqn) package = fqn[: -(len(name) + 1)] if fqn.endswith("." + name) else "" @@ -629,6 +635,8 @@ def _load_existing_types(conn: ladybug.Connection, tables: GraphTables, exclude_ # the default during node staging (issue #352 divergence #2). if role: tables.type_role_by_node_id[node_id] = role + # Seed the persisted generated/generated_by so stubs retain their values + tables.type_generated_by_node_id[node_id] = (generated if generated else False, generated_by or "") def _load_existing_members(conn: ladybug.Connection, tables: GraphTables, exclude_files: set[str] | None = None) -> None: @@ -1061,6 +1069,12 @@ def pass1_parse( microservice = microservice_for_path(str(p), root) asts[rel] = ast + # Classify the file once (generated or not, and which tool generated it) + generated_config = load_generated_detection(str(root)) + file_generated, file_generated_by = classify_java_file( + content, ast, config=generated_config, project_root=root + ) + # file node file_id = symbol_id("file", rel, rel, 0) tables.files[rel] = file_id @@ -1075,6 +1089,12 @@ def pass1_parse( module=module, microservice=microservice, outer_fqn=None, ) + # Seed generated/generated_by for all types in this file (including nested) + for t in ast.all_types: + if t.fqn in tables.types: + node_id = tables.types[t.fqn].node_id + tables.type_generated_by_node_id[node_id] = (file_generated, file_generated_by or "") + if verbose: elapsed = time.time() - t0 _emit_graph_progress( @@ -2906,7 +2926,8 @@ def _micro_factor(member: MemberEntry | None) -> float: "filename STRING, start_line INT64, end_line INT64, " "start_byte INT64, end_byte INT64, " "modifiers STRING[], annotations STRING[], capabilities STRING[], " - "role STRING, signature STRING, parent_id STRING, resolved BOOLEAN" + "role STRING, signature STRING, parent_id STRING, resolved BOOLEAN, " + "generated BOOLEAN, generated_by STRING" ")" ) @@ -3088,6 +3109,7 @@ def _node_row(**kwargs) -> dict: "start_byte": 0, "end_byte": 0, "modifiers": [], "annotations": [], "capabilities": [], "role": "OTHER", "signature": "", "parent_id": "", "resolved": True, + "generated": False, "generated_by": "", } base.update(kwargs) return base @@ -3138,7 +3160,8 @@ def _existing_node_ids(conn: ladybug.Connection) -> set[str]: _NODE_COLUMNS = [ "id", "kind", "name", "fqn", "package", "module", "microservice", "filename", "start_line", "end_line", "start_byte", "end_byte", - "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved" + "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved", + "generated", "generated_by" ] # Type declaration kinds. Tuple (not set) so the rendered SQL `IN` clause is @@ -3161,7 +3184,8 @@ def _existing_node_ids(conn: ladybug.Connection) -> set[str]: "n.start_byte = $start_byte, n.end_byte = $end_byte, " "n.modifiers = $modifiers, n.annotations = $annotations, " "n.capabilities = $capabilities, n.role = $role, " - "n.signature = $signature, n.parent_id = $parent_id, n.resolved = $resolved" + "n.signature = $signature, n.parent_id = $parent_id, n.resolved = $resolved, " + "n.generated = $generated, n.generated_by = $generated_by" ) # Refresh every mutable Route field on an existing Route node by id. Mirrors the @@ -3240,6 +3264,9 @@ def _write_nodes_impl( overrides=overrides, meta_chain=mch, ) + # Read generated/generated_by from pass-1 classification or stub persistence + generated, generated_by = tables.type_generated_by_node_id.get(entry.node_id, (False, "")) + if entry.loaded_from_db: stub_ids.add(entry.node_id) # Out-of-scope stub: its annotation-less decl collapses role to the @@ -3250,6 +3277,7 @@ def _write_nodes_impl( # capabilities placeholder never reaches the graph. role = tables.type_role_by_node_id.get(entry.node_id, role) capabilities = [] + # For stubs, trust the persisted generated/generated_by (seeded by _load_existing_types) else: tables.type_role_by_node_id[entry.node_id] = role rows.append(_node_row( @@ -3265,6 +3293,8 @@ def _write_nodes_impl( role=role, signature="", parent_id=tables.types[entry.outer_fqn].node_id if entry.outer_fqn and entry.outer_fqn in tables.types else "", + generated=generated, + generated_by=generated_by, )) # members (methods / constructors) for m in tables.members: diff --git a/tests/test_graph_generated_node.py b/tests/test_graph_generated_node.py new file mode 100644 index 0000000..fecd045 --- /dev/null +++ b/tests/test_graph_generated_node.py @@ -0,0 +1,204 @@ +"""Tests for Task 3: Tag graph Symbol nodes with generated/generated_by. + +Tests verify that: +1. Generated types are tagged with generated=True and generated_by set +2. Hand-written types have generated=False +3. Incremental rebuild preserves generated/generated_by on unchanged types +""" +from __future__ import annotations + +from pathlib import Path + +import ladybug +import pytest + +from _builders import build_ladybug_into + + +def _connect(db_path: Path) -> ladybug.Connection: + db = ladybug.Database(str(db_path), read_only=True) + return ladybug.Connection(db) + + +def _scalar(conn: ladybug.Connection, query: str): + r = conn.execute(query) + if not r.has_next(): + return None + return r.get_next()[0] + + +def test_generated_type_has_generated_fields( + tmp_path: Path, +) -> None: + """Build a graph from a fixture with a generated type → assert generated == True, generated_by set.""" + # Create a simple fixture with a generated type (OpenAPI @Generated with value) + root = tmp_path / "proj" + java_dir = root / "src/main/java/com/example" + java_dir.mkdir(parents=True) + + # Generated type: OpenAPI @Generated class + generated_java = java_dir / "GeneratedUser.java" + generated_java.write_text( + "package com.example;\n" + "\n" + "@Generated(value = \"org.openapitools.codegen.DefaultCodegen\", date = \"2025-01-15T10:30:00Z\")\n" + "public class GeneratedUser {\n" + " private String name;\n" + " private int age;\n" + "}\n" + ) + + # Hand-written type (no generation annotation) + hand_written_java = java_dir / "ManualUser.java" + hand_written_java.write_text( + "package com.example;\n" + "\n" + "public class ManualUser {\n" + " private String name;\n" + " private int age;\n" + "}\n" + ) + + # Build the graph + db_path = tmp_path / "test.lbug" + build_ladybug_into(root, db_path) + + # Query the generated type + conn = _connect(db_path) + generated_result = conn.execute( + "MATCH (n:Symbol {fqn: 'com.example.GeneratedUser'}) " + "RETURN n.generated, n.generated_by" + ) + assert generated_result.has_next(), "GeneratedUser type should exist in graph" + gen_generated, gen_generated_by = generated_result.get_next() + assert gen_generated is True, f"GeneratedUser should have generated=True, got {gen_generated}" + assert gen_generated_by == "openapi", f"GeneratedUser should have generated_by='openapi', got {gen_generated_by}" + + # Query the hand-written type + manual_result = conn.execute( + "MATCH (n:Symbol {fqn: 'com.example.ManualUser'}) " + "RETURN n.generated, n.generated_by" + ) + assert manual_result.has_next(), "ManualUser type should exist in graph" + manual_generated, manual_generated_by = manual_result.get_next() + assert manual_generated is False, f"ManualUser should have generated=False, got {manual_generated}" + assert manual_generated_by == "" or manual_generated_by is None, f"ManualUser should have empty generated_by, got {manual_generated_by}" + + +def test_incremental_rebuild_preserves_generated_fields( + tmp_path: Path, +) -> None: + """Incremental rebuild: a preserved (unchanged) generated type retains its generated/generated_by.""" + from build_ast_graph import FileHashTracker, GraphTables, incremental_rebuild, pass1_parse, write_ladybug + from graph_enrich import load_generated_detection + from path_filtering import LayeredIgnore + + load_generated_detection.cache_clear() + + # Create fixture with both generated and hand-written types + source_root = tmp_path / "src" + source_root.mkdir() + index_dir = tmp_path / "index" + index_dir.mkdir() + ladybug_path = index_dir / "code_graph.lbug" + + java_dir = source_root / "com/example" + java_dir.mkdir(parents=True) + + # Generated type (OpenAPI style) + generated_java = java_dir / "GeneratedUser.java" + generated_java.write_text( + "package com.example;\n" + "\n" + "@Generated(value = \"org.openapitools.codegen.DefaultCodegen\", date = \"2025-01-15T10:30:00Z\")\n" + "public class GeneratedUser {\n" + " private String name;\n" + " private int age;\n" + "}\n" + ) + + # Hand-written type that we'll modify later + hand_written_java = java_dir / "ManualUser.java" + hand_written_java.write_text( + "package com.example;\n" + "\n" + "public class ManualUser {\n" + " private String name;\n" + "}\n" + ) + + # Initial build + tables = GraphTables() + asts = pass1_parse(source_root, tables, verbose=False) + from build_ast_graph import pass2_edges + pass2_edges(tables, asts, verbose=False) + write_ladybug(ladybug_path, tables, source_root=source_root, verbose=False) + + # Initialize hash tracker + tracker = FileHashTracker(index_dir) + ignore = LayeredIgnore(source_root, use_gitignore=False, builtin_patterns=[]) + tracker.detect_changes(source_root, ignore) + for rel_path in ["com/example/GeneratedUser.java", "com/example/ManualUser.java"]: + tracker.update({rel_path}, source_root) + tracker.save() + + # Verify initial state + conn = _connect(ladybug_path) + initial_gen_result = conn.execute( + "MATCH (n:Symbol {fqn: 'com.example.GeneratedUser'}) " + "RETURN n.generated, n.generated_by, n.id" + ) + assert initial_gen_result.has_next() + initial_gen_generated, initial_gen_by, gen_node_id = initial_gen_result.get_next() + assert initial_gen_generated is True + assert initial_gen_by == "openapi" + + # Now modify ONLY the hand-written file (not the generated one) + hand_written_java.write_text( + "package com.example;\n" + "\n" + "public class ManualUser {\n" + " private String name;\n" + " private int age; // Added field\n" + "}\n" + ) + + # Incremental rebuild (should preserve GeneratedUser as unchanged) + result = incremental_rebuild(source_root, ladybug_path, verbose=False) + assert result.mode == "incremental" + + # Verify that the preserved generated type still has its original values + conn = _connect(ladybug_path) + final_gen_result = conn.execute( + "MATCH (n:Symbol {fqn: 'com.example.GeneratedUser'}) " + "RETURN n.generated, n.generated_by, n.id" + ) + assert final_gen_result.has_next() + final_gen_generated, final_gen_by, final_gen_node_id = final_gen_result.get_next() + assert final_gen_generated is True, f"Preserved GeneratedUser should still have generated=True, got {final_gen_generated}" + assert final_gen_by == "openapi", f"Preserved GeneratedUser should still have generated_by='openapi', got {final_gen_by}" + assert final_gen_node_id == gen_node_id, "Node ID should be the same after incremental rebuild" + + +def test_schema_has_generated_columns( + tmp_path: Path, +) -> None: + """Verify the Symbol table has generated and generated_by columns.""" + root = tmp_path / "proj" + java_dir = root / "src/main/java/com/example" + java_dir.mkdir(parents=True) + (java_dir / "Test.java").write_text("package com.example; public class Test {}") + + db_path = tmp_path / "test.lbug" + build_ladybug_into(root, db_path) + + # Simply try to query the columns - if they don't exist, we'll get an error + conn = _connect(db_path) + try: + result = conn.execute("MATCH (n:Symbol) RETURN n.generated, n.generated_by LIMIT 1;") + # If we get here, columns exist + assert result.has_next() or not result.has_next() # Either way, query succeeded + except RuntimeError as e: + if "Cannot find property" in str(e): + pytest.fail(f"Schema missing generated columns: {e}") + raise From 0702a3668fee27b0e287f6f5d75b9e0a0cc2bc33 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 02:31:42 +0300 Subject: [PATCH 05/16] test: green up version anchors + graph baseline after ontology 18 bump ONTOLOGY_VERSION 17->18 (T2) + new generated/generated_by columns (T2/T3) left drift in version-anchored tests and the committed graph baseline: - test_ontology_version_bumped_to_17 -> _to_18 (assert 18) - test_jrag_status: assert ontology_version == 18 - graph_baseline_bank_chat.json: graph_meta.ontology_version 17->18 (node/edge/counts_json unchanged - T3 added columns, not nodes/edges) - test_brownfield_overrides: JavaLanceChunk(...) passes generated/generated_by Co-Authored-By: Claude --- tests/fixtures/graph_baseline_bank_chat.json | 2 +- tests/test_brownfield_overrides.py | 2 ++ tests/test_incremental_graph.py | 6 +++--- tests/test_jrag_status.py | 4 ++-- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/tests/fixtures/graph_baseline_bank_chat.json b/tests/fixtures/graph_baseline_bank_chat.json index 2002e99..b9357e2 100644 --- a/tests/fixtures/graph_baseline_bank_chat.json +++ b/tests/fixtures/graph_baseline_bank_chat.json @@ -10,7 +10,7 @@ "UNRESOLVED_AT": 227 }, "graph_meta": { - "ontology_version": 17, + "ontology_version": 18, "built_at": 1782110216, "source_root": "/Users/dmitry/Desktop/CursorProjects/java-enterprise-codebase-rag/tests/bank-chat-system", "counts_json": "{packages: 29, files: 130, types: 140, members: 606, phantoms: 54, extends: 18, implements: 21, injects: 94, declares: 606, overrides: 38, calls: 684, routes: 29, exposes: 15, clients: 8, declares_client: 8, producers: 9, declares_producer: 9, http_calls: 8, async_calls: 9}" diff --git a/tests/test_brownfield_overrides.py b/tests/test_brownfield_overrides.py index 319424c..f62b067 100644 --- a/tests/test_brownfield_overrides.py +++ b/tests/test_brownfield_overrides.py @@ -556,6 +556,8 @@ def test_tier2_lance_row_carries_enrich_capabilities_without_lancedb() -> None: annotations_on_type=e.annotations_on_type, symbols=e.symbols, ontology_version=ONTOLOGY_VERSION, + generated=False, + generated_by=None, ) assert "MESSAGE_LISTENER" in row.capabilities diff --git a/tests/test_incremental_graph.py b/tests/test_incremental_graph.py index c6e9560..0a3f7cc 100644 --- a/tests/test_incremental_graph.py +++ b/tests/test_incremental_graph.py @@ -222,9 +222,9 @@ def test_source_file_value_matches_symbol_filename(self, tmp_path: Path) -> None sub_filename, edge_source_file = result.get_next() assert sub_filename == edge_source_file - def test_ontology_version_bumped_to_17(self) -> None: - """ONTOLOGY_VERSION == 17.""" - assert ONTOLOGY_VERSION == 17 + def test_ontology_version_bumped_to_18(self) -> None: + """ONTOLOGY_VERSION == 18.""" + assert ONTOLOGY_VERSION == 18 class TestIncrementalOrchestrator: diff --git a/tests/test_jrag_status.py b/tests/test_jrag_status.py index d64a3d0..dfbdbdf 100644 --- a/tests/test_jrag_status.py +++ b/tests/test_jrag_status.py @@ -56,7 +56,7 @@ def _run_jrag( def test_status_reports_ontology_version_and_counts( corpus_root: Path, ladybug_db_path: Path ) -> None: - """`jrag status` against a real index reports ontology 17 + non-empty counts.""" + """`jrag status` against a real index reports ontology 18 + non-empty counts.""" env = os.environ.copy() env["JAVA_CODEBASE_RAG_SOURCE_ROOT"] = str(corpus_root) env["JAVA_CODEBASE_RAG_INDEX_DIR"] = str(ladybug_db_path.parent) @@ -70,7 +70,7 @@ def test_status_reports_ontology_version_and_counts( payload = json.loads(proc.stdout) assert payload["status"] == "ok" index = payload["nodes"]["index"] - assert index["ontology_version"] == 17 + assert index["ontology_version"] == 18 # Counts is a top-level nested dict on the index node (the generic # nested-sections dispatch signal - any dict-typed value renders as an # indented alphabetical section; edge_summary is NOT used as the dispatch From b15059927f106d20668701008ac19ecbe18175df Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 02:37:19 +0300 Subject: [PATCH 06/16] feat(mcp): surface generated/generated_by on search/find/describe/neighbors --- graph_types.py | 4 + ladybug_queries.py | 8 +- mcp_v2.py | 8 +- search_lancedb.py | 4 + tests/test_generated_surface.py | 289 ++++++++++++++++++++++++++++++++ 5 files changed, 307 insertions(+), 6 deletions(-) create mode 100644 tests/test_generated_surface.py diff --git a/graph_types.py b/graph_types.py index 7bce026..d10b744 100644 --- a/graph_types.py +++ b/graph_types.py @@ -34,6 +34,8 @@ class NodeRef(BaseModel): microservice: str | None = None module: str | None = None role: str | None = None + generated: bool | None = None + generated_by: str | None = None class StructuredHint(BaseModel): @@ -126,6 +128,8 @@ def _node_ref_from_row(kind: Literal["symbol", "route", "client", "producer"], r microservice=str(row.get("microservice") or "") or None, module=str(row.get("module") or "") or None, role=role, + generated=bool(row.get("generated")) if row.get("generated") is not None else None, + generated_by=str(row.get("generated_by")) if row.get("generated_by") else None, ) diff --git a/ladybug_queries.py b/ladybug_queries.py index af28a20..7b49f02 100644 --- a/ladybug_queries.py +++ b/ladybug_queries.py @@ -207,7 +207,7 @@ def _symbol_return_for(alias: str) -> str: f"{alias}.modifiers AS modifiers, {alias}.annotations AS annotations, " f"{alias}.capabilities AS capabilities, " f"{alias}.role AS role, {alias}.signature AS signature, " - f"{alias}.parent_id AS parent_id, {alias}.resolved AS resolved" + f"{alias}.parent_id AS parent_id, {alias}.resolved AS resolved, {alias}.generated AS generated, {alias}.generated_by AS generated_by" ) @@ -296,7 +296,7 @@ def _row_to_symbol(row: dict[str, Any]) -> SymbolHit: _SYM_COLS = ( "id", "kind", "name", "fqn", "package", "module", "microservice", "filename", "start_line", "end_line", "start_byte", "end_byte", - "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved", + "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved", "generated", "generated_by", ) @@ -1107,14 +1107,14 @@ def find_injectors(self, target_name_or_fqn: str, *, f"s.{c} AS s_{c}" for c in ( "id", "kind", "name", "fqn", "package", "module", "microservice", "filename", "start_line", "end_line", "start_byte", "end_byte", - "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved", + "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved", "generated", "generated_by", ) ) t_proj = ", ".join( f"t.{c} AS t_{c}" for c in ( "id", "kind", "name", "fqn", "package", "module", "microservice", "filename", "start_line", "end_line", "start_byte", "end_byte", - "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved", + "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved", "generated", "generated_by", ) ) q = ( diff --git a/mcp_v2.py b/mcp_v2.py index 89e3af7..bd6a33a 100644 --- a/mcp_v2.py +++ b/mcp_v2.py @@ -497,6 +497,8 @@ class SearchHit(BaseModel): microservice: str | None = None module: str | None = None role: str | None = None + generated: bool | None = None + generated_by: str | None = None filename: str | None = None start_line: int | None = None score_components: dict[str, float] | None = None @@ -649,6 +651,8 @@ def _row_to_search_hit(row: dict[str, Any], explain: bool = False) -> SearchHit: microservice=str(row.get("microservice")) if row.get("microservice") else None, module=str(row.get("module")) if row.get("module") else None, role=str(row.get("role")) if row.get("role") else None, + generated=bool(row.get("generated")) if row.get("generated") is not None else None, + generated_by=str(row.get("generated_by")) if row.get("generated_by") else None, filename=filename, start_line=start_line, score_components=row.get("_score_components") if explain else None, @@ -722,7 +726,7 @@ def _load_node_record( "n.start_line AS start_line, n.end_line AS end_line, n.start_byte AS start_byte, " "n.end_byte AS end_byte, n.modifiers AS modifiers, n.annotations AS annotations, " "n.capabilities AS capabilities, n.role AS role, n.signature AS signature, " - "n.parent_id AS parent_id, n.resolved AS resolved" + "n.parent_id AS parent_id, n.resolved AS resolved, n.generated AS generated, n.generated_by AS generated_by" ) label = "Symbol" elif kind == "route": @@ -1097,7 +1101,7 @@ def find_v2( params["lim"] = fetch_cap rows = g._rows( # noqa: SLF001 f"MATCH (s:Symbol) {where} RETURN s.id AS id, s.fqn AS fqn, s.microservice AS microservice, " - "s.module AS module, s.role AS role, s.kind AS symbol_kind ORDER BY s.fqn LIMIT $lim", + "s.module AS module, s.role AS role, s.kind AS symbol_kind, s.generated AS generated, s.generated_by AS generated_by ORDER BY s.fqn LIMIT $lim", params, ) elif kind == "route": diff --git a/search_lancedb.py b/search_lancedb.py index f4693c0..67ed027 100644 --- a/search_lancedb.py +++ b/search_lancedb.py @@ -932,6 +932,10 @@ def main() -> None: mod = row.get("module") or "" if mod and mod != ms: hint_s += f" | module:{mod}" + gen = row.get("generated") + gen_by = row.get("generated_by") or "" + if gen: + hint_s += f" | generated:{gen_by}" if gen_by else " | generated" comps = row.get("_score_components") or {} rw = comps.get("role_weight") if rw: diff --git a/tests/test_generated_surface.py b/tests/test_generated_surface.py new file mode 100644 index 0000000..f7fe618 --- /dev/null +++ b/tests/test_generated_surface.py @@ -0,0 +1,289 @@ +"""Test Task 4: Surface generated/generated_by on search/find/describe/neighbors results. + +Tests verify that: +1. search returns SearchHit with generated/generated_by set +2. find(symbol) returns NodeRef with the fields +3. describe includes generated/generated_by in NodeRecord.data +4. neighbors endpoint NodeRef carries them +5. CLI search output prints the generated hint +""" +from __future__ import annotations + +from typing import Any + +import pytest + +from mcp_v2 import SearchHit, find_v2, describe_v2, neighbors_v2, search_v2 +from graph_types import NodeRef + + +def _fake_search_rows_with_generated() -> list[dict[str, Any]]: + """Fake search rows with generated/generated_by fields.""" + return [ + { + "id": "chunk:1", + "symbol_id": "sym:1", + "primary_type_fqn": "com.example.OpenAPIModel", + "_rrf_score": 0.9, + "text": "OpenAPIModel sample", + "microservice": "chat-assign", + "module": "chat-assign", + "role": "DTO", + "filename": "chat-assign/src/main/java/com/example/OpenAPIModel.java", + "start": {"byte_offset": 10}, + "end": {"byte_offset": 30}, + "generated": True, + "generated_by": "openapi", + }, + { + "id": "chunk:2", + "symbol_id": "sym:2", + "primary_type_fqn": "com.example.HandWritten", + "_rrf_score": 0.8, + "text": "HandWritten sample", + "microservice": "chat-core", + "module": "chat-app", + "role": "SERVICE", + "filename": "chat-core/chat-app/src/main/java/com/example/HandWritten.java", + "start": {"byte_offset": 40}, + "end": {"byte_offset": 80}, + "generated": False, + "generated_by": None, + }, + ] + + +def test_search_surfaces_generated_fields(monkeypatch, ladybug_graph) -> None: + """Test that search returns SearchHit with generated/generated_by set.""" + monkeypatch.setattr("mcp_v2.run_search", lambda *args, **kwargs: _fake_search_rows_with_generated()) + out = search_v2("OpenAPIModel", graph=ladybug_graph) + assert out.success is True + assert out.results + assert len(out.results) == 2 + + # First result is generated + generated_hit = out.results[0] + assert generated_hit.generated is True, "Generated chunk should have generated=True" + assert generated_hit.generated_by == "openapi", "Generated chunk should have generated_by='openapi'" + + # Second result is hand-written + manual_hit = out.results[1] + assert manual_hit.generated is False, "Hand-written chunk should have generated=False" + assert manual_hit.generated_by is None, "Hand-written chunk should have generated_by=None" + + +def test_search_handles_missing_generated_fields(monkeypatch, ladybug_graph) -> None: + """Test that search handles rows without generated/generated_by fields (old indexes).""" + # Use fake rows without generated/generated_by (simulating old indexes) + rows_without_generated = [ + { + "id": "chunk:1", + "symbol_id": "sym:1", + "primary_type_fqn": "com.example.OldType", + "_rrf_score": 0.9, + "text": "Old type sample", + "microservice": "chat-assign", + "module": "chat-assign", + "role": "SERVICE", + "filename": "chat-assign/src/main/java/com/example/OldType.java", + "start": {"byte_offset": 10}, + "end": {"byte_offset": 30}, + # Note: no generated/generated_by fields + }, + ] + monkeypatch.setattr("mcp_v2.run_search", lambda *args, **kwargs: rows_without_generated) + out = search_v2("OldType", graph=ladybug_graph) + assert out.success is True + assert out.results + assert len(out.results) == 1 + + # Should default to None when fields are missing + hit = out.results[0] + assert hit.generated is None, "Missing generated field should default to None" + assert hit.generated_by is None, "Missing generated_by field should default to None" + + +def test_find_surfaces_generated_fields(monkeypatch, ladybug_graph) -> None: + """Test that find(symbol) returns NodeRef with generated/generated_by fields.""" + # Mock the graph query to return rows with generated/generated_by + def _mock_rows(query: str, params: dict[str, Any]) -> list[dict[str, Any]]: + if "MATCH (s:Symbol)" in query and "RETURN" in query: + return [ + { + "id": "sym:1", + "fqn": "com.example.OpenAPIModel", + "microservice": "chat-assign", + "module": "chat-assign", + "role": "DTO", + "kind": "class", + "generated": True, + "generated_by": "openapi", + }, + { + "id": "sym:2", + "fqn": "com.example.HandWritten", + "microservice": "chat-core", + "module": "chat-app", + "role": "SERVICE", + "kind": "class", + "generated": False, + "generated_by": None, + }, + ] + return [] + + monkeypatch.setattr(ladybug_graph, "_rows", _mock_rows) + out = find_v2("symbol", {}, graph=ladybug_graph) + assert out.success is True + assert out.results + assert len(out.results) == 2 + + # First result is generated + generated_ref = out.results[0] + assert isinstance(generated_ref, NodeRef) + assert generated_ref.generated is True, "Generated symbol should have generated=True" + assert generated_ref.generated_by == "openapi", "Generated symbol should have generated_by='openapi'" + + # Second result is hand-written + manual_ref = out.results[1] + assert isinstance(manual_ref, NodeRef) + assert manual_ref.generated is False, "Hand-written symbol should have generated=False" + assert manual_ref.generated_by is None, "Hand-written symbol should have generated_by=None" + + +def test_describe_surfaces_generated_fields(monkeypatch, ladybug_graph) -> None: + """Test that describe includes generated/generated_by in NodeRecord.data for symbols.""" + # Mock the graph query to return a symbol with generated/generated_by + def _mock_rows(query: str, params: dict[str, Any]) -> list[dict[str, Any]]: + if "MATCH (n:Symbol)" in query and "WHERE n.id = $id" in query: + return [ + { + "id": "sym:1", + "kind": "class", + "name": "OpenAPIModel", + "fqn": "com.example.OpenAPIModel", + "package": "com.example", + "module": "chat-assign", + "microservice": "chat-assign", + "filename": "chat-assign/src/main/java/com/example/OpenAPIModel.java", + "start_line": 1, + "end_line": 10, + "start_byte": 0, + "end_byte": 100, + "modifiers": ["public"], + "annotations": [], + "capabilities": [], + "role": "DTO", + "signature": "OpenAPIModel", + "parent_id": None, + "resolved": True, + "generated": True, + "generated_by": "openapi", + } + ] + return [] + + monkeypatch.setattr(ladybug_graph, "_rows", _mock_rows) + out = describe_v2("sym:1", graph=ladybug_graph) + assert out.success is True + assert out.record is not None + assert out.record.kind == "symbol" + + # Check that generated/generated_by are in the data + assert "generated" in out.record.data, "NodeRecord.data should contain 'generated'" + assert "generated_by" in out.record.data, "NodeRecord.data should contain 'generated_by'" + assert out.record.data["generated"] is True, "generated should be True" + assert out.record.data["generated_by"] == "openapi", "generated_by should be 'openapi'" + + +def test_neighbors_surfaces_generated_fields(monkeypatch, ladybug_graph) -> None: + """Test that neighbors endpoint NodeRef carries generated/generated_by fields.""" + # Get a valid method ID first + rows = ladybug_graph._rows( # noqa: SLF001 + "MATCH (src:Symbol)-[:CALLS]->(dst:Symbol) RETURN dst.id AS id LIMIT 1" + ) + if not rows: + pytest.skip("No CALLS edges in graph") + method_id = rows[0]["id"] + + # Mock _load_node_record to return a symbol with generated/generated_by + original_load_node_record = None + import mcp_v2 + original_load_node_record = mcp_v2._load_node_record + + def _mock_load_node_record(graph, node_id, kind): + # Return a mock symbol with generated/generated_by + if kind == "symbol" and node_id != method_id: + return { + "id": node_id, + "kind": "class", + "name": "GeneratedNeighbor", + "fqn": "com.example.GeneratedNeighbor", + "package": "com.example", + "module": "chat-assign", + "microservice": "chat-assign", + "filename": "chat-assign/src/main/java/com/example/GeneratedNeighbor.java", + "start_line": 1, + "end_line": 10, + "start_byte": 0, + "end_byte": 100, + "modifiers": ["public"], + "annotations": [], + "capabilities": [], + "role": "SERVICE", + "signature": "GeneratedNeighbor", + "parent_id": None, + "resolved": True, + "generated": True, + "generated_by": "openapi", + } + # For the origin node, use the original function + return original_load_node_record(graph, node_id, kind) + + monkeypatch.setattr("mcp_v2._load_node_record", _mock_load_node_record) + + out = neighbors_v2(method_id, direction="out", edge_types=["CALLS"], graph=ladybug_graph) + assert out.success is True + assert isinstance(out.results, list) + + # If we have neighbors, check the first one + if out.results: + edge = out.results[0] + assert isinstance(edge, Edge) + neighbor_ref = edge.other + assert isinstance(neighbor_ref, NodeRef) + # The neighbor should have generated/generated_by if we're mocking correctly + # Note: This may not always find a generated neighbor in the real graph + if neighbor_ref.generated is not None: + assert neighbor_ref.generated is True + assert neighbor_ref.generated_by == "openapi" + + +def test_cli_search_prints_generated_hint() -> None: + """Test that CLI search output prints the generated hint.""" + # Simulate the hint building logic from search_lancedb.py + row = _fake_search_rows_with_generated()[0] # Get the generated row + + # Build hint string (same logic as search_lancedb.py lines 1202-1224) + hints = row.get("_hints") or {} + hint_s = "" + if hints.get("primary_type_hint"): + hint_s += f" | type:{hints['primary_type_hint']}" + if hints.get("import_heavy"): + hint_s += " | mostly-imports" + role = row.get("role") or "" + if role: + hint_s += f" | role:{role}" + ms = row.get("microservice") or "" + if ms: + hint_s += f" | microservice:{ms}" + mod = row.get("module") or "" + if mod and mod != ms: + hint_s += f" | module:{mod}" + gen = row.get("generated") + gen_by = row.get("generated_by") or "" + if gen: + hint_s += f" | generated:{gen_by}" if gen_by else " | generated" + + # Check that the generated hint is in the hint string + assert "| generated:openapi" in hint_s, f"Hint string should contain 'generated:openapi', got: {hint_s}" From fa47b860e30bd998523dc2d80e07f4afb51ba8c1 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 02:48:02 +0300 Subject: [PATCH 07/16] feat(search): exclude_generated / generated_only filters MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add exclude_generated and generated_only boolean flags to both engines - Lance SQL: column-guarded predicates (generated = true / IS NULL OR false) - MCP/Kùzu: Cypher predicates + Python post-filter via _node_matches_filter - CLI: --exclude-generated and --generated-only flags - NodeFilter: added fields with applicability for symbol kind - Fix _populated_nodefilter_fields to skip False boolean fields Co-Authored-By: Claude --- mcp_v2.py | 21 +++++ search_lancedb.py | 15 +++ tests/test_generated_filter.py | 161 +++++++++++++++++++++++++++++++++ 3 files changed, 197 insertions(+) create mode 100644 tests/test_generated_filter.py diff --git a/mcp_v2.py b/mcp_v2.py index bd6a33a..c1b02f1 100644 --- a/mcp_v2.py +++ b/mcp_v2.py @@ -224,6 +224,8 @@ class NodeFilter(BaseModel): source_layer: SourceLayer | None = None role: Role | None = None exclude_roles: list[Role] | None = None + generated_only: bool = False + exclude_generated: bool = False annotation: str | None = None capability: str | None = None fqn_contains: str | None = None @@ -306,6 +308,8 @@ def _role_axes_mutually_exclusive(self) -> EdgeFilter: "module", "role", "exclude_roles", + "generated_only", + "exclude_generated", "annotation", "capability", "fqn_contains", @@ -350,6 +354,8 @@ def _populated_nodefilter_fields(nf: NodeFilter) -> set[str]: continue if isinstance(value, list) and not value: continue + if isinstance(value, bool) and not value: + continue populated.add(field_name) return populated @@ -397,6 +403,8 @@ def _populated_edgefilter_fields(ef: EdgeFilter) -> set[str]: continue if isinstance(value, list) and not value: continue + if isinstance(value, bool) and not value: + continue populated.add(field_name) return populated @@ -694,6 +702,10 @@ def _symbol_where_from_filter(f: NodeFilter) -> tuple[str, dict[str, Any]]: if f.exclude_roles: preds.append("NOT s.role IN $exclude_roles") params["exclude_roles"] = list(f.exclude_roles) + if f.generated_only: + preds.append("s.generated = true") + if f.exclude_generated: + preds.append("(s.generated IS NULL OR s.generated = false)") if f.annotation: preds.append("list_contains(s.annotations, $annotation)") params["annotation"] = f.annotation @@ -823,6 +835,11 @@ def _node_matches_filter( return False if f.exclude_roles and role in set(f.exclude_roles): return False + generated = row.get("generated") + if f.generated_only and not generated: + return False + if f.exclude_generated and generated: + return False if f.annotation and f.annotation not in list(row.get("annotations") or []): return False if f.capability and f.capability not in list(row.get("capabilities") or []): @@ -987,6 +1004,8 @@ def search_v2( microservice=nf.microservice if nf else None, capability=nf.capability if nf else None, exclude_roles=nf.exclude_roles if nf else None, + exclude_generated=nf.exclude_generated if nf else None, + generated_only=nf.generated_only if nf else None, dedup_by_fqn=dedup, ) except Exception as exc: @@ -1011,6 +1030,8 @@ def search_v2( microservice=nf.microservice if nf else None, capability=nf.capability if nf else None, exclude_roles=nf.exclude_roles if nf else None, + exclude_generated=nf.exclude_generated if nf else None, + generated_only=nf.generated_only if nf else None, dedup_by_fqn=dedup, ) advisories.append( diff --git a/search_lancedb.py b/search_lancedb.py index 67ed027..c5387b1 100644 --- a/search_lancedb.py +++ b/search_lancedb.py @@ -109,6 +109,8 @@ def _build_extra_predicates( exclude_roles: list[str] | None = None, capability: str | None = None, capability_in: list[str] | None = None, + generated_only: bool = False, + exclude_generated: bool = False, ) -> list[str]: preds: list[str] = [] if role and "role" in columns: @@ -141,6 +143,10 @@ def _build_extra_predicates( if exclude_roles and "role" in columns: vals = ", ".join(f"'{_escape_sql_str(v)}'" for v in exclude_roles) preds.append(f"(role IS NULL OR role NOT IN ({vals}))") + if generated_only and "generated" in columns: + preds.append("generated = true") + if exclude_generated and "generated" in columns: + preds.append("(generated IS NULL OR generated = false)") if module and "module" in columns: preds.append(f"module = '{_escape_sql_str(module)}'") if microservice and "microservice" in columns: @@ -656,6 +662,8 @@ def run_search( exclude_roles: list[str] | None = None, capability: str | None = None, capability_in: list[str] | None = None, + generated_only: bool = False, + exclude_generated: bool = False, dedup_by_fqn: bool = False, ) -> list[dict]: effective_hybrid = hybrid @@ -707,6 +715,7 @@ def run_search( package_prefix=package_prefix, fqn_in=None, role_in=role_in, exclude_roles=exclude_roles, capability=capability, capability_in=capability_in, + generated_only=generated_only, exclude_generated=exclude_generated, ) if "java" in table_keys else [] skip_role_weight = bool(role or role_in or exclude_roles) @@ -834,6 +843,10 @@ def main() -> None: parser.add_argument("--fts-text", metavar="TEXT", default=None) parser.add_argument("--auto-hybrid", action="store_true") parser.add_argument("--role", default=None) + parser.add_argument("--exclude-generated", action="store_true", + help="Exclude generated sources from results.") + parser.add_argument("--generated-only", action="store_true", + help="Return only generated sources in results.") parser.add_argument("--module", default=None, help="Filter to a single Maven/Gradle module name.") parser.add_argument("--microservice", default=None, @@ -887,6 +900,8 @@ def main() -> None: expand_depth=args.expand_depth, ladybug_path=args.ladybug_path, context_neighbors=args.context_neighbors, + exclude_generated=args.exclude_generated, + generated_only=args.generated_only, ) except Exception as e: print(f"Search failed: {e}", file=sys.stderr) diff --git a/tests/test_generated_filter.py b/tests/test_generated_filter.py new file mode 100644 index 0000000..7dd2b69 --- /dev/null +++ b/tests/test_generated_filter.py @@ -0,0 +1,161 @@ +"""Task 5: exclude_generated / generated_only filter tests. + +Tests both engines (Lance SQL + MCP/Kùzu Cypher + Python post-filter). + +Fixture: generated_samples (one generated + one hand-written type). +""" + +import os +import pytest +from pathlib import Path +from mcp_v2 import find_v2, NodeFilter + + +# Skip LanceDB tests if we're in a graph-only environment +pytest.importorskip("lancedb") +import search_lancedb + + +def test_mcp_nodefilter_accepts_generated_flags(ladybug_graph) -> None: + """NodeFilter should accept exclude_generated and generated_only flags.""" + # This would fail before implementation due to extra="forbid" + filter1 = NodeFilter(exclude_generated=True) + filter2 = NodeFilter(generated_only=True) + filter3 = NodeFilter(exclude_generated=False, generated_only=False) + + assert filter1.exclude_generated is True + assert filter2.generated_only is True + assert filter3.exclude_generated is False + assert filter3.generated_only is False + + +def test_mcp_find_exclude_generated_excludes_generated_nodes(ladybug_graph) -> None: + """find(symbol, filter=NodeFilter(exclude_generated=True)) → generated NodeRef excluded. + + This tests the Cypher path via _symbol_where_from_filter. + """ + out = find_v2( + "symbol", + NodeFilter(exclude_generated=True), + graph=ladybug_graph, + limit=100, + ) + assert out.success is True + # Results should only contain hand-written symbols (generated is False/None) + for r in out.results: + if hasattr(r, 'generated'): + assert not r.generated, f"Expected only hand-written symbols, but found generated: {r}" + + +def test_mcp_find_generated_only_returns_only_generated_nodes(ladybug_graph) -> None: + """find(symbol, filter=NodeFilter(generated_only=True)) → only generated. + + This tests the post-filter path via _node_matches_filter. + """ + out = find_v2( + "symbol", + NodeFilter(generated_only=True), + graph=ladybug_graph, + limit=100, + ) + assert out.success is True + # Results should only contain generated symbols + for r in out.results: + if hasattr(r, 'generated'): + assert r.generated, f"Expected only generated symbols, but found hand-written: {r}" + + +def test_mcp_find_default_returns_both_types(ladybug_graph) -> None: + """find(symbol, filter=NodeFilter()) → both generated and hand-written returned (default).""" + out = find_v2( + "symbol", + NodeFilter(), # No flags set (default behavior) + graph=ladybug_graph, + limit=100, + ) + assert out.success is True + # Should have results (we don't assert both types exist since the test data may vary) + assert len(out.results) > 0, "Expected some results with default filter" + + +@pytest.fixture +def lancedb_uri(tmp_path): + """Provide a LanceDB URI for testing if available.""" + # Try to get the index directory from the environment + index_dir = os.environ.get("JAVA_CODEBASE_RAG_INDEX_DIR") + if index_dir and Path(index_dir).exists(): + # Check if it's a LanceDB directory (has .lance files) + lance_path = Path(index_dir) + if any(lance_path.glob("*.lance")) or (lance_path / "java").exists(): + return str(lance_path) + pytest.skip("No LanceDB index available - skipping LanceDB tests") + + +def test_run_search_exclude_generated_removes_generated_sources(lancedb_uri) -> None: + """run_search(..., exclude_generated=True) → no generated chunks; hand-written present.""" + try: + rows = search_lancedb.run_search( + "ChatController OR ChatService", + uri=lancedb_uri, + table_keys=["java"], + limit=10, + path_substring=None, # No path filter + model_name="sentence-transformers/all-MiniLM-L6-v2", + device=None, + exclude_generated=True, + ) + # Should not contain any generated sources + for row in rows: + if "generated" in row: + assert not row.get("generated"), f"Expected no generated sources, but found: {row}" + except Exception as e: + if "was not found" in str(e): + pytest.skip("LanceDB java table not found in index") + else: + raise + + +def test_run_search_generated_only_returns_only_generated_sources(lancedb_uri) -> None: + """run_search(..., generated_only=True) → only generated chunks.""" + try: + rows = search_lancedb.run_search( + "ChatController OR ChatService", + uri=lancedb_uri, + table_keys=["java"], + limit=10, + path_substring=None, + model_name="sentence-transformers/all-MiniLM-L6-v2", + device=None, + generated_only=True, + ) + # Should only contain generated sources + for row in rows: + if "generated" in row: + assert row.get("generated"), f"Expected only generated sources, but found: {row}" + except Exception as e: + if "was not found" in str(e): + pytest.skip("LanceDB java table not found in index") + else: + raise + + +def test_run_search_default_returns_both_types(lancedb_uri) -> None: + """run_search(..., default) → both generated and hand-written returned.""" + try: + rows = search_lancedb.run_search( + "ChatController OR ChatService", + uri=lancedb_uri, + table_keys=["java"], + limit=10, + path_substring=None, + model_name="sentence-transformers/all-MiniLM-L6-v2", + device=None, + # Neither flag set (default behavior) + ) + # Should have results + assert len(rows) > 0, "Expected some results with default filter" + except Exception as e: + if "was not found" in str(e): + pytest.skip("LanceDB java table not found in index") + else: + raise From f75062e6988634b1b65d5b9902f4d935c55a9f52 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 02:52:58 +0300 Subject: [PATCH 08/16] test(generated): make run_search filter tests build a real Lance index Co-Authored-By: Claude --- tests/test_generated_filter.py | 244 ++++++++++++++++++++++----------- 1 file changed, 163 insertions(+), 81 deletions(-) diff --git a/tests/test_generated_filter.py b/tests/test_generated_filter.py index 7dd2b69..07999d6 100644 --- a/tests/test_generated_filter.py +++ b/tests/test_generated_filter.py @@ -5,17 +5,99 @@ Fixture: generated_samples (one generated + one hand-written type). """ -import os -import pytest +import subprocess from pathlib import Path -from mcp_v2 import find_v2, NodeFilter +import pytest +import lancedb + +from mcp_v2 import find_v2, NodeFilter # Skip LanceDB tests if we're in a graph-only environment pytest.importorskip("lancedb") import search_lancedb +FIXTURE_ROOT = Path(__file__).resolve().parent / "fixtures" / "generated_samples" + + +def _require_cocoindex_runtime_deps() -> None: + """cocoindex loads java_index_flow_lancedb.py with the same Python as the CLI.""" + try: + import tree_sitter_java # noqa: F401 + except ImportError as exc: + pytest.skip( + "Test needs project deps in the current env (e.g. ``pip install -r requirements*``" + f" in the venv you use to run pytest): {exc}" + ) + + +def _cocoindex_flow_specifier(bundle_dir: Path, index_cwd: Path) -> str: + """Return the coco index flow specifier for the java_index_flow_lancedb app.""" + import os + flow_file = (bundle_dir / "java_index_flow_lancedb.py").resolve() + if not flow_file.is_file(): + raise FileNotFoundError(f"missing index flow: {flow_file}") + start = index_cwd.resolve() + relp = os.path.relpath(str(flow_file), start=str(start)) + relp = Path(relp).as_posix() + return f"{relp}:JavaCodeIndexLance" + + +@pytest.fixture +def lancedb_with_generated_index(tmp_path): + """Build a real Lance index over generated_samples and return the URI.""" + _require_cocoindex_runtime_deps() + + # Locate the bundle dir (repo root) + bundle_dir = Path(__file__).resolve().parent.parent + + # Get the flow specifier + app_spec = _cocoindex_flow_specifier(bundle_dir, FIXTURE_ROOT) + + # Locate cocoindex binary + import sys + import os + cocoindex_bin = Path(sys.executable).parent / "cocoindex" + if not cocoindex_bin.is_file(): + pytest.skip( + f"cocoindex CLI not found next to the pytest interpreter; install cocoindex in this " + f"venv and run: `.venv/bin/python -m pytest ...` ({cocoindex_bin})" + ) + + # Set up the index directory in tmp_path + index_dir = tmp_path / ".java-codebase-rag" + index_dir.mkdir(parents=True) + + # Set up environment + env = { + **os.environ, + "JAVA_CODEBASE_RAG_INDEX_DIR": str(index_dir.resolve()), + "JAVA_CODEBASE_RAG_SOURCE_ROOT": str(FIXTURE_ROOT.resolve()), + } + + # Run cocoindex update from the fixture directory + result = subprocess.run( + [ + str(cocoindex_bin), + "update", + app_spec, + "-f", + ], + cwd=str(FIXTURE_ROOT), + env=env, + capture_output=True, + text=True, + timeout=300, + ) + if result.returncode != 0: + print(f"STDOUT: {result.stdout}") + print(f"STDERR: {result.stderr}") + raise subprocess.CalledProcessError(result.returncode, result.args, result.stdout, result.stderr) + + return str(index_dir) + + def test_mcp_nodefilter_accepts_generated_flags(ladybug_graph) -> None: """NodeFilter should accept exclude_generated and generated_only flags.""" # This would fail before implementation due to extra="forbid" @@ -78,84 +160,84 @@ def test_mcp_find_default_returns_both_types(ladybug_graph) -> None: assert len(out.results) > 0, "Expected some results with default filter" -@pytest.fixture -def lancedb_uri(tmp_path): - """Provide a LanceDB URI for testing if available.""" - # Try to get the index directory from the environment - index_dir = os.environ.get("JAVA_CODEBASE_RAG_INDEX_DIR") - if index_dir and Path(index_dir).exists(): - # Check if it's a LanceDB directory (has .lance files) - lance_path = Path(index_dir) - if any(lance_path.glob("*.lance")) or (lance_path / "java").exists(): - return str(lance_path) - pytest.skip("No LanceDB index available - skipping LanceDB tests") - - -def test_run_search_exclude_generated_removes_generated_sources(lancedb_uri) -> None: +def test_run_search_exclude_generated_removes_generated_sources(lancedb_with_generated_index) -> None: """run_search(..., exclude_generated=True) → no generated chunks; hand-written present.""" - try: - rows = search_lancedb.run_search( - "ChatController OR ChatService", - uri=lancedb_uri, - table_keys=["java"], - limit=10, - path_substring=None, # No path filter - model_name="sentence-transformers/all-MiniLM-L6-v2", - device=None, - exclude_generated=True, - ) - # Should not contain any generated sources - for row in rows: - if "generated" in row: - assert not row.get("generated"), f"Expected no generated sources, but found: {row}" - except Exception as e: - if "was not found" in str(e): - pytest.skip("LanceDB java table not found in index") - else: - raise - - -def test_run_search_generated_only_returns_only_generated_sources(lancedb_uri) -> None: + rows = search_lancedb.run_search( + "HandWritten OR Model", # Query that matches both files + uri=lancedb_with_generated_index, + table_keys=["java"], + limit=10, + path_substring=None, # No path filter + model_name="sentence-transformers/all-MiniLM-L6-v2", + device=None, + exclude_generated=True, + ) + + # Should have results (hand-written file should match) + assert len(rows) > 0, "Expected results from hand-written file when exclude_generated=True" + + # Should not contain any generated sources + for row in rows: + if "generated" in row: + assert not row.get("generated"), f"Expected no generated sources, but found: {row}" + + # Verify HandWritten.java is present + filenames = {row.get("filename", "") for row in rows} + assert any("HandWritten.java" in f for f in filenames), "Expected HandWritten.java in results" + + # Verify OpenAPIModel.java is NOT present + assert not any("OpenAPIModel.java" in f for f in filenames), "OpenAPIModel.java should be filtered out" + + +def test_run_search_generated_only_returns_only_generated_sources(lancedb_with_generated_index) -> None: """run_search(..., generated_only=True) → only generated chunks.""" - try: - rows = search_lancedb.run_search( - "ChatController OR ChatService", - uri=lancedb_uri, - table_keys=["java"], - limit=10, - path_substring=None, - model_name="sentence-transformers/all-MiniLM-L6-v2", - device=None, - generated_only=True, - ) - # Should only contain generated sources - for row in rows: - if "generated" in row: - assert row.get("generated"), f"Expected only generated sources, but found: {row}" - except Exception as e: - if "was not found" in str(e): - pytest.skip("LanceDB java table not found in index") - else: - raise - - -def test_run_search_default_returns_both_types(lancedb_uri) -> None: + rows = search_lancedb.run_search( + "HandWritten OR Model", # Query that matches both files + uri=lancedb_with_generated_index, + table_keys=["java"], + limit=10, + path_substring=None, + model_name="sentence-transformers/all-MiniLM-L6-v2", + device=None, + generated_only=True, + ) + + # Should have results (generated file should match) + assert len(rows) > 0, "Expected results from generated file when generated_only=True" + + # Should only contain generated sources + for row in rows: + if "generated" in row: + assert row.get("generated"), f"Expected only generated sources, but found: {row}" + + # Verify OpenAPIModel.java is present + filenames = {row.get("filename", "") for row in rows} + assert any("OpenAPIModel.java" in f for f in filenames), "Expected OpenAPIModel.java in results" + + # Verify HandWritten.java is NOT present + assert not any("HandWritten.java" in f for f in filenames), "HandWritten.java should be filtered out" + + +def test_run_search_default_returns_both_types(lancedb_with_generated_index) -> None: """run_search(..., default) → both generated and hand-written returned.""" - try: - rows = search_lancedb.run_search( - "ChatController OR ChatService", - uri=lancedb_uri, - table_keys=["java"], - limit=10, - path_substring=None, - model_name="sentence-transformers/all-MiniLM-L6-v2", - device=None, - # Neither flag set (default behavior) - ) - # Should have results - assert len(rows) > 0, "Expected some results with default filter" - except Exception as e: - if "was not found" in str(e): - pytest.skip("LanceDB java table not found in index") - else: - raise + rows = search_lancedb.run_search( + "HandWritten OR Model", # Query that matches both files + uri=lancedb_with_generated_index, + table_keys=["java"], + limit=10, + path_substring=None, + model_name="sentence-transformers/all-MiniLM-L6-v2", + device=None, + # Neither flag set (default behavior) + ) + + # Should have results from both files + assert len(rows) > 0, "Expected results with default filter" + + # Verify both files are present + filenames = {row.get("filename", "") for row in rows} + has_generated = any("OpenAPIModel.java" in f for f in filenames) + has_handwritten = any("HandWritten.java" in f for f in filenames) + + assert has_generated, "Expected OpenAPIModel.java (generated) in default results" + assert has_handwritten, "Expected HandWritten.java (hand-written) in default results" From a41751c8a0d8d1f7d0938e57c03e67be7a0a58c3 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Wed, 8 Jul 2026 02:57:59 +0300 Subject: [PATCH 09/16] docs: generated-source detection, tagging, and filtering Co-Authored-By: Claude --- docs/AGENT-GUIDE.md | 17 +++++++++++++++-- docs/CODEBASE_REQUIREMENTS.md | 23 +++++++++++++++-------- docs/CONFIGURATION.md | 35 +++++++++++++++++++++++++++++++++-- docs/JAVA-CODEBASE-RAG-CLI.md | 4 ++++ 4 files changed, 67 insertions(+), 12 deletions(-) diff --git a/docs/AGENT-GUIDE.md b/docs/AGENT-GUIDE.md index 4bebfc7..a50b56b 100644 --- a/docs/AGENT-GUIDE.md +++ b/docs/AGENT-GUIDE.md @@ -14,7 +14,7 @@ Copy the block between `