Rust version of codebaseGraph#26
Merged
Merged
Conversation
Define the native extension boundary before implementation starts, including the Python-owned orchestration contract, Rust-owned deterministic batch scope, opt-in fallback behavior, benchmark command, and parity fixture strategy. Constraint: Rust acceleration must preserve existing Python CLI, MCP, graph query, Notion, and Scryer component boundaries Rejected: Add pyo3 or maturin dependencies now | this task establishes the design and benchmark baseline before native implementation Confidence: high Scope-risk: narrow Directive: Keep native execution opt-in until parity fixtures and benchmark evidence pass in Python-default and native-opt-in modes Tested: ./.venv/bin/pytest -q; ./.venv/bin/ruff check .; ./.venv/bin/python scripts/benchmark_materialization.py --repo-root tests/fixtures/sample_project --iterations 1 --warmups 0 --no-fts --no-semantic-enrichment; ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Native extension build, because Rust implementation has not been introduced yet
Add a multi-language golden materialization fixture and structural snapshot test before native graph rewrites. The snapshot locks stable node records, edge records, spans, labels, metadata, and counts across Python, Rust, Go, C, C++, Fortran, Markdown, syntax captures, and semantic enrichment. Constraint: Future Rust output must compare against current Python graph contracts with stable ordering Rejected: Snapshot volatile graph-level semantic evidence lists | first-class semantic evidence is already checked through graph edges and duplicated metadata can change order Confidence: high Scope-risk: moderate Directive: Update golden_graph_parity.json only when an intentional graph contract change is reviewed with a clear before/after explanation Tested: ./.venv/bin/pytest tests/test_golden_graph_parity.py tests/test_materializer.py tests/test_supported_language_parsing.py tests/test_semantic_symbol_table.py tests/test_semantic_reference_resolver.py tests/test_semantic_call_type_resolver.py tests/test_semantic_enrichment_writer.py -q; ./.venv/bin/pytest -q; ./.venv/bin/ruff check .; ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Future Rust output comparison, because native implementation has not been introduced yet
Add a private Rust workspace and Python wrapper for the Graph Builder capture-bundle boundary. The native binary accepts normalized capture records over an internal line protocol and emits CodeGraph rows that match Python for the golden parity fixture languages, while the Python wrapper keeps native execution opt-in and falls back to the existing builder when disabled or unsupported. Constraint: Scryer Graph Builder boundary is captures to CodeGraph rows; public CLI/MCP output shape stays owned by Python until the materializer feature gate task. Rejected: Add PyO3/maturin integration now | it would change packaging before the prototype has materializer opt-in coverage Rejected: Replace the existing Python GraphBuilder entry point | Task 4 owns feature-gated materialization integration Confidence: high Scope-risk: moderate Directive: Keep CODEBASE_GRAPH_NATIVE opt-in until parity is verified through materialization, not just capture bundles Tested: cargo fmt --manifest-path rust/Cargo.toml -- --check Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: cargo test --manifest-path rust/Cargo.toml Tested: ./.venv/bin/pytest tests/test_native_graph_builder.py tests/test_golden_graph_parity.py tests/test_graph_builder.py tests/test_materializer.py tests/test_supported_language_parsing.py -q Tested: ./.venv/bin/pytest -q Tested: ./.venv/bin/ruff check . Tested: ./.venv/bin/ruff check tests/test_native_graph_builder.py src/codebase_graph/_native Tested: ./.venv/bin/ruff format --check tests/test_native_graph_builder.py src/codebase_graph/_native Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Native graph builder materializer integration; that remains the next feature-gate task
Wire GraphMaterializer through a private build helper that keeps Python as the default path and calls the native graph builder only when CODEBASE_GRAPH_NATIVE=1. Injected graph builders remain Python-owned so caller-supplied behavior is not changed by the opt-in native path. Constraint: Scryer keeps Materializer Orchestrator in Python; Rust may replace Graph Builder internals only behind stable Python-callable boundaries. Rejected: Make the native wrapper unconditional | default materialization must remain Python until parity and benchmark gates pass Rejected: Enable native for injected graph_builder collaborators | custom builder behavior is caller-owned and may not match native capture mapping Confidence: high Scope-risk: narrow Directive: Do not make CODEBASE_GRAPH_NATIVE the default until benchmark and large-repo parity tasks are complete Tested: ./.venv/bin/pytest tests/test_materializer.py tests/test_native_graph_builder.py -q Tested: CODEBASE_GRAPH_NATIVE=1 CODEBASE_GRAPH_NATIVE_GRAPH_BUILDER=rust/target/debug/codebase_graph_native_graph_builder ./.venv/bin/pytest tests/test_golden_graph_parity.py tests/test_materializer.py -q Tested: ./.venv/bin/pytest -q Tested: ./.venv/bin/ruff check . Tested: cargo fmt --manifest-path rust/Cargo.toml -- --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Large-repo benchmark speedup; that remains the follow-on benchmark task
Add a native BULK protocol to the Rust helper so the opt-in native path can group bulk node, edge, and connector rows and write COPY-compatible JSON and CSV staging files while Python keeps LadyBug database lifecycle and COPY execution ownership. The Python store preserves fallback behavior when native staging is disabled, unavailable, or duplicate row metadata would require Python's metadata update semantics. Constraint: Scryer architecture keeps LadyBug store lifecycle, partition replacement, atomic rebuild, and database COPY execution in Python. Constraint: No new Rust dependencies; staging serialization uses the existing native helper and standard library only. Rejected: Move COPY execution into Rust | would cross the Bulk Loader boundary and take over LadyBug lifecycle responsibilities owned by Python. Rejected: Parse arbitrary row JSON in Rust | unnecessary for COPY-compatible file generation and would require a new dependency or a larger parser. Confidence: high Scope-risk: moderate Directive: Keep native bulk staging behind CODEBASE_GRAPH_NATIVE and preserve Python fallback for semantic edge cases. Tested: cargo fmt --manifest-path rust/Cargo.toml -- --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/pytest tests/test_native_bulk_staging.py -q Tested: CODEBASE_GRAPH_NATIVE=1 CODEBASE_GRAPH_NATIVE_GRAPH_BUILDER=rust/target/debug/codebase_graph_native_graph_builder CODEBASE_GRAPH_NATIVE_BULK_STAGING=rust/target/debug/codebase_graph_native_graph_builder ./.venv/bin/pytest tests/test_materializer.py -q Tested: ./.venv/bin/pytest tests/test_native_bulk_staging.py tests/test_native_graph_builder.py tests/test_golden_graph_parity.py tests/test_materializer.py tests/test_schema.py -q Tested: ./.venv/bin/pytest -q Tested: ./.venv/bin/ruff check . Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Performance benchmark comparison for the new native bulk staging path
Add a native TSNORM protocol to the Rust helper so profiled tree-sitter syntax trees can be reconstructed, marked with data-driven capture mappings, and returned through the existing ParserQueryResult and ParseBundle contracts. Python still owns parser creation, language profile loading, raw tree-sitter access, and fallback behavior. Constraint: Scryer keeps parser registry and materializer orchestration in Python; Rust only replaces deterministic Universal Tree-sitter Adapter internals behind the native opt-in. Constraint: Language profile rules must remain data-driven, so capture mappings, root node types, and context rules are encoded into the native protocol instead of hard-coded per language. Constraint: No new Rust dependencies; the native helper uses the existing binary and line protocol approach. Rejected: Add Rust tree-sitter grammar dependencies now | would broaden packaging and FFI scope beyond the approved normalization boundary. Rejected: Make native normalization the default | parity is proven under opt-in first, matching the Rust Rewrite compatibility plan. Confidence: high Scope-risk: moderate Directive: Keep CODEBASE_GRAPH_NATIVE as the opt-in gate until large-repo parity and benchmark tasks validate default replacement. Tested: cargo fmt --manifest-path rust/Cargo.toml -- --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/pytest tests/test_tree_sitter_adapter.py -q Tested: CODEBASE_GRAPH_NATIVE=1 CODEBASE_GRAPH_NATIVE_TREE_SITTER_NORMALIZER=rust/target/debug/codebase_graph_native_graph_builder ./.venv/bin/pytest tests/test_supported_language_parsing.py -q Tested: ./.venv/bin/pytest tests/test_tree_sitter_adapter.py tests/test_supported_language_parsing.py tests/test_profiled_parser_registry.py tests/test_language_profiles.py -q Tested: CODEBASE_GRAPH_NATIVE=1 CODEBASE_GRAPH_NATIVE_GRAPH_BUILDER=rust/target/debug/codebase_graph_native_graph_builder CODEBASE_GRAPH_NATIVE_TREE_SITTER_NORMALIZER=rust/target/debug/codebase_graph_native_graph_builder CODEBASE_GRAPH_NATIVE_BULK_STAGING=rust/target/debug/codebase_graph_native_graph_builder ./.venv/bin/pytest tests/test_golden_graph_parity.py tests/test_materializer.py -q Tested: ./.venv/bin/pytest -q Tested: ./.venv/bin/ruff check . Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Large-repository parser normalization benchmark
Local-only symbol table construction, reference resolution, call/type promotion, and evidence-link preparation now share a native batch path behind CODEBASE_GRAPH_NATIVE. The Python writer still owns provider-backed enrichment and public report/evidence compatibility, so callers keep the same API surface and fallback behavior. Constraint: Provider-backed enrichment remains Python-owned for this Rust Rewrite task Rejected: Port semantic provider resolution to Rust | acceptance criteria explicitly keeps provider-backed enrichment in Python unless moved later Confidence: high Scope-risk: moderate Directive: Keep provider_results on the Python path until provider semantics are explicitly migrated and tested Tested: cargo test --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml -- -D warnings; ./.venv/bin/ruff check .; ./.venv/bin/pytest tests/test_native_semantic_enrichment.py tests/test_semantic_symbol_table.py tests/test_semantic_reference_resolver.py tests/test_semantic_call_type_resolver.py tests/test_semantic_enrichment_writer.py -q; ./.venv/bin/pytest -q (257 passed, 3 skipped); ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Large-repository semantic enrichment performance benchmarks
Repository walking, exclusion filtering, SHA-256 file hashing, and manifest diff selection now have an opt-in native batch path. Python still owns parser selection, locks, atomic database swaps, manifest writes, and fallback behavior so lifecycle semantics stay unchanged. Constraint: Materializer lifecycle, store writes, and atomic recovery remain Python-owned Rejected: Move parser execution or lock handling into Rust | the Rust Rewrite boundary keeps orchestration in Python first Confidence: high Scope-risk: moderate Directive: Keep native scan/diff payloads derived from ParserRegistry suffixes and manifest contract constants to preserve changed/full rebuild decisions Tested: cargo test --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml -- -D warnings; ./.venv/bin/ruff check .; ./.venv/bin/pytest tests/test_materializer.py -q; ./.venv/bin/pytest -q (259 passed, 3 skipped); ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Large-repository scan benchmark speedup
Benchmark output now reports throughput and peak RSS so Python-default and native-opt-in runs can be compared on wall-clock, files/sec, nodes/sec, edges/sec, and memory. The Rust rewrite document records the 2026-06-16 local benchmark evidence and keeps native execution opt-in because the current Rust path is slower on completed runs and native semantic enrichment failed the practical benchmark window. Constraint: Benchmark evidence comes from one completed local repository run plus an interrupted NumPy probe, not a representative multi-repository corpus. Rejected: Switch Rust materialization defaults now | completed native opt-in runs were slower and semantic-enabled native full materialization did not finish promptly. Confidence: medium Scope-risk: narrow Directive: Do not flip CODEBASE_GRAPH_NATIVE defaults until semantic-enabled native full benchmarks complete with parity and measured speedups on representative large repositories. Tested: ./.venv/bin/ruff check . Tested: ./.venv/bin/pytest tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/pytest -q Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: representative multi-repository benchmark suite; phase-level timing for parser normalization versus graph builder versus bulk loader.
The native Rust path now has user-facing build and usage guidance, release-gate compatibility checks, and explicit maintenance rules for stable graph IDs. The docs keep Rust framed as internal accelerators behind Python-owned CLI, MCP, state, and result-shape contracts. Constraint: Benchmark evidence does not justify a default switch and native semantic enrichment still needs phase-level performance work. Rejected: Document Rust as a production default | the benchmark task showed completed native opt-in runs were slower and semantic-enabled native full materialization did not complete promptly. Confidence: high Scope-risk: narrow Directive: Keep Rust documentation aligned with Scryer's Python-owned materialization boundary unless the architecture model is intentionally updated. Tested: ./.venv/bin/ruff check README.md docs/release.md docs/rust_rewrite.md tests/test_rust_rewrite_baseline.py Tested: ./.venv/bin/pytest tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/pytest -q Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: hosted release workflow rendering for future release-please notes.
Rust now exposes a modular batch materialization library that scans, diffs, builds graph and staging rows, and writes LadybugDB through the lbug crate while Python keeps CLI, config, fallback, manifest, and atomic swap ownership. Constraint: Semantic enrichment remains Python-owned for v1 Constraint: Native batch materialization stays opt-in behind CODEBASE_GRAPH_NATIVE=1 Rejected: Keep Python executing native COPY statements | user requested Rust-owned LadybugDB writing Confidence: medium Scope-risk: broad Directive: Do not enable the native batch path by default until parity benchmarks cover representative repositories Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo test --manifest-path rust/Cargo.toml --features python-extension Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: cargo clippy --manifest-path rust/Cargo.toml --features python-extension -- -D warnings Tested: ./.venv/bin/python -m pytest -q Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Native-vs-Python benchmark comparison on a larger mixed-language repository
Native syntax materialization now receives the canonical Python ontology payload, passes relation specs into Rust tree graph construction, and validates emitted syntax edges against that contract. The Rust parser path also uses native tree-sitter grammars for the supported profiled languages while Python retains the CLI/config/fallback surface. Constraint: Python ontology remains the public schema authority for node and relation compatibility Constraint: Native Ladybug writes must work without relying on the JSON extension loader in this local lbug build Rejected: Maintain a separate Rust relation allowlist as the source of truth | it drifted from Python and produced invalid connector writes Rejected: Return graph rows across PyO3 for Python-side validation | the batch kernel contract keeps rows inside Rust and returns only control metadata Confidence: medium Scope-risk: broad Directive: Do not add new native relation pairs without passing the Python ontology schema through the request path Tested: cargo test --manifest-path rust/Cargo.toml --quiet Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/pytest tests/test_materializer.py::test_native_syntax_batch_payload_includes_canonical_ontology_schema tests/test_native_materialization.py tests/test_tree_sitter_adapter.py tests/test_native_graph_builder.py tests/test_supported_language_parsing.py tests/test_golden_graph_parity.py -q Tested: CODEBASE_GRAPH_NATIVE=1 CODEBASE_GRAPH_NATIVE_STRICT=1 strict golden fixture materialization Tested: syntax-only native-vs-Python benchmark on golden fixture Not-tested: Full native graph parity; fixture still emits fewer native nodes/edges than Python Not-tested: Native full-repo benchmark completion; strict full-repo native run was stopped after it failed to finish in a useful time window
Rust native syntax materialization was producing graph rows that were not parity-equivalent with the Python GraphBuilder path. This aligns Rust tree traversal, Python-specific normalization, parser-like metadata handling, and root provenance rules with the Python-side ontology behavior, then locks the golden mixed-language fixture with a native-vs-Python type-count parity test. Constraint: Python ontology and GraphBuilder behavior remain the authority for public graph semantics Constraint: Native syntax materialization remains opt-in behind CODEBASE_GRAPH_NATIVE=1 Rejected: Treat Rust parser output as a separate richer ontology | it broke native-vs-Python parity and changed public graph semantics Confidence: medium Scope-risk: moderate Directive: Do not add native semantic rows unless the Python ontology and GraphBuilder path expose the same node and relation behavior Tested: cargo test --manifest-path rust/Cargo.toml --quiet Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/pytest tests/test_native_materialization.py -q Tested: ./.venv/bin/pytest tests/test_tree_sitter_adapter.py tests/test_native_graph_builder.py tests/test_native_bulk_staging.py tests/test_materializer.py tests/test_rust_rewrite_baseline.py -q Tested: env -u CODEBASE_GRAPH_NATIVE -u CODEBASE_GRAPH_NATIVE_STRICT ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Full-repo native benchmark completion; interrupted after exceeding six minutes inside _native.materialize_syntax_batch
The native writer was intercepting JSON COPY statements and replaying them as row-by-row CREATE executions. Letting Ladybug execute the staged COPY statements keeps the Rust path aligned with the Python bulk loader and removes duplicate JSON deserialization work. Constraint: Rust lbug JSON extension loading can fail in local environments, so the Rust unit test tolerates that local loader failure while production still returns it Rejected: Keep the JSON row insertion fallback | it preserves the performance bottleneck this change removes Confidence: high Scope-risk: narrow Directive: Do not reintroduce row-by-row JSON insertion in the native writer without a measured Ladybug COPY regression Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml native_writer Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py Tested: ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Optional full native materialization benchmark, skipped because this local path previously ran for many minutes
Rust lbug links Ladybug statically by default, while LOAD json dlopens a C++ extension that resolves Ladybug APIs from the host process. Exporting the final binary symbols preserves real Ladybug COPY loading without falling back to row-by-row insertion. Constraint: lbug 0.17.x documents -rdynamic as required for extension-capable Rust binaries when statically linked Rejected: Switch to shared Ladybug linkage immediately | the documented static-link export fix resolved the native binary and isolated PyO3 smoke paths Confidence: high Scope-risk: narrow Directive: Do not remove the native build.rs link arg while JSON/FTS extensions are loaded dynamically through Ladybug Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml native_writer -- --nocapture Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py Tested: isolated PyO3 native materialization smoke with database_written=true Not-tested: long native materialization benchmark intentionally skipped
Expose detailed materialization phase timings from both the Rust native path and the Python wrapper so benchmark output can show where time is spent across scan, parse, graph build, staging, DB write, and JSON serialization boundaries. Constraint: Benchmark diagnostics need native internal timings without changing the public materialization protocol shape beyond optional timing metadata. Rejected: Time only inside the benchmark script | that cannot separate Rust scan/parse/build/staging or native database write costs. Confidence: high Scope-risk: moderate Directive: Keep phase timing keys stable for benchmark report consumers. Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml native_writer -- --nocapture Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py Tested: ./.venv/bin/python -m pytest tests/test_materializer.py Not-tested: Full benchmark rerun after adding timers; previous full benchmark was stopped on request. Not-tested: Live PyO3 extension smoke with maturin-built module because maturin is unavailable in the local venv.
Ladybug loads JSON support as a dynamic extension, and that extension resolves engine symbols from the host process. The Rust native path statically links liblbug into a PyO3 cdylib, but Rust cdylib export filtering hid Ladybug's C++ symbols on macOS, causing LOAD json to fail before COPY could run. This build script now generates a macOS cdylib-only export list from the prebuilt liblbug archive when building the Python extension. That keeps normal Rust builds unchanged while making the PyO3 module act as the symbol provider Ladybug's extension loader expects. Constraint: Ladybug 0.17.x JSON extension uses dlopen flat-namespace resolution against host Ladybug symbols. Rejected: Export one missing symbol at a time | the JSON extension has many unresolved lbug::* symbols and failures only move to the next one. Rejected: Use row-by-row JSON fallback | it breaks the native bulk COPY semantics and performance goal. Confidence: high Scope-risk: moderate Directive: Do not remove the macOS cdylib export list unless Ladybug extensions stop resolving against host symbols or lbug ships a shared-library mode used by the PyO3 module. Tested: ./.venv/bin/python -m maturin develop --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: nm -gU src/codebase_graph/_native/_native.cpython-312-darwin.so shows DUMMY_TRANSACTION and ClientContext::getDatabase exported Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml native_writer -- --nocapture Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m pytest tests/test_materializer.py Not-tested: Full benchmark rerun after the loader fix.
Replace the native syntax batch's internal TREEGRAPH and BULK text round-trips with typed Rust rows so graph build and staging no longer retain large serialized protocol strings before Ladybug COPY. Constraint: Legacy CLI/stdin TREEGRAPH and BULK protocol helpers remain available for compatibility. Rejected: Reuse decode_graph_output and encode_bulk_payload internally | preserves the serialization overhead this change is meant to remove. Confidence: high Scope-risk: moderate Directive: Keep native staging COPY statement layout compatible with the Python bulk loader and Ladybug COPY semantics. Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: split pytest runs for tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py and tests/test_materializer.py Not-tested: combined pytest invocation still segfaults after Rust lbug 0.17.1 is loaded before Python real-ladybug 0.15.3 in the same interpreter
Avoid retaining every native GraphPartition by feeding each built partition into staging and manifest accumulators immediately. Connector rows are still materialized at finish so cross-partition endpoints remain valid. Constraint: Public Python response shape and Ladybug COPY staging layout must remain unchanged. Rejected: Disk-backed table merge in this pass | duplicate merge semantics require a larger design and this change targets retained partition vectors only. Confidence: high Scope-risk: moderate Directive: Keep connector validation deferred until finish unless cross-partition edge ordering is redesigned. Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: CODEBASE_GRAPH_NATIVE=1 benchmark full iteration, no FTS/enrichment: total 11.859296s, RSS 3471884288, graph_build 2.248852s, staging 2.164152s, DB write 6.102232s Not-tested: combined pytest invocation remains avoided because Rust lbug 0.17.1 and Python real-ladybug 0.15.3 segfault when loaded sequentially in one interpreter
The native PyO3 materialization path was still paying for generic row maps and dense tree BTreeMap adaptation after the text-protocol removal. This keeps external output deterministic at file/COPY boundaries while letting the hot path use fixed rows, HashMap state, and a dense syntax-node lookup. Constraint: Legacy CLI/stdin text protocols and public Python response shape must remain unchanged Rejected: Change legacy TREEGRAPH parsing to Vec | the parser accepts arbitrary ids and benefits from preserving the existing map semantics Confidence: high Scope-risk: moderate Directive: Keep deterministic sorting at JSON/CSV/COPY boundaries when changing staging internals Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: native benchmark total 21.063s, peak RSS 2529099776 bytes, graph build 4.559s, staging 3.814s Not-tested: combined pytest process because Rust/Python Ladybug versions are known to segfault when loaded together
Move the native materialization row DTOs out of legacy.rs so graph partitioning and staging no longer depend on legacy protocol ownership. The legacy builder still imports the rows for its current compatibility output, keeping task 1 behavior-preserving. Constraint: Public Python response shape, graph IDs, manifests, and staged file layout must remain unchanged Rejected: Move Builder/Node/Edge now | task 1 is intentionally limited to row DTO ownership before replacing the builder Confidence: high Scope-risk: narrow Directive: Keep native modules importing row DTOs from graph_rows, not legacy Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none
Introduce a native_graph module as the PyO3 graph-materialization boundary and route partition building through it. The boundary currently delegates to the legacy typed builder so later tasks can replace internals behind parity tests without changing graph.rs again. Constraint: Legacy CLI/stdin behavior and current graph output must remain unchanged Rejected: Replace Builder internals in this task | task 2 only establishes the native-owned boundary before metadata and traversal rewrites Confidence: high Scope-risk: narrow Directive: Keep graph.rs calling native_graph; replace native_graph internals incrementally behind parity tests Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none
Replace native graph row metadata JSON strings with typed serde_json values so staging can serialize rows directly instead of reparsing Rust-produced strings. Legacy text protocol encoders still use their existing JSON string path. Constraint: Staged JSON shape, graph IDs, manifests, Python APIs, and legacy CLI protocol output must remain unchanged Rejected: Convert legacy Node metadata storage now | task 3 only removes the native row/staging JSON round-trip while preserving compatibility encoders Confidence: high Scope-risk: moderate Directive: Keep JSON string encoding confined to legacy text protocol boundaries Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: native benchmark total 20.073651s, peak RSS 2435383296 bytes, graph build 3.895127s, staging 3.704475s Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: combined pytest process because Rust/Python Ladybug versions are known to segfault when loaded together
Native syntax graph building now uses a compact arena of SyntaxNode references instead of cloning every parsed node into the legacy TsNode shape. The shared traversal abstraction keeps legacy TREEGRAPH compatibility while removing the adapter from the PyO3 materialization path. Constraint: Legacy TREEGRAPH behavior and graph output parity must remain unchanged. Rejected: Rewrite the legacy Builder in the same change | task 4 only removes the TsNode adapter before the row-first builder replacement. Confidence: high Scope-risk: moderate Directive: Keep native syntax traversal on NativeSyntaxArena; TsNode should remain only for legacy TREEGRAPH compatibility until the legacy split/deletion tasks. Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Combined pytest process for native and Python Ladybug coverage due known same-process Ladybug version crash.
The PyO3 native materialization path now constructs GraphNodeRow and GraphEdgeRow values directly in native_graph.rs. NativeBuilder keeps node, edge, relation, and symbol lookup state in HashMap/HashSet collections and sorts only at the typed row return boundary, so legacy Node/Edge objects and Builder are no longer constructed for native graph rows. Constraint: Legacy CLI/text protocols and output parity must remain available while task 6 and task 7 split or delete compatibility code. Rejected: Remove legacy Builder now | compatibility protocols still use it and have a dedicated follow-up task. Confidence: high Scope-risk: moderate Directive: Do not route PyO3 native syntax materialization back through legacy::Builder; keep legacy references in native_graph.rs test-only until the compatibility split. Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: CODEBASE_GRAPH_NATIVE=1 ./.venv/bin/python scripts/benchmark_materialization.py --repo-root . --mode full --iterations 1 --warmups 0 --no-semantic-enrichment --no-fts --state-dir /private/tmp/codebasegraph-native-row-builder-bench Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Combined pytest process for native and Python Ladybug coverage due known same-process Ladybug version crash.
Legacy text protocol handlers now live in legacy_cli.rs and the binary entry point calls that compatibility module directly. Native/shared code no longer imports the old legacy module for hash utilities; hash.rs owns stable SHA-1 partition IDs and SHA-256 file hashes with local regression coverage. Constraint: BULK, TREEGRAPH, TSNORM, SCAN, and SEMANTIC CLI behavior must remain available until the explicit caller audit and deletion/gating task. Rejected: Delete protocol handlers in this split | caller audit and compatibility gating are task 7. Confidence: high Scope-risk: moderate Directive: Keep production native materialization independent of legacy_cli; test-only parity imports are acceptable until task 7 removes or gates obsolete protocols. Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Combined pytest process for native and Python Ladybug coverage due known same-process Ladybug version crash.
The compatibility audit now records which stdin protocols still have Python shell-out callers and why they remain isolated in legacy_cli.rs. Uncalled Rust helper APIs for build_graph_output and write_bulk_staging_output were removed so only active protocol handlers and test parity support remain. Constraint: Python still calls BULK, TSNORM, SCAN, and SEMANTIC through the compatibility binary; deleting those handlers would break opt-in native helper paths. Rejected: Feature-gate all legacy_cli handlers now | production Python callers still need the binary until replacement wrappers exist. Confidence: high Scope-risk: narrow Directive: Before deleting more compatibility protocols, remove or replace the documented Python shell-out callers and rerun the split materialization tests. Tested: rg caller audit for BULK TREEGRAPH TSNORM SCAN SEMANTIC build_graph_output write_bulk_staging_output Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: CODEBASE_GRAPH_NATIVE=1 ./.venv/bin/python scripts/benchmark_materialization.py --repo-root . --mode full --iterations 1 --warmups 0 --no-semantic-enrichment --no-fts --state-dir /private/tmp/codebasegraph-native-compat-audit-bench --output /private/tmp/codebasegraph-task7-bench.json Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Combined pytest process for native and Python Ladybug coverage due known same-process Ladybug version crash.
The Rust native materializer modules now use operation-focused names: partition_builder, syntax_materializer, staging_writer, ladybug_writer, graph_rows, hash, scan, parser, profiles, and legacy_cli. Public Python names and response fields are unchanged, and the architecture note lists the final module map. Constraint: Do not rename Python public APIs, environment variables, or binary names in this pass. Rejected: Rename codebase_graph_native_graph_builder | Python wrappers and tests still use that public compatibility binary name. Confidence: high Scope-risk: narrow Directive: Keep future module names tied to actual responsibilities; do not reintroduce historical names like native_graph.rs for new native materialization work. Tested: cargo fmt --manifest-path rust/Cargo.toml --check Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: ./.venv/bin/python -m maturin develop --release --manifest-path rust/crates/codebase_graph_native/Cargo.toml --features python-extension Tested: ./.venv/bin/python -m pytest tests/test_native_bulk_staging.py tests/test_native_materialization.py tests/test_rust_rewrite_baseline.py -q Tested: ./.venv/bin/python -m pytest tests/test_materializer.py -q Tested: rg stale-name scan for native_graph, graph.rs, staging.rs, ladybug.rs, crate::graph, crate::native_graph, crate::staging, crate::ladybug Tested: CODEBASE_GRAPH_NATIVE=0 ./.venv/bin/codebase-graph setup --repo-root . --mcp-client none Not-tested: Combined pytest process for native and Python Ladybug coverage due known same-process Ladybug version crash.
Native materialization responses can include large snapshots, rebuilt entries, and copy statements. Dropping the self-referential Rust encode timing keeps the response on a single serialization path while preserving decode and database timings. Constraint: Accurate rust_json_encode_seconds inside the returned JSON required serializing the response twice or patching a serialized placeholder Rejected: Patch encode timing into serialized bytes | brittle and still complicates the one-allocation response path Confidence: high Scope-risk: narrow Directive: Do not reintroduce response encode timing inside the same payload without preserving single-pass serialization Tested: cargo check --workspace; cargo clippy --workspace --all-targets --all-features -- -D warnings; cargo test --workspace; ./.venv/bin/pytest tests/test_native_materialization.py
Native syntax materialization only reads request configuration, while the PyO3 caller still needs selected fields for the database write. Borrowing the request removes the full request clone across manifests, profiles, and ontology data without changing payload or response behavior. Constraint: PyO3 entrypoint must still move db_path, include_fts, and schema_statements into the DB write request after materialization Rejected: Move only selected DB fields before materialization | keeps materializer consuming ownership and makes future request borrowing harder Confidence: high Scope-risk: narrow Directive: Keep native materialization request access read-only unless a future change genuinely needs ownership Tested: cargo check --workspace; cargo clippy --workspace --all-targets --all-features -- -D warnings; cargo test --workspace; ./.venv/bin/pytest tests/test_native_materialization.py
Precompute native semantic promotion lookup maps so evidence and type-annotation promotion no longer rescans every edge and node for each semantic evidence item. Constraint: Native benchmark must run in strict mode with LadyBugDB FTS permissions or it can silently fall back in non-strict mode Rejected: Keep scan-based helpers | synthetic hotspot spent 57.10s in semantic_edge_promotion_seconds Confidence: high Scope-risk: narrow Directive: Keep semantic promotion indexes updated when adding new semantic edge types used by promotion helpers Tested: cargo test --manifest-path rust/Cargo.toml Tested: cargo clippy --manifest-path rust/Cargo.toml -- -D warnings Tested: .venv/bin/python -m pytest tests/test_native_materialization.py tests/test_materializer.py::test_materializer_routes_local_semantic_enrichment_to_native_batch tests/test_materializer.py::test_materializer_rejects_native_provider_semantics_in_strict_mode -q Tested: .venv/bin/ruff check . Tested: native hotspot benchmark semantic_edge_promotion_seconds 0.09048s after patch Tested: representative native benchmark mean 21.535499s vs Python mean 51.539962s Not-tested: Graph-summary parity between native and Python remains unresolved
- Introduced `smoke_native_artifact.py` for testing the native artifact's functionality through various commands. - Implemented functions to validate the setup, materialization, health checks, and search capabilities of the native binary. - Added `native_binary.py` to resolve the native product binary path, considering environment variables and repository structure. - Created `version.py` to extract the Rust package version from the Cargo.toml file, ensuring compatibility with Python 3.10 and above.
Move the production crate to the repository root, remove Python implementation and pytest harnesses, drop PyO3 and compatibility binaries, and replace Python release tooling with Rust-native xtask checks. Bundle LadyBug JSON and FTS extension binaries for supported release platforms so the existing INSTALL/LOAD extension path works offline. Constraint: Repository must publish as the root codebase-graph crate with Rust-only CI and crates.io release flow Constraint: LadyBug JSON and FTS extension behavior must remain enabled and offline-capable Rejected: Disable FTS or remove JSON COPY staging | changes product behavior and violates migration requirements Confidence: high Scope-risk: broad Tested: cargo fmt --check; cargo clippy --workspace --all-targets --all-features --locked -- -D warnings; cargo test --workspace --locked; cargo run -p xtask -- release-gate --production --confirm release-environment --confirm hosted-ci-green --confirm private-vulnerability-reporting; cargo build --locked --release --bin codebase-graph; cargo run -p xtask -- smoke-artifact target/release/codebase-graph; cargo publish --dry-run --locked --allow-dirty Not-tested: cargo audit locally because cargo-audit is not installed; live crates.io publish with CARGO_REGISTRY_TOKEN
Keep local automation and MCP state out of version control so status output reflects source changes only. Confidence: high Scope-risk: narrow Tested: git status --short Not-tested: runtime behavior unchanged
Wire production language profiles for Rust, Go, C, C++, and Fortran so setup indexes the languages already advertised by the CLI docs. Parser tests now use production profiles, and setup coverage proves a fresh mixed-language repo materializes and searches each supported language. Constraint: README already documents default ingestion for Rust, Go, C, C++, and Fortran Rejected: Update docs to narrow supported languages | parser dependencies and grammar hooks already exist for the documented languages Rejected: Capture every language construct exhaustively | support target is conservative syntax-level ingestion and search Confidence: high Scope-risk: moderate Directive: Keep parser tests tied to production ProfileSet so runtime support cannot drift from test-only profiles Tested: cargo fmt --check; cargo clippy --workspace --all-targets --all-features --locked -- -D warnings; cargo test --workspace --locked; fresh /private/tmp/codebase-graph-language-smoke setup and graph-search for RustSmoke, GoSmoke, CSmoke, CppSmoke, FortranSmoke Not-tested: Deep semantic resolution quality for every language construct
Keep repo-local executable shims out of version control while preserving a stable path for Codex MCP and AGENTS instructions. Constraint: codebase_graph MCP config expects a project-local .venv/bin/codebase-graph path Constraint: existing source edits currently prevent cargo install from compiling cleanly Rejected: Commit generated binary or virtualenv shim | generated local artifacts should stay untracked Confidence: high Scope-risk: narrow Tested: codebase-graph graph-health --repo-root . Tested: codebase-graph graph-search codebase-graph --repo-root . --no-refresh --detail slim --context-limit 1 Not-tested: Live MCP exposure in the already-running Codex session; MCP tool registry requires session reload
Add planning, watch, Git-aware file selection, configurable ignore rules, and optional parallel/progress materialization controls around the existing manifest-hash correctness model. Plan mode reports exact file-level rebuild/delete/skip/ignore decisions without writing, while materialization keeps atomic database replacement for changed refreshes because current partitions include shared graph rows. Constraint: Existing Ladybug copy path rejects duplicate shared Repository/SourceRoot/dependency rows during partial database writes Rejected: Partial changed-mode database writes | current partition shape emits shared rows that would duplicate or require broader writer semantics Confidence: high Scope-risk: moderate Directive: Do not make changed-mode materialization non-atomic until shared support rows can be safely upserted or excluded from partial copies Tested: cargo test; cargo clippy --all-targets --all-features -- -D warnings; cargo run -- plan --repo-root . --git-diff --json; cargo run -- watch --repo-root . --once --no-semantic-enrichment; cargo run -- setup --repo-root . --mcp-client none --no-semantic-enrichment --json; cargo run -- graph-health --repo-root . --json Not-tested: Long-running watch loop under sustained filesystem churn
Replace the polling watch loop with a notify-backed recursive watcher that feeds an event channel, filters ignored paths before refresh, debounces quiet windows, and bounds sustained churn with a max-wait refresh. The command keeps the existing watch CLI surface, including --poll-ms as a compatibility no-op, while reporting event and changed-path counts in watch status output. Constraint: notify 8.2.0 is the stable cross-platform watcher dependency required by the feature plan Constraint: changed-mode materialization remains atomic through the existing materialize path Rejected: keep snapshot polling as fallback | the requested behavior is OS watcher mode with --poll-ms compatibility only Confidence: high Scope-risk: moderate Directive: Do not reintroduce polling semantics for --poll-ms without an explicit fallback design and tests Tested: cargo test Tested: cargo clippy --all-targets --all-features -- -D warnings Tested: cargo run -- watch --repo-root . --debounce-ms 250 --no-semantic-enrichment --once Tested: cargo run -- setup --repo-root . --mcp-client none --no-semantic-enrichment --json Tested: cargo run -- graph-health --repo-root . --json Not-tested: live sustained churn on every notify backend and operating system
Accept notify event paths whether they arrive as absolute source-root paths, current-directory-relative paths, or source-root-relative paths before applying the existing ignore rules. Also treats coarse backend events and write-close notifications as refresh-worthy while preserving suppression for read/open access-only noise. Constraint: macOS and other notify backends can emit backend-specific path shapes and coarse event kinds Rejected: Depend only on absolute notify paths | live verification showed this assumption is too narrow for robust watcher behavior Confidence: high Scope-risk: narrow Directive: Keep event filtering permissive for non-access backend events; suppress only clearly read/open access-only noise Tested: cargo test Tested: cargo clippy --all-targets --all-features -- -D warnings Tested: live watch smoke outside sandbox refreshed with event_count=3 changed_paths=1 rebuilt=1 Not-tested: live event delivery inside Codex sandbox, which timed out despite succeeding outside sandbox
Add an auto watch backend that probes native notify delivery and falls back to a filter-aware polling loop when events do not arrive. Polling now uses --poll-ms intentionally, while native remains available through --watch-backend native. The fallback path emits an explicit watch fallback status line and then uses the same materialization and refresh status path as the native watcher. Constraint: sandboxed environments can allow notify initialization while suppressing event delivery Rejected: unconditional polling fallback on watcher startup errors only | the observed failure mode is successful startup with no delivered events Rejected: make poll the default backend | native notify should remain the preferred path when the health probe succeeds Confidence: high Scope-risk: moderate Directive: Keep auto mode probe-based; do not remove explicit native and poll backend selections because they are useful for diagnosis Tested: cargo test Tested: cargo clippy --all-targets --all-features -- -D warnings Tested: sandbox smoke auto fallback refreshed with backend=poll event_count=1 changed_paths=1 rebuilt=1 Tested: cargo run -- setup --repo-root . --mcp-client none --no-semantic-enrichment --json Tested: cargo run -- graph-health --repo-root . --json Not-tested: live auto fallback on every notify backend and operating system
The CLI/MCP surface and syntax materializer had grown into large flat files that made ownership boundaries hard to see. This change moves the existing code into focused Rust submodules while preserving command behavior, protocol shapes, and public entrypoints. Constraint: Preserve codebase_graph::product_cli public entrypoints and existing CLI/MCP behavior Rejected: Rewrite behavior while moving files | broader regression risk than a mechanical boundary refactor Confidence: high Scope-risk: moderate Directive: Keep future CLI/MCP additions inside the focused product_cli submodules instead of rebuilding a monolithic command file Tested: cargo fmt --check; cargo test; codebase-graph setup --repo-root . --mcp-client none Not-tested: Manual end-to-end invocation outside the existing CLI and MCP test coverage
Split the internal product CLI and syntax materializer surfaces into focused modules while preserving command behavior, MCP wire shapes, JSON output, and public product_cli entrypoints. Constraint: Preserve CLI flags, MCP protocol shapes, JSON payloads, graph schema assets, and public product_cli entrypoints Rejected: Combine the boundary split with behavior rewrites | would make regression review harder Confidence: high Scope-risk: moderate Directive: Keep subsystem facades explicit; do not reintroduce production use super::* imports Tested: cargo test; cargo fmt --check; codebase-graph setup --repo-root . --mcp-client none Not-tested: Manual interactive MCP client smoke outside existing automated coverage
- Introduced a new module for handling watch commands, including the ability to run watches with different backends (polling or native). - Added support for filtering watch events based on user-defined patterns and paths. - Implemented a batching mechanism to collect watch events and debounce them to reduce noise. - Created a snapshot mechanism to track file changes and detect modifications efficiently. - Enhanced output functionality to provide detailed information about watch events and statuses. - Removed obsolete product_cli graph and install modules, consolidating watch-related functionality into the cli module.
Move the native Rust pipeline into explicit execution, parser, semantic enrichment, and CLI build module boundaries while preserving the existing public CLI command surface. Constraint: codebaseGraph graph instructions require refreshed graph state after material source moves Constraint: src/cli/build must be indexable, so default source exclusions can no longer treat every build path segment as generated output Rejected: Keep shim files at src/parser.rs and src/semantic_enrichment.rs | user explicitly requested removing old flat files Rejected: Rename the user-facing materialize command to build | CLI build is internal-only for compatibility Confidence: high Scope-risk: moderate Reversibility: clean Directive: Keep codebase-graph materialize and plan as the public commands unless a separate breaking-change plan is approved Tested: cargo fmt --check Tested: cargo clippy --workspace --all-targets -- -D warnings Tested: cargo test --workspace Tested: cargo build --workspace Tested: ./target/debug/codebase-graph setup --repo-root . --mcp-client none Tested: codebase graph search finds src/cli/build modules after refresh Not-tested: External consumers importing private internal module paths; public crate functions were re-exported unchanged
Split staging output and native database writing into focused module directories while preserving the CLI surface, materialization response shape, and existing test behavior. Constraint: Requested replacement of flat writer files with staging_writer/ and db_writer/ Rejected: Keep a ladybug_writer compatibility shim | requested structure explicitly removes the old module name Confidence: high Scope-risk: moderate Directive: Keep db_writer as the public database-write module; do not reintroduce crate::ladybug_writer without an explicit compatibility requirement Tested: cargo fmt --check; cargo clippy --workspace --all-targets -- -D warnings; cargo test --workspace; ./target/debug/codebase-graph setup --repo-root . --mcp-client none; graph_search resolved src/db_writer and src/staging_writer paths Not-tested: External downstream crates importing the removed crate::ladybug_writer module
Hard-rename the public CLI command surface to the new product names and add a repo-scoped uninstall path for generated graph state, instruction blocks, and matching MCP client registrations. Constraint: User requested hard renames with no backward-compatible aliases Constraint: Uninstall must remain repo-scoped and preserve unrelated MCP entries and instruction text Rejected: Keep deprecated aliases | request explicitly chose hard rename Confidence: high Scope-risk: moderate Directive: Do not reintroduce old command aliases without an explicit compatibility decision Tested: cargo test --workspace --locked Tested: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings Tested: cargo fmt --all --check Tested: git diff --check Tested: target/debug/codebase-graph install --repo-root . --mcp-client none --instructions-target skip --json Not-tested: Real user MCP client config removal outside temp-file adapters
Move ongoing refresh responsibility out of install and into the long-lived MCP runtime. MCP stdio/http now start a background native watcher with poll fallback, and watcher batches feed explicit candidate paths into materialization while graph tool reads are serialized against refresh writes. Incremental refresh now keeps the graph schema fixed: schema statements come from the declared graph schema, staging filters edges that violate declared endpoint pairs, and retained shared node/edge IDs from unchanged partitions are not copied again. Constraint: Install should remain first-time setup and not be required after every source edit. Constraint: Graph schema must not be changed during refresh. Rejected: Keep using install for refresh | preserves the slow manual workflow the change is meant to remove Rejected: Auto full-rebuild on missing relation endpoint pairs | changes schema as a side effect of refresh Confidence: high Scope-risk: moderate Directive: Do not make install refresh existing graph state; use build/watch/MCP runtime for ongoing updates. Directive: Refresh must conform to the declared graph schema instead of evolving DB relation tables from observed rows. Tested: cargo test --workspace --locked; cargo clippy --workspace --all-targets --all-features --locked -- -D warnings; cargo fmt --all --check; git diff --check; release install smoke; long-lived MCP editor workload with atomic saves, create, rename, delete, nested file churn Not-tested: Multi-hour editor session
Update generated and checked-in codebaseGraph instructions so agents do not rerun install for graph refresh. Refresh is now described as automatic through the running MCP server, with build reserved for explicit manual rebuilds. Constraint: MCP/watch now owns ongoing graph freshness after first install. Rejected: Keep install as the recommended refresh command | contradicts automatic refresh behavior and causes unnecessary rebuilds Confidence: high Scope-risk: narrow Directive: Do not reintroduce instructions that tell agents to refresh by rerunning install. Tested: cargo test --workspace --locked install; cargo test --workspace --locked dispatch_materialize; cargo clippy --workspace --all-targets --all-features --locked -- -D warnings; cargo fmt --all --check; git diff --check Not-tested: Full workspace test suite for this docs-only update
The CI package job tried to build a universal macOS binary on macos-latest, but GitHub now runs that label on arm64 and the lbug static archive is architecture-specific. Build native macOS arm64 and Intel artifacts instead, and normalize LadyBug COPY paths so Windows test and smoke runs do not feed backslash escapes into quoted COPY statements. Constraint: lbug 0.17.1 exposes architecture-specific prebuilt static libraries and no source-build feature. Constraint: GitHub-hosted macos-latest is arm64; Intel macOS requires an explicit Intel runner label. Rejected: Keep the lipo-based universal artifact | x86_64 linking picks up arm64 lbug objects on arm64 runners. Confidence: high Scope-risk: moderate Directive: Do not reintroduce universal macOS packaging unless lbug supports cross-architecture static libraries. Tested: cargo fmt --all --check Tested: cargo test --workspace --locked Tested: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings Tested: cargo run -p xtask -- release-gate Tested: cargo build --locked --release --bin codebase-graph Tested: cargo run -p xtask -- smoke-artifact ./dist/smoke/codebase-graph Not-tested: Windows runner execution after this commit; local host is macOS.
The macOS arm64 hosted runner flushes the sustained-churn watch batches in roughly half a second, which is still bounded behavior but slower than the previous local-machine 200ms assertion. Keep the behavioral checks for batching and event collection while allowing a one-second ceiling to avoid CI-only timing flakes. Constraint: GitHub-hosted macos-latest runs on macos-15-arm64 with lower and more variable filesystem timing. Rejected: Remove the timing check entirely | a broad upper bound still catches stalled batching. Confidence: high Scope-risk: narrow Directive: Keep these tests focused on bounded batching behavior, not workstation-speed latency. Tested: cargo test --workspace --locked watch_batch_flushes_under_sustained_churn Tested: cargo test --workspace --locked watch_poll_batch_flushes_under_sustained_churn Tested: cargo fmt --all --check Tested: cargo test --workspace --locked Not-tested: Direct rerun on macos-latest before push.
Windows runners surface extended temp paths during artifact smoke tests, and LadyBug does not resolve COPY paths with the //?/ prefix after separator normalization. macOS runners can also schedule the auto-watch integration test slowly enough for the real file event to satisfy native probing instead of forcing poll fallback. Normalize extended Windows COPY paths before emitting LadyBug statements, keep the watcher environment lock usable after a failed assertion, and let the auto-watch test accept either valid probe resolution path while lower-level tests still assert timeout fallback behavior. Constraint: GitHub-hosted Windows temp dirs can use extended path prefixes that LadyBug does not match in COPY patterns. Rejected: Restore universal macOS package builds | lbug static archives remain architecture-specific on hosted runners. Confidence: high Scope-risk: narrow Directive: Keep watcher tests tolerant of scheduler delays when they exercise real notify backends. Tested: cargo test --workspace --locked Tested: cargo fmt --all --check Tested: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings Tested: cargo build --locked --release --bin codebase-graph Tested: cargo run -p xtask -- smoke-artifact ./target/release/codebase-graph Tested: git diff --check
GitHub macos-latest now runs the package and test jobs on macOS arm64, where the lbug crate emits force-loaded static archives that include duplicate component objects. Darwin's default linker rejects those duplicate symbols, even though the same code links on the Intel macOS runner. Scope the Darwin duplicate-symbol suppression to macos-latest CI and release artifact builds, and normalize Windows verbatim paths before comparing watcher-relative event paths. The Windows test failure came from canonical source roots using an extended prefix while the relative event path was joined from the normal current directory. Constraint: lbug 0.17.1 controls the static archive link directives; the repository does not own the emitted force_load list. Rejected: Disable macos-latest native package checks | arm64 release artifacts are required. Rejected: Apply duplicate-symbol suppression on all platforms | only Darwin arm64 showed this linker behavior. Confidence: high Scope-risk: narrow Directive: Keep the macOS RUSTFLAGS scoped to hosted arm64 unless lbug changes its static archive layout. Tested: cargo fmt --all --check Tested: cargo test --workspace --locked Tested: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings Tested: RUSTFLAGS='-C link-arg=-Wl,-multiply_defined,suppress' cargo test --workspace --locked watch_auto_backend_refreshes_after_probe_resolution Tested: RUSTFLAGS='-C link-arg=-Wl,-multiply_defined,suppress' cargo build --locked --release --bin codebase-graph Tested: cargo run -p xtask -- smoke-artifact ./target/release/codebase-graph Tested: git diff --check
The Windows-only watcher path helper used String::as_ref(), which left the PathBuf conversion target ambiguous under cfg(windows). GitHub Actions exposed the issue because local macOS builds do not compile that branch. Constraint: Windows-specific code is verified primarily by GitHub's Windows runner because local macOS lacks the MSVC C toolchain. Rejected: Broaden the helper beyond cfg(windows) | unnecessary surface area for a one-line compile fix. Confidence: high Scope-risk: narrow Tested: cargo fmt --all --check; cargo test --workspace --locked watch_filter_accepts_relative_notify_paths; cargo test --workspace --locked; cargo clippy --workspace --all-targets --all-features --locked -- -D warnings; git diff --check Not-tested: Full local x86_64-pc-windows-msvc check cannot complete on macOS without the MSVC C toolchain.
The hosted macos-latest arm64 release build still rejected duplicate symbols from lbug's bundled static archives even with multiply_defined suppression. Adding ld_classic to only the arm64 package and release artifact builds matches the local arm64 release build that links and smokes successfully. Constraint: lbug vendors overlapping static archives that emit duplicate symbols on Apple arm64 release links. Rejected: Apply ld_classic to every macOS job | tests and Intel packages already pass without widening deprecated linker usage. Confidence: medium Scope-risk: narrow Directive: Keep ld_classic scoped to macos-latest packaging until lbug or Apple ld no longer requires duplicate-symbol suppression. Tested: RUSTFLAGS='-C link-arg=-Wl,-ld_classic -C link-arg=-Wl,-multiply_defined,suppress' cargo build --locked --release --bin codebase-graph; cargo run -p xtask -- smoke-artifact ./target/release/codebase-graph; cargo run -p xtask -- release-gate; git diff --check Not-tested: Hosted macos-latest arm64 package job before push.
GitHub runners failed before compiling because Cargo's crates.io fetch for cxxbridge-cmd hit curl error 16 in the HTTP2 framing layer. Disabling Cargo HTTP multiplexing at workflow scope keeps registry downloads on the more conservative transport path across CI and release jobs. Constraint: The failure happens before project code runs and can affect any cargo command that fetches crates. Rejected: Only rerun the failed job | reruns can pass but leave the same HTTP2 failure mode in place. Confidence: medium Scope-risk: narrow Directive: Keep this workflow env unless Cargo/GitHub runner HTTP2 registry fetches prove stable for this dependency graph. Tested: cargo run -p xtask -- release-gate; git diff --check Not-tested: Hosted runner registry download after push.
Hosted macos-latest arm64 release packaging still failed after linker-flag suppression because lbug source builds force-load liblbug plus each bundled native archive, producing thousands of duplicate symbols. The package and release artifact jobs now use the pinned prebuilt static liblbug archive only for macOS arm64, while Linux, Windows, and Intel macOS keep their existing native package paths. Constraint: lbug 0.17.1 emits static:+whole-archive for both liblbug and bundled native dependencies on Rust 1.82+, which breaks hosted macOS arm64 release linking. Constraint: CI also saw transient crates.io HTTP/2 framing errors, so Cargo registry retries are raised across all workflow jobs. Rejected: Add more macOS linker suppression flags | hosted macOS arm64 still failed with ld_classic and multiply_defined suppression. Rejected: Switch every OS package to prebuilt lbug | the failure is isolated to macOS arm64, and other supported OS package jobs already pass natively. Confidence: high Scope-risk: narrow Directive: Keep prebuilt_lbug scoped to macos-latest arm64 unless another OS proves the same duplicate-archive failure mode. Tested: LBUG_LIBRARY_DIR=/tmp/codebasegraph-lbug-prebuilt/lib LBUG_INCLUDE_DIR=/tmp/codebasegraph-lbug-prebuilt/lib cargo build --locked --release --bin codebase-graph; cargo run -p xtask -- smoke-artifact ./target/release/codebase-graph; cargo run -p xtask -- release-gate; git diff --check Not-tested: Hosted macos-latest arm64 package job after this commit.
The Intel macOS package job stayed on the slow source-build path while arm64 used a pinned prebuilt lbug archive. Parameterizing the archive lets both macOS package variants use matching prebuilt static archives, while Linux and Windows continue their native package builds. Constraint: macOS package jobs are sensitive to lbug's native archive layout and hosted runner linker behavior. Rejected: Leave macos-15-intel on source build | it kept the supported OS matrix asymmetrical and slow after fixing arm64. Confidence: high Scope-risk: narrow Directive: Keep lbug_archive explicit per macOS package row so architecture changes are visible in the matrix. Tested: curl -L --head https://github.com/LadybugDB/ladybug/releases/download/v0.17.1/liblbug-static-osx-x86_64.tar.gz; cargo run -p xtask -- release-gate; git diff --check Not-tested: Hosted macos-15-intel package job after this commit.
The macOS package jobs already avoid the lbug source-build duplicate-symbol issue by using the upstream prebuilt static archive. The macOS test job still built lbug from source, so its test binary hit the same duplicate native symbols during linking. This moves the macOS arm64 test lane onto the same external-library path while leaving Linux and Windows on their existing source-build path. Constraint: lbug 0.17.1 source builds link liblbug and bundled native dependency archives separately on hosted macOS arm64 Rejected: Keep linker duplicate suppression only | it did not prevent the test binary link failure on the hosted runner Confidence: high Scope-risk: narrow Tested: LBUG_LIBRARY_DIR=/tmp/codebasegraph-lbug-prebuilt/lib LBUG_INCLUDE_DIR=/tmp/codebasegraph-lbug-prebuilt/lib cargo test --workspace --locked Tested: LBUG_LIBRARY_DIR=/tmp/codebasegraph-lbug-prebuilt/lib LBUG_INCLUDE_DIR=/tmp/codebasegraph-lbug-prebuilt/lib cargo run -p xtask -- release-gate Tested: git diff --check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
codebase-graphwith CLI, graph, install, MCP, watch, and formatting modulesTesting