Skip to content

Release 9.3.0 — large-repo hardening + Cartographer fast-tier visibility#236

Merged
SimplyLiz merged 16 commits into
mainfrom
develop
Jul 13, 2026
Merged

Release 9.3.0 — large-repo hardening + Cartographer fast-tier visibility#236
SimplyLiz merged 16 commits into
mainfrom
develop

Conversation

@SimplyLiz

Copy link
Copy Markdown
Owner

Summary

Cuts 9.3.0. Hardens the vendored Cartographer graph engine for very large C/C++ trees, re-vendors it at 4.0.0, parallelizes the hot paths, and adds a ckb doctor check so users can confirm the SCIP-free fast tier is linked.

Highlights

  • FFI crash fixed on very large repos — a full Linux-kernel checkout (~64k C/H files) aborted the whole host process via two unbounded recursions. Recursive tree-sitter walkers now run on a 256 MB-stack pool, tarjan_scc is iterative, and walkers carry a hard depth ceiling (cap 50,000; measured real peak 3,348). Verified with an FFI regression test.
  • Re-vendor CodeCartographer 4.0.0 — FFI/C-ABI unchanged; pulls upstream import-resolution correctness, doc-subgraph exclusion, UTF-8-safe truncation, resolved reach edges, churn/coupling noise filtering.
  • Parallel, deterministic betweenness — Brandes pass over dense index-keyed buffers across cores, summed in fixed source order → bit-identical regardless of core count. Godot: full rebuild 3,197ms → 76ms (~42×).
  • Directory-level rollup — fold the file graph to directory granularity so bridges/cycles/god-modules/health describe subsystems, not files (the comprehensible view on huge trees).
  • Betweenness topology cache — non-import edits reuse cached centrality, skipping the whole Brandes pass.
  • ckb doctor cartographer check — reports whether the fast structural tier is linked (CGO-free builds warn with a make build hint).

Testing

  • -tags cartographer build clean; FFI-tagged tests green.
  • Measured on the Linux kernel (64,494 files, 362,312 edges): deterministic health/bridges/cycles, cold ~26–34s, peak RSS 4.24 GB, no crash.
  • End-to-end MCP smoke on the 64k-file kernel: server handshake → getArchitecture returned a module-level result via the fast tier with no SCIP index, no crash.

Version

9.2.0 → 9.3.0 (version.go, npm/package.json, sarif testdata, CHANGELOG).

🤖 Generated with Claude Code

SimplyLiz and others added 15 commits April 25, 2026 08:49
Bring the static numbers in README.md back in sync with the
released binary: startup-banner example (v9.0.0 → v9.2.0,
14/76 → 25/110), preset tool counts (core 14→25, review
19→42, full 80+→110, etc.), "20 quality checks" → 21,
"LIP v2.0" → "LIP v2.3+", and drop the "v9.0" prefix on the
Compound Operations heading. The npm version shield is dynamic
so no change needed there.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Upstream renamed the crate; CKB follows. Mechanical rename only — no
functional change to existing call sites; the Go API surface is identical.

- Vendor path third_party/cartographer/mapper-core/cartographer/ →
  third_party/nyx-navigator/ (flattened; stale Python tooling and examples
  no longer carried).
- Build tag cartographer → navigator. make build / make test resolve
  build-navigator / test-navigator. make build-fast (no Rust toolchain)
  unchanged.
- Static lib libcartographer.a → libnavigator.a; FFI prefix cartographer_*
  → navigator_*; header cartographer.h → navigator.h; sync script
  scripts/sync-cartographer.sh → scripts/sync-nyx-navigator.sh.
- Go package internal/cartographer → internal/navigator; error type
  CartographerError → NavigatorError.
- ckb status backend label "cartographer" → "navigator".

New FFI surfaces available for follow-up wiring (not yet exposed):
navigator_poll_changes, navigator_doc_index, navigator_doc_context,
navigator_query_docs. The navigator_render_architecture FFI shape is
unchanged but accepts new format values (sequence, class, quadrant, er,
ascii, html, plus cross-file sequence with an entry-point) — these can
flow through getArchitecture / renderArchitecture without further bridge
work.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Content edits to renamed files: build tag (//go:build navigator),
package declaration (package navigator), C.cartographer_* → C.navigator_*
calls in bridge.go, CFLAGS / LDFLAGS to third_party/nyx-navigator,
header include navigator.h, and corresponding string updates in the
sync script.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three small fixes that surfaced from test-driving the navigator rename:

1. review_coupling.go: hidden-coupling pass only filtered the *missing*
   side for noise; when the *changed* side was itself a noise file
   (.gitignore, CHANGELOG.md, etc.) every co-change still got flagged.
   Added the symmetric filter; also added third_party/ to the noise
   prefix list (vendored code, same intent as vendor/ and node_modules/);
   and capped findings at 3 per changed file to match the regular pass.
   On the navigator rename PR: 779 → 43 hidden-coupling findings (95%
   reduction).

2. impact_outgoing_test.go: panic from "send on closed channel" race
   in the LIP fake. Cleanup closed reqC while connection-handler
   goroutines were still in flight on `reqC <- req`. Added a
   sync.WaitGroup for handlers and a tracked list of accepted conns;
   cleanup now closes the conns first, drains the handlers, then closes
   reqC. Verified clean across 20× stress under -tags navigator.

3. wide_result_budget_test.go: maxGetHotspotsBytes 10000 → 13000.
   Under -tags navigator each hotspot is augmented with up to 3
   co-change partners from navigator.GitCochange (~1.6KB extra at
   limit=20). The 10000 budget was set when the augmentation didn't
   exist — the augmentation is intentional value-add, so accommodate
   it rather than truncate.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…p nyx naming

Re-vendors the Rust crate from upstream CodeCartographer 1.3.3 (was the interim
nyx-navigator 1.1.0 fork) and renames the CKB side back to `cartographer`:

- third_party/nyx-navigator -> third_party/cartographer
- internal/navigator -> internal/cartographer (package cartographer,
  CartographerError, status backend id "cartographer")
- build tag navigator -> cartographer; FFI navigator_* -> codecartographer_*;
  header navigator.h -> codecartographer.h; -lnavigator -> -lcode_cartographer
- scripts/sync-nyx-navigator.sh -> scripts/sync-cartographer.sh
- Makefile/.gitignore updated; Makefile runs the localize script via `bash`
  so a re-vendor (rsync drops the exec bit) doesn't break `make build`

Bundles the upstream go.mod-aware resolver fix, which eliminated fabricated
dependency edges that leaked to users through getArchitecture, getBlastRadius,
renderArchitecture, getModuleContext, analyzeImpact, prepareChange, and the
review layers/arch-health checks. On CKB itself: edges 996 -> 769 (~23%
fabricated), cycles 22 -> 0, bridges 244 -> 187, health 38 -> 71.

Also pulls upstream 1.2-1.3 features: six new languages (Java, C#, Ruby,
Kotlin, Swift, PHP) with call graphs and class diagrams, symbol-aware BM25
search, scope-qualified reach_symbol, orientation-first ranking, and O(N)
import resolution.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-vendors the upstream edge-confidence change: every GraphEdge carries a
`resolution` tag (exact | suffix | fuzzy), now exposed on
cartographer.GraphEdge. Cycle detection and the "will create a cycle"
prediction ignore fuzzy edges, so false cycles can't rest on a low-confidence
stem guess — a safety net for the languages that still resolve heuristically,
matching the go.mod backstop already in place for Go.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…endor)

Re-vendors the upstream resolver extension: extensionless imports that carry
directory structure (Python `from app.services import auth`, monorepo
`components/Button`) resolve against the full path — module files and package
dirs, preferring __init__.py / index.* entry points — instead of the last stem.
Unresolvable qualified imports are external (no edge). Includes the Python
extractor fix that preserves import keywords so dotted paths normalize.

Cross-package same-name collisions (two `auth.py`) and third-party imports no
longer fabricate edges, extending the go.mod precision to Python and pathed
TS/JS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-vendors the resolver change: JS/TS relative imports (`./bar`,
`../models/user`) resolve against the source file's directory (path-precise,
`..` walks up, package dirs use index.*); unresolved relatives are dangling
refs (no edge). `export … from './x'` re-exports are now captured. Completes
the standalone-resolver story alongside Go (go.mod) and Python (qualified
dotted paths).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…re-vendor)

Re-vendors the change that computes all structural metrics (bridges, cycles,
god modules, layer violations, blast radius) on non-fuzzy edges only, so
getArchitecture/getBlastRadius/analyzeImpact/prepareChange and the review
layers/arch-health checks can't surface a finding built on a fabricated edge.

CKB side: review's PR-split clustering (addCartographerEdges) skips fuzzy edges
so a split recommendation isn't driven by a low-confidence dependency.

No change to CKB's own numbers (Go edges are all exact/suffix); the effect is
on polyglot targets CKB analyzes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…llback (re-vendor)

Re-vendors the scaling work: parallel edge resolution and parallel betweenness
centrality in rebuild_graph (the latter was the dominant serial cost at scale),
plus an O(1) directory-suffix fallback (by_dir_last index) replacing the old
O(dirs)-per-import scan. Betweenness node order is sorted before sampling so
bridge/god-module counts are deterministic.

Measured on a 40k-file tree: 56s -> 12.7s (~4.4x); Godot 16s -> ~2s (~8x), same
results. CKB uses the parallel FFI path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (re-vendor)

Re-vendor Cartographer with the betweenness rewrite. Centrality was 98% of a
graph rebuild; it's now index-based + parallel (bit-identical across core
counts) and topology-keyed cached so non-import edits skip the Brandes pass.

Godot via the FFI path: full rebuild 3,197ms -> 76ms (~42x), identical output
(3182 bridges / 68 god-modules / 0 cycles). The cache is the incremental win
for watch/serve sessions and scales with repo size. FFI-tagged query tests
green; full -tags cartographer build clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (re-vendor)

Re-vendor Cartographer with directory rollup and the resolution determinism
fixes. Rollup folds the file-level graph to directory granularity and runs the
full structural analysis on subsystems instead of files — on Godot, 4336 files
/ 3182 bridges becomes 160 subsystems / 139 bridges with actionable per-
subsystem hints. Available via rebuild_graph_rolled_up(depth).

Also picks up two determinism fixes (sorted resolver candidates + strongest-
resolution edge dedup) that make the graph fully reproducible run-to-run.

FFI-tagged query tests green; full -tags cartographer build clean. File-level
metrics unchanged (3182 bridges / 60.0 on Godot).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
….0.0 + fixes)

Re-vendor CodeCartographer at 4.0.0 plus the very-large-repo crash fixes. On a
full Linux-kernel checkout (~64k C/H files) the vendored engine aborted the
whole process — including the CKB host via the FFI — through two unbounded
recursions (recursive tree-sitter walkers on deeply nested macro-generated C,
and recursive petgraph tarjan_scc). Parse now runs on a 256 MB-stack pool and
cycle detection uses an iterative Tarjan.

Verified: Health("/tmp/linux") via cgo completes without crashing (new
regression test in internal/cartographer, skips if /tmp/linux absent). FFI
C-ABI unchanged (no bridge changes). FFI-tagged query tests green; full
-tags cartographer build clean. Directly de-risks the very-large-C++-repo case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-vendor the walker depth guard: a hard recursion ceiling (cap 50,000; the
Linux kernel's measured real peak is 3,348) as a belt against adversarial or
machine-generated deep nesting, on top of the 256 MB parse-pool stack. Prevents
a pathological single file from aborting the CKB host via the FFI. Verified:
Health("/tmp/linux") via FFI completes without crashing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a `cartographer` check to `ckb doctor` so a user can confirm the
SCIP-free structural fast tier (dependents, blast-radius, rollup) is
actually linked into the binary. CGO-free npm/Homebrew builds warn with
a `make build` fix; source builds report the linked engine version.

Bumps version 9.2.0 → 9.3.0 (version.go, npm, sarif testdata) and cuts
the [Unreleased] CHANGELOG section as [9.3.0], covering the large-repo
FFI crash fixes, depth guard, 4.0.0 re-vendor, parallel betweenness,
directory rollup, and this doctor check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

🟢 Change Impact Analysis

Metric Value
Risk Level LOW 🟢
Files Changed 86
Symbols Changed 82
Directly Affected 0
Transitively Affected 0

Blast Radius: 0 modules, 0 files, 0 unique callers

📝 Changed Symbols (82)
Symbol File Type Confidence
.gitignore .gitignore modified 30%
CHANGELOG.md CHANGELOG.md modified 30%
Makefile Makefile modified 30%
README.md README.md modified 30%
cmd/ckb-bench/main.go cmd/ckb-bench/main.go modified 30%
internal/cartographer/bridge.go internal/cartographer/bridge.go modified 30%
internal/cartographer/bridge_stub.go internal/cartographer/bridge_stub.go modified 30%
internal/cartographer/types.go internal/cartographer/types.go modified 30%
internal/mcp/tool_impls.go internal/mcp/tool_impls.go modified 30%
internal/mcp/tool_impls_v65.go internal/mcp/tool_impls_v65.go modified 30%
internal/mcp/tools.go internal/mcp/tools.go modified 30%
internal/query/doctor.go internal/query/doctor.go modified 30%
internal/query/review.go internal/query/review.go modified 30%
internal/query/review_coupling.go internal/query/review_coupling.go modified 30%
internal/query/review_split.go internal/query/review_split.go modified 30%
+67 more

Recommendations

  • ℹ️ coverage: 82 symbols have low mapping confidence. Index may be stale.
    • Action: Run 'ckb index' to refresh the SCIP index

Generated by CKB

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

NFR Tests ✅ 39 unchanged

Comparing PR against main branch (dynamic baseline).

Regressions: 0 ✅

Thresholds: WARN ≥ +5% • FAIL ≥ +10%

All scenarios
Scenario Change Actual (B) Base (B) Time
analyzeChange / large +0.0% 193,169 193,169 694µs
analyzeChange / medium +0.0% 38,575 38,575 178µs
analyzeChange / small +0.0% 4,046 4,046 32µs
analyzeChange / xlarge +0.0% 387,417 387,417 1.271447ms
analyzeImpact / large +0.0% 17,966 17,966 96µs
analyzeImpact / small +0.0% 1,924 1,924 19µs
batchGet / large +0.0% 11,789 11,789 52µs
batchGet / small +0.0% 4,733 4,733 36µs
batchSearch / large +0.0% 90,816 90,816 316µs
batchSearch / medium +0.0% 18,036 18,036 95µs
batchSearch / small +0.0% 3,379 3,379 23µs
explore / large +0.0% 94,262 94,262 498µs
explore / small +0.0% 4,253 4,253 32µs
findReferences / large +0.0% 445,943 445,943 3.533514ms
findReferences / medium +0.0% 44,123 44,123 539µs
findReferences / small +0.0% 4,395 4,395 57µs
getAffectedTests / large +0.0% 7,521 7,521 71µs
getAffectedTests / medium +0.0% 3,110 3,110 25µs
getAffectedTests / small +0.0% 903 903 15µs
getAffectedTests / xlarge +0.0% 14,870 14,870 118µs
getArchitecture / large +0.0% 6,690 6,690 51µs
getArchitecture / small +0.0% 960 960 14µs
getCallGraph / deep +0.0% 15,238 15,238 80µs
getCallGraph / shallow +0.0% 887 887 16µs
getHotspots / large +0.0% 16,748 16,748 146µs
getHotspots / small +0.0% 886 886 14µs
listEntrypoints / large +0.0% 23,798 23,798 116µs
listEntrypoints / small +0.0% 4,795 4,795 58µs
prepareChange / large +0.0% 16,194 16,194 103µs
prepareChange / small +0.0% 2,483 2,483 27µs
searchSymbols / large +0.0% 90,246 90,246 765µs
searchSymbols / medium +0.0% 17,766 17,766 327µs
searchSymbols / small +0.0% 3,588 3,588 37µs
summarizeDiff / large +0.0% 19,939 19,939 166µs
summarizeDiff / small +0.0% 2,133 2,133 27µs
traceUsage / large +0.0% 7,728 7,728 77µs
traceUsage / small +0.0% 725 725 13µs
understand / large +0.0% 460,608 460,608 2.439739ms
understand / small +0.0% 5,555 5,555 39µs

* = new scenario, compared against static baseline

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

CKB Analysis

Risk Files +19927 -9798 Modules

🎯 82 changed → 0 affected · 🔥 11 hotspots · 📊 9 complex · 💣 3 blast · 📚 197 stale

Risk factors: Large PR with 88 files • High churn: 29725 lines changed • Touches 11 hotspot(s)

👥 Suggested: @lisa.welsch1985@gmail.com (11%), @talantyyr@gmail.com (5%), @lisa@tastehub.io (10%)

Metric Value
Impact Analysis 82 symbols → 0 affected 🟢
Doc Coverage 6.598984771573605% ⚠️
Complexity 9 violations ⚠️
Coupling 0 gaps
Blast Radius 0 modules, 0 files
Index indexed (0s) 💾
🎯 Change Impact Analysis · 🟢 LOW · 82 changed → 0 affected
Metric Value
Symbols Changed 82
Directly Affected 0
Transitively Affected 0
Modules in Blast Radius 0
Files in Blast Radius 0

Symbols changed in this PR:

Recommendations:

  • ℹ️ 82 symbols have low mapping confidence. Index may be stale.
    • Action: Run 'ckb index' to refresh the SCIP index
💣 Blast radius · 0 symbols · 3 tests · 0 consumers

Tests that may break:

  • internal/cartographer/linux_smoke_test.go
  • internal/mcp/wide_result_budget_test.go
  • internal/query/impact_outgoing_test.go
🔥 Hotspots · 11 volatile files
File Churn Score
.gitignore 2.74
CHANGELOG.md 10.47
Makefile 4.00
cmd/ckb-bench/main.go 5.02
internal/query/doctor.go 3.81
internal/query/review_split.go 3.06
third_party/cartographer/src/answer.rs 7.29
third_party/cartographer/src/api.rs 22.64
📦 Modules · 2 at risk
Module Files
🔴 third_party/cartographer 42
🔴 }/src 18
📊 Complexity · 9 violations
File Cyclomatic Cognitive
cmd/ckb-bench/main.go ⚠️ 31 ⚠️ 90
internal/mcp/tool_impls.go ⚠️ 51 ⚠️ 135
internal/mcp/tool_impls_v65.go ⚠️ 18 ⚠️ 29
internal/mcp/wide_result_budget_test.go ⚠️ 46 ⚠️ 95
internal/query/doctor.go ⚠️ 16 ⚠️ 35
internal/query/impact_outgoing_test.go ⚠️ 18 ⚠️ 36
internal/query/review.go ⚠️ 97 ⚠️ 157
internal/query/review_coupling.go ⚠️ 51 ⚠️ 135
💡 Quick wins · 10 suggestions
📚 Stale docs · 197 broken references

Generated by CKB · Run details

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

CKB Review: 🟡 WARN — 59/100

88 files (+29725 changes) · 19 modules · go python rust
87 reviewable · 1 generated (excluded)

Changes 88 files across 19 modules (go, python, rust). 88 files across 13 independent clusters — split recommended; Risk score: 1.00 (high). Consider splitting into 13 smaller PRs.

Check Status Detail
risk 🟡 WARN Risk score: 1.00 (high)
coupling 🟡 WARN 5 commonly co-changed file(s) missing from changeset
split 🟡 WARN 88 files across 13 independent clusters — split recommended
generated ℹ️ INFO 1 generated files detected and excluded
test-gaps ℹ️ INFO 64 untested function(s) in changed files (showing top 10)
blast-radius ℹ️ INFO No symbols with callers in changes
hotspots ℹ️ INFO 11 hotspot file(s) touched (top 10 shown)
breaking ✅ PASS No breaking API changes
format-consistency ✅ PASS No format consistency issues
comment-drift ✅ PASS No comment/code drift detected
tests ✅ PASS 3 test(s) cover the changes
dead-code ✅ PASS No dead code in changed files
secrets ✅ PASS No secrets detected
complexity ✅ PASS +12 cyclomatic complexity across 4 file(s)
health ✅ PASS 1 new file(s), 21 unchanged
bug-patterns ✅ PASS 6 new bug pattern(s) (27 pre-existing filtered) (all on unchanged lines)
unwired ✅ PASS All exported symbols are reachable from entrypoints
arch-health ⚪ SKIP Cartographer not compiled in this build
layers ⚪ SKIP Cartographer not compiled in this build

Top Risks

  • Risk score: 1.00 (high)
  • 5 commonly co-changed file(s) missing from changeset
Findings (6 actionable, 24 informational)
Severity File Finding
🟡 cmd/ckb-bench/main.go Missing co-change: internal/mcp/tool_impls_v86.go (100% co-change rate)
🟡 cmd/ckb-bench/main.go Missing co-change: internal/query/review_layers.go (100% co-change rate)
🟡 cmd/ckb-bench/main.go Missing co-change: internal/query/compound_refactor.go (100% co-change rate)
🟡 internal/cartographer/bridge.go Missing co-change: internal/mcp/tool_impls_v86.go (80% co-change rate)
🟡 internal/cartographer/bridge_stub.go Missing co-change: internal/mcp/tool_impls_v86.go (80% co-change rate)
ℹ️ internal/query/doctor.go Complexity 150→155 (+5 cyclomatic) in checkLsp()
✂️ Suggested PR Split (13 clusters)
Cluster Files Changes Independent
third_party/cartographer 42 +13015 −8150
}/src 18 +6386 −1445
internal/query 17 +373 −79
}/scripts 2 +15 −15
npm/package.json 1 +1 −1
testdata/review 1 +2 −2
scripts/sync-cartographer.sh 1 +24 −25
}/Cargo.toml 1 +18 −6
}/build.rs 1 +1 −1
}/cbindgen.toml 1 +1 −1

... and 3 more clusters

Code Health

New files: 1 (avg health: 90)

Estimated review: not feasible as a single PR (88 files, 29725 lines, 13 clusters)

Reviewers: lisa.welsch1985 (11%) · talantyyr (5%) · lisa (10%)

The analyzeImpact LIP staleness check read the annotation key
`lip:nyx-agent-lock` and reported "an active nyx.code agent". Debrand:
key → `lip:agent-lock`, message → "an active agent". The LIP writer
side is being updated in lockstep to the same generic key.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@SimplyLiz
SimplyLiz merged commit 8af06ae into main Jul 13, 2026
19 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant