codedb over codebase-memory-mcp & gitnexus

[mochadwi]

Could you please help maybe share me the what codedb can and not the others? this benchmark/comparison helps determine our decisions to pick things up on our company atm!

Really appreciate it mate!

Ref:

• https://github.com/DeusData/codebase-memory-mcp
• https://github.com/abhigyanpatwari/GitNexus

[justrach]

Section	Details
Request	Compare codedb with codebase-memory-mcp and GitNexus for internal tool evaluation.
Resolution	Fulfilled in the documented benchmark scope for release v0.2.5831.
Evidence	README: codedb/README.md Lines 341 to 368 in c71bad7 ### Normalized 1:1 engine comparison Issue [#679](https://github.com/justrach/codedb/issues/679) was explored on `justrach/smolify@835070d` using `toFtsMatch`. Each engine ran in a separate, identical Linux x86_64 sandbox with 2 vCPU and 2 GB RAM. Cold indexing used a clean cache. Each warm task used one warmup and 10 separate CLI processes; cells below are **p50 / p95 wall time · output bytes / tokens · exact anchor recall**. | Same outcome | **codedb 0.2.5831** | [codebase-memory-mcp 0.9.0](https://github.com/DeusData/codebase-memory-mcp) | [GitNexus 1.6.9](https://github.com/abhigyanpatwari/GitNexus) | |---|---:|---:|---:| | Exact definition | **0.711 / 0.741 ms · 250 B / 74 tok · 2/2** | 11.815 / 15.351 ms · 1,273 B / 515 tok · 2/2 | 1,729.866 / 1,770.555 ms · 889 B / 274 tok · 2/2 | | Direct production caller | **0.480 / 0.522 ms** · 951 B / 299 tok · 1/1 | 6.730 / 9.252 ms · **177 B / 44 tok** · 1/1 | 1,712.238 / 1,741.577 ms · 889 B / 274 tok · **0/1** | | Definition + caller + callee | **0.494 / 0.535 ms · 594 B / 182 tok · 3/3** | 8.156 / 11.335 ms · 1,664 B / 627 tok · 3/3 | 1,720.325 / 1,755.486 ms · 1,170 B / 355 tok · **2/3** | | Cold metric | **codedb 0.2.5831** | [codebase-memory-mcp 0.9.0](https://github.com/DeusData/codebase-memory-mcp) | [GitNexus 1.6.9](https://github.com/abhigyanpatwari/GitNexus) | |---|---:|---:|---:| | Index wall time | **186.293 ms** | 422.518 ms | 8,305.209 ms | | Peak child RSS | **41,360 KiB** | 85,304 KiB | 709,348 KiB | | Persistent index | 5,027,199 B | **4,759,552 B** | 28,815,656 B | This is a normalized outcome comparison, not a claim that the internal operations are identical. Each tool received its closest documented one-call operation. GitNexus returned the definition and callee but duplicated test-file incoming edges and missed the production caller; codebase-memory was explicitly asked to exclude tests, while codedb callers intentionally includes test and import sites. One TypeScript fixture and one symbol do not establish total product quality. See the [full protocol and limitations](docs/performance-0.2.5831.md#normalized-11-sandbox-comparison). \nFull report: codedb/docs/performance-0.2.5831.md Lines 191 to 275 in c71bad7 ## Normalized 1:1 sandbox comparison A separate trial explored [`codedb` issue #679](https://github.com/justrach/codedb/issues/679) without installing the competing engines into the workstation. It used the [`sandbox-gateway` protocol](https://github.com/justrach/sandbox-gateway/blob/main/docs.md) with `justrach/smolify` at commit `835070d0826ab581f7efb564a1457b72c2bbe3da`. The extracted fixture has 96 files and 1,541,092 bytes; its tar SHA-256 is `a2e3ba722bb8ab19d212ebca20a4d0f10b970ca6287a992da6e9201759cb9afb`. Each tool ran in a separate `node`-template Linux x86_64 sandbox with 2 vCPU and 2,048 MB on the same gateway node. The protocol normalizes outcomes rather than pretending the engines expose the same command. Each engine receives its closest documented one-call operation: 1. Find the exact `toFtsMatch` definition (`toFtsMatch` and `lib/docs/search.ts` are required anchors). 2. Find its direct non-test caller (`searchActiveDocs`). 3. Return the definition, direct production caller, and direct callee in one call (`toFtsMatch`, `searchActiveDocs`, and `terms`). | Outcome | codedb operation | codebase-memory operation | GitNexus operation | |---|---|---|---| | Definition | `symbol toFtsMatch` | `search_graph --name-pattern ^toFtsMatch$` | `context toFtsMatch` | | Direct caller | `callers toFtsMatch` | `trace_path --direction inbound --depth 1 --include-tests false` | `context toFtsMatch` | | One-call neighborhood | `context "toFtsMatch"` | `get_code_snippet --include-neighbors true` | `context toFtsMatch --content` | Ground truth was checked directly in the fixture: the definition is `lib/docs/search.ts:76`, and the production calls are at lines 33 and 38. Cold indexing is one clean-cache run. Warm tasks use one warmup followed by 10 separate CLI processes. Time is process wall time, peak RSS is `getrusage(RUSAGE_CHILDREN)`, and output tokens use tiktoken 0.13.0 with `o200k_base`. | Same outcome | **codedb 0.2.5831** | [codebase-memory-mcp 0.9.0](https://github.com/DeusData/codebase-memory-mcp) | [GitNexus 1.6.9](https://github.com/abhigyanpatwari/GitNexus) | |---|---:|---:|---:| | Exact definition, p50 / p95 | **0.711 / 0.741 ms** | 11.815 / 15.351 ms | 1,729.866 / 1,770.555 ms | | Definition output / recall | **250 B / 74 tokens · 2/2** | 1,273 B / 515 tokens · 2/2 | 889 B / 274 tokens · 2/2 | | Production caller, p50 / p95 | **0.480 / 0.522 ms** | 6.730 / 9.252 ms | 1,712.238 / 1,741.577 ms | | Caller output / recall | 951 B / 299 tokens · 1/1 | **177 B / 44 tokens** · 1/1 | 889 B / 274 tokens · **0/1** | | One-call neighborhood, p50 / p95 | **0.494 / 0.535 ms** | 8.156 / 11.335 ms | 1,720.325 / 1,755.486 ms | | Neighborhood output / recall | **594 B / 182 tokens · 3/3** | 1,664 B / 627 tokens · 3/3 | 1,170 B / 355 tokens · **2/3** | | Cold metric | **codedb 0.2.5831** | [codebase-memory-mcp 0.9.0](https://github.com/DeusData/codebase-memory-mcp) | [GitNexus 1.6.9](https://github.com/abhigyanpatwari/GitNexus) | |---|---:|---:|---:| | Index process wall | **186.293 ms** | 422.518 ms | 8,305.209 ms | | Peak child RSS | **41,360 KiB** | 85,304 KiB | 709,348 KiB | | Persistent index | 5,027,199 B | **4,759,552 B** | 28,815,656 B | | Warm peak child RSS | **codedb 0.2.5831** | codebase-memory-mcp 0.9.0 | GitNexus 1.6.9 | |---|---:|---:|---:| | Definition | **6,272 KiB** | 7,644 KiB | 370,312 KiB | | Direct caller | **6,272 KiB** | 7,900 KiB | 373,492 KiB | | One-call neighborhood | **6,272 KiB** | 8,412 KiB | 374,376 KiB | codedb was 2.27× faster than codebase-memory and 44.58× faster than GitNexus for the single cold-index run. Across the three warm outcomes its p50 process latency was 14.03–16.62× lower than codebase-memory and 2,433.62–3,570.17× lower than GitNexus. Its cold peak RSS was 51.51% below codebase-memory and 94.17% below GitNexus. codebase-memory's persistent index was 5.62% smaller than codedb's, and it produced the smallest caller response. All three engines found the exact definition. codedb and codebase-memory also returned full anchors for the caller and one-call neighborhood. GitNexus's documented `context` operation returned the definition and `terms`, but its incoming edges duplicated the test file and missed `searchActiveDocs`; a separate GitNexus query can surface that symbol, but using it would violate the one-call outcome. codedb callers intentionally includes test and import sites, whereas codebase-memory was asked to exclude tests, explaining part of their caller-output difference. The artifact pins codedb's exact source commit and binary SHA-256, codebase-memory's release asset and binary SHA-256, and GitNexus's npm integrity and lockfile SHA-256. The full per-run timings, response hashes, complete final outputs, and recall scores are retained in [`engine-comparison.json`](../bench/results/0.2.5831/engine-comparison.json). Limitations remain: this is one small TypeScript repository and one symbol; cold indexing has only one sample per tool; CLI timings include process startup and index loading; and the products expose broader, different feature sets. GitNexus's FTS extension was unavailable, but these graph-context tasks did not use FTS. No competing engine was installed locally. After the evidence was copied and validated, all three task-created sandboxes returned HTTP 204 on deletion and were absent from the follow-up listing; the preexisting sandbox
Environment	2-vCPU/2-GB Linux sandbox; pinned smolify@835070d; definition/caller/context tasks.
Metrics	Latency, bytes, tokens, RSS, recall.
Scope caveat	Focused benchmark: one small TypeScript fixture and one symbol; not a full product verdict.
Status	Closed as completed for this benchmark scope (c71bad7).