Independent benchmark of GitNexus 1.6.12 with Claude Code and Codex agents: results and data (mostly n=1)

[100yenadmin]

We tested several claims from the README, setup output, skills and eval/workflow_bench against hidden tests and ground truth that we built without GitNexus. We are sharing the numbers as data for the maintainers. Most cells are one run, so small differences are noise. Related: #601 asked for benchmark data. The workflow_bench oracle problems we hit are in a separate issue: #3492.

Issues filed from this bench: #3486 (rename), #3487 (route_map), #3488 (api_impact), #3489 and #3490 (taint), #3491 (explain line numbers), #3492 (workflow_bench).

Setup

• GitNexus: 1.6.12 (c4ecf398), with embeddings enabled. Some tracks used a --pdg index.
• Agents:
  • Opus 5.5 (claude-opus-5-5) through Claude Code 2.1.286, medium effort.
  • Codex Sol 6.1 (gpt-6.1-sol) through codex exec, medium effort (high for review).
  • Haiku 4.5 (claude-haiku-4-5-20251001) through Claude Code, medium effort.
• Setups per model:
  • X0, no GitNexus: no index, every GitNexus entry point (MCP, CLI, npx, .gitnexus/run.cjs) stubbed to "not found", and a one-line note that GitNexus is not available.
  • X1, read-only MCP: for Claude, also our own GitNexus skills and instructions; for Codex, MCP only.
  • X2, full upstream install: read-write MCP including rename, the repo skills and the AGENTS.md/CLAUDE.md block. Claude setups also got the PreToolUse/PostToolUse hooks (Codex has no equivalent).
  • X3: X2 plus the gitnexus-work direct-mode prompt (Opus only).
  • Where the tasks were upstream's own (tracks 2 and 7), we used upstream's arms and prompts verbatim, with no instructions of ours.
  • Disclosure: in tracks 1 and 3-6, every Claude setup, X0 included, also carried our own user-level instruction file. It is general-purpose, but one section tells agents to prefer GitNexus over grep. In X0 the "not available" note overrides it.
• Isolation: each run got a fresh copy of the task base, its own GitNexus home, temp directory and npm prefix, and a sandbox. Mined task bases were history-sanitized, so the fix commit was absent.
• Grading: all mechanical, and no model grades another model.
  • Change tasks: the hidden tests plus a regression command, re-applied from the saved patch on a fresh copy. A regression failure counts only if the failing tests fail again when rerun on their own.
  • Probes: ground truth from the TypeScript compiler, its rename engine and its AST, or from seeded flows.
• Cost: notional, at API list prices. Claude costs are the CLI's total_cost_usd; Codex costs are computed from token counts. Compare costs only within a batch.
• Size: 340 graded runs, about $243 notional including smoke tests and discarded runs.

Results by track

1. Real bug fixes. 12 merged fixes with tests, mined from two public Python repos: 7 from electricsheephq/lcm-x and 5 from electricsheephq/WorldOS. 9 setups, n=1 per cell. "Resolved" means the hidden tests pass and the regression suite still passes.

Setup	Resolved	Cost vs X0	GitNexus calls / run
Opus X0 / X1 / X2 / X3	8 / 7 / 9 / 8 of 12	1.00 / 1.07 / 1.11 / 1.41x	– / 0.0 / 1.1 / 4.7
Sol X0 / X1 / X2	11 / 10 / 10 of 12	1.00 / 1.10 / 1.20x	– / 3.0 / 7.8
Haiku X0 / X2	7 / 5 of 12	1.00 / 0.95x	– / 1.1

2. Upstream's 4 workflow_bench tasks. Opus, with arms baseline_nomcp, baseline (MCP) and workflow_direct. 2 trials: a plain index, then a --pdg index.

• Every arm resolved the same 2 of 4 tasks in both trials.
• Mean cost per run was $1.09 / $1.22 / $1.78.
• workflow_direct cost the most on all 8 task-trials.
• On the cross-module task it averaged $2.97, against $1.66 for MCP and $1.85 for no GitNexus.
• The README reports direct mode 47% cheaper and 56% faster on that cell, at n=1. See the separate issue for the oracle caveats.

3. Impact. 10 symbols whose signature we changed. The TypeScript compiler's list of breaking sites is the ground truth, n=1 per symbol.

• Opus and Sol had recall and precision 1.00 in every setup.
• Haiku had recall 0.99 and precision 0.92 without GitNexus, and 0.87 / 0.82 with the full install. One of its full-install answers was saved outside its run directory and scored as missing; counting it, recall would be 0.97.
• Sol cost $0.12 per question without GitNexus and $0.18 with the full install.

4. Rename. 6 targets, 3 of them heavily overloaded names (close, resolve, declare). TypeScript's rename engine is the oracle.

• Agents passed 41 of 42 runs. The one failure was Haiku with the full install.
• Only Haiku's runs executed the rename tool. We scored the tool's own edit lists from those 6 runs, and applied them alone to fresh copies. They pass the oracle for 2 of 6 targets:
  • it found 138 of 146 rename locations;
  • it renamed two other symbols;
  • it rewrote 38 words inside comments;
  • it labelled all 185 of its edits "graph".
• Headless Codex refuses the write tool: "MCP tool call requires approval, but approval policy is never".

5. Structural tools, called directly. Ground truth comes from the TypeScript AST and language service.

• tool_map found 17 of 17 tools.
• route_map found 21 of 27 routes. It missed helper-registered streams, a regex catch-all, GET /api/info and POST /api/query, labelled ALL /api/mcp as GET, and added 32 entries from tests and other servers.
• api_impact found 0 of 9 consumers.
• Opus and Sol answered all three questions exactly in every setup. Haiku with the full install repeated route_map's misses.

6. Taint (--pdg). 3 seeded flows (command injection, path traversal, code injection) and 2 decoys, in Express routes.

• With the sources in inline route callbacks, explain found 0 of 3 flows.
• After a rewrite to named handlers it found 3 of 4, missed a factory-returned handler, and flagged a regex-guarded decoy.
• Agents (Opus and Sol, X0 and X2) found 3 of 3 flows and 0 decoys in every setup, with n=2 per setup.

7. Review. Upstream's 6 labelled review cases, scored with upstream's rules, 2 trials.

Setup	Labelled issues found	Cost per review
Opus, plain review	2 of 42	$0.81
Opus, gitnexus-review skill	2 of 42	$1.05
Sol, high effort, no MCP	2 of 42	$0.41
Sol, high effort, read-only MCP	4 of 42	$0.48

No setup found a labelled blocking issue. All of Sol's hits were on one PR, so the 2-vs-4 gap is within noise.

8. Usage. How often each setup called GitNexus at all:

• Codex: 100% of runs with read-only MCP and 97% with the full install.
• Opus: 21% with read-only MCP, 49% with the full install, and 93% with direct mode.
• Hook: in the full install, the search hook added context to 81 of 1338 intercepted searches, and returned the "use the MCP query tool" hint 64 times. While the MCP server holds the database it stands down and gives that hint, which is the behaviour fix(hook): emit MCP query hint when server owns DB lock (#2396) #2397 added for PreToolUse hook silently skips when MCP server owns DB lock — defeats the purpose #2396. In an earlier hooks-only test without MCP, it added context to 25% of calls.

Limits

• Sample size: most cells are n=1. A difference of one or two tasks out of 12 is within run-to-run noise.
• Noise calibration: on the 12 real fixes, Opus with read-only MCP made 0 GitNexus calls, yet its result differed from plain Opus on 3 of 12 tasks (1 better, 2 worse). That is our measured noise floor at n=1.
• Detection: this bench can detect large effects, such as a 47% cost change. It cannot detect an effect of about one task in twelve.
• Wall time: measured on a loaded machine, so it is indicative only.
• Track 2 regression check: tsc --noEmit only.
• Harness: ours runs on macOS with APFS clones, not upstream's Bubblewrap runner.

What was not tested

• Repos much larger than these (the GitNexus repo itself, lcm-x and WorldOS) and monorepos.
• Languages other than TypeScript and Python.
• Work spread over several sessions, and the full plan→work workflow.
• Models other than the three above.
• Humans browsing the graph or the web UI.
• Non-agent consumers such as CI impact reports.

What worked

• tool_map was exact: 17 of 17 tools.
• Hybrid query (BM25 plus embeddings) worked as a code-search tool. In an earlier, separate A/B (163 agent runs on 16 repos, with 70 answers blind-graded by a different model, so not mechanically graded like this bench), answers to find-and-understand questions were as accurate as ripgrep plus reading. Runs that used GitNexus cost 20-60% more.
• Multi-hop impact questions were the one consistent gain in that A/B: Opus was 14% to 43% cheaper on 2 of 3 questions, at n=1 each.
• Contamination guard: isolating the no-GitNexus arm needed blocking the CLI as well as MCP. The CLAUDE.md/AGENTS.md block sends agents to npx gitnexus or .gitnexus/run.cjs when MCP is missing. With every entry point stubbed, our audit found 0 contaminated scored runs.

Offer

We are happy to share the harness, the task definitions, the graders and ground truth, and the per-run patches and transcripts. We can also rerun a slice on a 1.6.13 release, or with --runs 3, if that is useful. Corrections to our method are welcome.

[100yenadmin]

Follow-up: a large Python repo (NousResearch/hermes-agent), impact questions

Our first round did not include a repo of this size, so we repeated the impact probes on hermes-agent.

• Repo: 2,383 non-test Python files.
• Index: GitNexus 1.6.12, with 330,628 nodes, 831,653 edges and 267,728 voyage-code-4 embeddings. It was built at the same commit the agents worked on.

Questions. We asked 8 questions of the form "we are about to add a required parameter to X — list every place that must change". Each question gave the exact target name, declaration file and line. We chose targets that are awkward for text search:

• format_bytes, which is imported under three aliases;
• _norm, _write, _fail and _section, each a name that 12–26 functions in the repo share;
• two methods called on instances across modules (CredentialPool.has_credentials, ProviderRegistry.export);
• normalize_provider, which has a same-named function in another module.

We excluded targets where more than 20% of the sites had to be judged by hand.

Ground truth. The answer key is the set of lines that break under pyright 1.1.414, run on 3 platforms, plus 2 runtime-only sites we found by reading the code: 147 sites in all. An independent check with jedi plus code reading agreed with all 147. One site needed manual tracing.

impact on its own

We ran impact once per target, upstream, depth 1. It returns symbols, not lines, so we scored it by caller coverage:

• A ground-truth site counts as covered when a returned symbol encloses it. Module-level sites are matched to returned File symbols. Several sites in one caller all count as covered.
• Precision counts the returned symbols that enclose at least one ground-truth site.

target	sites	symbols returned	site recall via caller coverage	symbol precision
format_bytes	24	16	1.00	1.00
_norm	27	21	1.00	1.00
_write	13	11	1.00	1.00
_fail	20	7	0.50	1.00
_section	9	9	1.00	1.00
CredentialPool.has_credentials	27	25	0.93	1.00
ProviderRegistry.export	7	7	1.00	1.00
normalize_provider	20	16	0.90	1.00

Overall, the returned symbols covered 133 of 147 sites, and every returned symbol enclosed a ground-truth site. The uncovered sites:

• 6 _fail calls made through a function that returns a module (_pc()._fail(...)) (Python: calls on a module returned by a function (_pc().target()) get no CALLS edge (1.6.12) #3497);
• 4 _fail calls in hermes_cli/plugin_packs.py, each importing _fail inside the calling function; they resolve to an unrelated _fail nested inside another function in that file, overriding the import (Python: a nested function captures same-named calls elsewhere in its file, overriding their imports (1.6.12) #3499);
• two CredentialPool.has_credentials sites (Python: annotated factory results lose receiver types through conditional expressions and module-qualified calls (1.6.12) #3498). One loses its receiver type through x if c else None; the other is a factory call through a module object, and that module comes from a module-returning helper plus a re-export;
• two runtime-only normalize_provider sites, which pyright misses as well.

Agents, 2 trials per question

Agents answered with file and line. A listed line matched a site if it was within ±2 lines of it. Recall and precision below are means over 16 runs per arm.

arm	recall	precision	mean cost	GitNexus calls/run
Opus 5.5, no GitNexus (ripgrep + reads)	1.000	0.998	$0.50	0
Opus 5.5, prompt says "use GitNexus first"	1.000	0.998	$0.55	0.2
Opus 5.5, prompt requires calling impact	0.997	1.000	$0.60	2.2
GPT-6.1 Sol, no GitNexus	1.000	1.000	$0.17	0
GPT-6.1 Sol, prompt says "use GitNexus first"	1.000	1.000	$0.22	3.1

• Accuracy and cost. Agent scores were near-perfect in every arm. Mean cost per run rose 10% for Opus's optional-GitNexus arm, 20% for its required arm and 30% for Sol's GitNexus arm. Sol's mean wall time rose from 39 s to 52 s, about 35%.
• Opus. Opus rarely called the GitNexus tools when only asked to (0.2 calls per run), even though they were connected and listed. When the prompt required them, it called them about twice per run. The one site any arm missed came from that required arm, and it is one the graph also misses. That is one run, so treat it as anecdotal; the overlap does not show that the tool result caused the miss.
• Sol. Sol followed the prompt, calling list_repos, context and impact about 3 times per run.
• Caveat. Every question supplied the exact target name, file and line. These results say nothing about finding a symbol whose name you don't know, or about mostly untyped data flow.

A fix-task track on the same repo is still running: 8 real upstream bug fixes with hidden tests, using the same arms. We will post it here when it is graded.

[100yenadmin]

@abhigyanpatwari @magyargergo @azizur100389, tagging you as the people listed in CODEOWNERS so this reaches you directly.

We have summarised the core result of this Discussion as an issue: #3500. In short:

• In our tests, GitNexus 1.6.12 showed no measurable accuracy gain for Opus 5.5, Sol 6.1 or Haiku 4.5.
• It usually made Opus and Sol more expensive. Opus was cheaper on a small multi-hop sample.
• Haiku was often cheaper with the full install, but less accurate.
• On a large Python repo, impact covered 133 of 147 sites and reported exact for all 8 targets, including the 3 with misses.

The issue lists the claims we tested, what we saw for each, and three concrete asks. The first is results from the repo's own eval/ SWE-bench harness, which we would gladly rerun. The tool-level reports are linked from the issue. We know that is a lot of reports in one day: the issue is the one-page summary, and there is no rush.

One correction to our follow-up above: we stopped the hermes-agent fix-task track before the Opus setups finished, so we will post only the Sol results here.

[100yenadmin]

Follow-up: hermes-agent fix tasks (Sol only)

These are the fix-task results promised above. As noted in the previous comment, we stopped the Opus setups early, so this covers Codex Sol 6.1 (medium effort) only.

Tasks. 8 merged upstream fixes from NousResearch/hermes-agent: 2 small fixes, 3 that need some investigation, and 3 that touch several modules. The source PRs are NousResearch/hermes-agent #133157, #133012, #132997, #133229, #133231, #133004, #132773 and #133289.

Setup.

• Each task starts at the fix commit's first parent, with later Git history removed. The fix is absent from the task checkout, and the fix's tests are used only for grading.
• The agent gets a bug-report-style description of the problem, with any links to the fix removed.
• The GitNexus setup used read-only MCP. Its prompt told the agent to run context on the symbols involved, and upstream impact on every symbol it changes, before editing.
• The index was built at d62fbea7, the repo's nightly index. Task bases are 14–301 first-parent commits after it, so the index contains none of the fixes. It was not rebuilt for each task, which is how an index looks in daily use. These results do not show how an index rebuilt at each task base would do.
• Grading: each saved patch is applied to a fresh copy of the task base. It must pass the fix's hidden tests, then the regression files for the modules the fix touches. A regression failure counts only if the failing tests fail again in two isolated reruns.

Setup	Hidden tests pass	No regressions as well	Mean notional cost	Mean agent wall time	GitNexus calls / run
Sol, no GitNexus	16/16	15/16	$0.32	127 s	0
Sol, prompt says "use GitNexus first"	16/16	14/16	$0.39	144 s	9.4

2 trials per task per setup. Costs are estimated from token counts. Wall times exclude indexing and grading, and are indicative only, because the machine was loaded.

• Both setups passed the hidden tests in all 16 runs.
• All three runs that failed regression were on the same task (#133004), and each failed the same two repo-convention tests on both reruns. The agent added a bare which call that is not in the repo's resolution allowlist, and left a stale allowlist row behind. Two trials per task cannot show a difference in regression rate.
• Averaged over the two trials, the GitNexus setup cost more on all 8 tasks. Over all runs, mean cost was 24% higher and mean agent wall time 13% higher.

For this Sol setup on these eight tasks, GitNexus raised mean cost without improving hidden-test pass rates. This is the last benchmark result we plan to post. Issue #3500 has the summary and our asks.

[magyargergo]

Thanks for putting this together and for being explicit about the sample sizes and limitations. The concrete repros are useful.

We have been working on stabilizing GitNexus since 1.6.12 (c4ecf398). I checked this against main at 047354c3. There are changes since that release which could affect the numbers, with evidence attached to the merged work:

• Index integrity and native search: fix(index): guard graph integrity and fail closed on incomplete risk #3442 upgraded LadybugDB to 0.21.1 and added checks before publishing an index. In the documented 8,192-row native fixture, incorrect tuples after incremental COPY went from 3,936 on 0.18.3 to 0 on 0.21.1. test(search): guard native lookup and Unicode FTS after COPY #3477 adds regression coverage for missing name/UID lookups and Unicode FTS failures. Those are controlled reproductions, rather than a replay of your benchmark databases.
• Search recovery and diagnostics: fix(augment): recover symbols missed by FTS file ranking #3422 recovers exact-name symbols when BM25 ranks caller/test files above the definition; its regression failed before the fix and the 17-test augmentation suite passed afterwards. fix(search): report partially missing FTS indexes in query results #3448 exposes partially missing FTS indexes, and fix(search): reject incomplete FTS repairs and verify worktree lookup #3475 stops an incomplete repair being reported as successful.
• Query response size: fix(query): send hub content once across process_symbols rows #3356 sends a shared symbol's source content once, instead of repeating it for every execution flow, while preserving the flow memberships. This reduces those tool payloads; we have not measured the effect on your agents' total token cost.
• Interrupted embedding runs: fix(embeddings): defer staged checkpoint count until publication #3461 and fix(embeddings): reuse completed vectors after interrupted analyze #3463 keep published counts consistent and recover completed staged vectors. The controlled crash/retry fixture reused 5,004 durable chunks and submitted only 128 interrupted nodes again. As you noted, this addresses indexing/retry cost, which is separate from agent task cost.

I would expect some measurements to move in either direction as this work lands: fewer missing results and duplicate payloads, but also extra validation work and more explicit reports of uncertainty. We need a rerun to quantify that. These fixes do not establish higher solve rates or lower end-to-end agent cost on the workloads here.

We also checked the new reports against main. The rename, Python resolution, route/API extraction and inline-callback taint gaps still have reproducible causes. In particular, the truncated drop summary can still produce exact; that needs fixing. The stabilization work above does not make those findings obsolete.

Please share the harness, task definitions and per-run patches/transcripts. For a follow-up, I'd pin both revisions, repair the stale workflow_bench oracles in #3492, and use repeated paired runs with a baseline with no GitNexus access. For the latest Hermes fix-task results, I'd keep a rebuilt-at-task-base index and the disclosed 14–301-commit-old nightly index as separate conditions, so we can measure freshness as well as tool value.

[100yenadmin]

Thank you for checking these against main and for the detailed list of stabilization work. Keeping the index-integrity fixes separate from agent-level outcomes is helpful. It is also useful to know that the truncated drop summary can still produce exact on 047354c3.

An update on our side: since filing #3500, when GitNexus was optional in our setup, we have turned it off for now and stopped benchmarking it ourselves. That pauses the rerun offers in #3500 and in our opening post. If we pick this up again, for example once 1.6.13 is stable, we would use the design you describe:

• both revisions pinned;
• the eval/workflow_bench: 3 of 4 hidden oracles fail against v1.6.12 for non-task reasons, 2 accept only one design, and the baseline arm keeps MCP #3492 oracles repaired;
• repeated paired runs against a baseline with no GitNexus access;
• for Hermes, an index rebuilt at each task base and the older nightly index run as separate conditions.

One suggestion, given how fast the stabilization work is landing: build that comparison into your own release testing. That would check tool accuracy on every RC and measure agent outcomes before each stable release, so you would see whether GitNexus helps agents, not only whether the graph is correct. RCs are published automatically for eligible pushes to main, and agent runs cost money, so we would split it by cost:

• Every RC: tool-accuracy cases. Turn the repros in rename: "graph"-confidence edits are whole-file word replacements; vs TypeScript's rename they miss references, rename other symbols and rewrite comments (1.6.12) #3486–explain: cross-function (interprocedural: true) findings show 0-based line numbers, one less than the intra-procedural findings in the same response #3491 and Python: calls on a module returned by a function (_pc().target()) get no CALLS edge (1.6.12) #3497–Python: a nested function captures same-named calls elsewhere in its file, overriding their imports (1.6.12) #3499 into fixed-answer tests. They cover rename edit lists, route and API extraction, taint and explain findings, Python call resolution, and exact vs lower-bound when the drop summary is truncated. They are cheap, run like any other CI test, and show when a fixed case regresses.
• Before each stable release, and on a regular schedule against the latest RC: paired agent runs. Run agents with and without GitNexus on a pinned task set with hidden tests. workflow_bench already has a baseline_nomcp arm (off by default). The skill-evolution workflow is configured to run workflow_bench with repeated runs per arm per task (3 by default), so this could reuse that runner.
• Publish the numbers with each release. Solve rate, cost and wall time against the no-GitNexus arm would show users the trend and give the README's claims a measurement.

Our task set could seed it. We will share the harness (with local paths removed), the task definitions, graders, answer keys and per-run patches. The tasks are built on public repositories. The exception is the earlier A/B: 3 of its 16 repositories are private, so we will leave those tasks out. Our opening post and #3500 also offered per-run transcripts. On checking them, the raw transcripts include our internal agent instruction files and local paths, so instead we will share per-run summaries (tool calls, token counts, outcome). We will link the bundle here when it is up.

One correction to our disclosures, found while preparing the bundle. We said the Claude setups in tracks 1 and 3–6 carried our own instruction file. Two other sets of setups also carried our own configuration:

• Codex setups: they ran with our global Codex configuration (instruction file, skills and memories). In the claims bench and the A/B, that instruction file told agents to use GitNexus first. In the Hermes tracks it described GitNexus as optional. In the claims bench, the Codex setup without GitNexus got the same "GitNexus is not available" note as the Claude one.
• Claude setups in the Hermes impact track: they carried the same instruction file.

The bundle README will list what each setup carried.

[100yenadmin]

The data bundle is up: https://github.com/100yenadmin/gitnexus-agent-bench-1.6.12

It contains:

• the harness, with local paths moved into a config file;
• task definitions, hidden tests and answer keys;
• per-run patches and structured answers;
• a summary of every run (tool calls, GitNexus calls, tokens, cost, wall time and grade), in place of the transcripts.

The README maps each claim in #3493 and #3500 to the files behind it. tracks/verify_tracks.py recomputes the published claims-bench numbers from the bundle alone. The three private A/B repositories are left out (26 runs).

While building it we found two more setups that carried our own configuration, beyond the correction above:

• The A/B's Claude real* and aug* setups also carried our instruction file. The setups behind the multi-hop result (rg, gnx, gnxf) did not.
• The Opus Hermes-impact setups with GitNexus (A1f, A1m) also had our own local versions of the gitnexus-* skills.

Two small corrections to figures we posted, both explained in the README under "Known differences":

• Counted one row per run, three usage figures change: 49% becomes 51%, 93% becomes 100%, and the hook totals become 75 / 1136 / 58 instead of 81 / 1338 / 64.
• In the A/B, the 16 find questions were scored against a hand-picked target symbol. Only the callers and impact questions had fully hand-checked answer keys.