Benchmarks

What CodeLoupe's answers cost an agent compared with grep and whole-file reads, and what it costs to have the tool, measured by a script anyone can run (Development). Claims here are measurements of these cases, not guarantees. What a whole agent run costs, measured on real agents, is on Agent runs.

What an answer costs

An agent working on a Kotlin repository keeps asking a few questions: where is this declared, what does this file contain, who uses this, who calls this, what are its subtypes, what did my branch change. Without an index it answers them with rg and by reading files. CodeLoupe answers each with one call that returns the relevant piece of code. What the agent has to read, median over the questions of each kind on two public repositories (tokens are characters divided by 3.16; method and every row in docs/benchmarks.md):

Task	Tool	CodeLoupe	grep + read	grep, minimal
Read a type	symbol	326	789	322
Read a member	symbol	117	1,746	118
Outline of a file	outline	206	615	197
Who uses a type	usages	560	2,997	626
Who calls a member	calls	169	1,352	286
Subtypes of a type	hierarchy	59	464	102
Text search, 30 hits	grep	1,298	n/a	1,497
What a branch changed	changes	2,423	79,728	2,247

grep + read is what an agent without an index typically does: rg with context lines, a whole-file read, the full git diff. grep, minimal is a best case that assumes the agent never reads a line it does not need (for source lookups it is given the exact line range, for the branch git diff --stat). Against grep + read, CodeLoupe's answers are 3–41 % of the size (8 % summed over all questions) and take one call where grep needs two for source lookups. Against the best case they are about the same for source lookups, outlines and branch changes (8 % more), and smaller for usages, callers and subtypes. The default answers are compact: hierarchy lists the direct subtypes and changes the changed declarations; supertypes (supers=true, deep=true for transitive links) and the callers and tests of each changed declaration (callers=true) are one more parameter away.

Beyond navigation, and not part of the benchmark: worktrees of one repository share one index and each adds only its own edits; long commands run in the daemon so an agent's turn can end (Jobs and events); a tracker mirror answers issue reads locally; codeloupe metrics shows from Claude Code transcripts where an agent's context goes (Measuring agent runs).

Method and results against GitNexus

node tools/benchmark.mjs asks the same questions of a grep-and-read baseline, CodeLoupe and GitNexus over public repositories at pinned commits (JetBrains/Exposed 023a6a3, 319 Kotlin files without tests; CodeLoupe f370022, 457 files), and writes docs/benchmarks.md, docs/benchmarks.json and the chart below. Run of 2026-10-09 on Windows 11, i7-13700F, 64 GB, CodeLoupe 0.1.0 (commit 38eab82), gitnexus@1.6.12 from npm.

Tokens read per question, median over 15–16 questions of each kind (2 for the branch, 6 for text search):

Task	CodeLoupe	GitNexus	grep + read	grep, minimal
Read a type	326	1,962	789	322
Read a member	117	848	1,746	118
Outline of a file	206	n/a	615	197
Who uses a type	560	1,542	2,997	626
Who calls a member	169	666	1,352	286
Subtypes of a type	59	1,352	464	102
Text search, 30 hits	1,298	n/a	n/a	1,497
What a branch changed	2,423	21,740	79,728	2,247

GitNexus 1.6.12 has no outline or text-search tool, so those rows are n/a. Its context answers are JSON cards (callers, callees, process membership), not the same content as CodeLoupe's, so the comparison is of what is read, not of what is learned. Both tools answered every question they have a tool for; for 8 of GitNexus's 82 answers (names that exist more than once) it needed a second call with the file path, and both answers are counted.

Cost of having the tool

Same run:

	CodeLoupe	GitNexus
Tool definitions in the agent's context (tools/list)	14 tools, 4,223 tokens (3,429 after trimming the descriptions, Agent runs)	17 tools, 22,140 tokens
First index, Exposed / CodeLoupe	7.3 s / 2.7 s	128 s / 68 s
Peak memory while indexing, Exposed	517 MB	2,583 MB
Memory after the queries	206 MB daemon, both repositories	3,325 MB MCP server after 82 queries (114 MB at start)
CPU while idle, 30 s	0 ms	15 ms
Warm call (median, per question kind)	10–11 ms; 26 ms text search; 750 ms branch changes	155–213 ms; 214 ms branch changes
Index on disk, Exposed / CodeLoupe	62 MB for both	1,067 MB / 141 MB
Files written into your checkout by indexing	0	8 (AGENTS.md, CLAUDE.md, .claude/skills/; off with --skip-agents-md and --skip-skills)

What the numbers do not show

• They count what an agent reads, not whether it finishes a task; the controlled runs of real agents are in Agent runs, and they are less flattering.
• One run on a shared workstation: token counts are deterministic, timings and memory are indicative and vary between runs.
• Two repositories, one of them CodeLoupe's own, questions chosen by a fixed rule that favours grep (names unique in the repository). GitNexus's default database buffer pool (428 MiB here) was too small to index Exposed; the benchmark sets 2 GiB for analyze, as the error message suggests. GitNexus's worktree handling and newer builds than 1.6.12 were not measured. An IDE-based MCP server cannot be started by a script and is not measured.

Speed of the tool itself

Three changes to CodeLoupe's own speed, measured with the old and the new build alternating on the same machine (medians):

Change	Measured	Before	After
AOT cache (a class-data archive) for the CLI and the daemon	CLI find, ms (local, 10-12 calls)	302-322 without archive, 182-213 dynamic archive	135-142
	same, CI medians Linux / Windows / macOS arm64 / macOS Intel, vs the dynamic archive	136 / 346 / 190 / 469	112 / 276 / 144 / 373 (-18 to -24 %)
	daemon start to listening, ms	1,031	559
Parallel workspace scan, 76 workspaces	four routes at once, s, three series	2.52 / 1.30 / 1.23	0.68 / 1.09 / 0.64
Overlay refresh after an edit	edit with 8 other overlays, ms, three series	160 / 146 / 171	67 / 73 / 75

Agent runs

The tables above count what an answer costs. They do not say what a whole agent run costs, or whether an agent that has the tool uses it. That is measured separately, with controlled experiments on real agents, on the page Agent runs. The honest summary: the tool's own costs came down and, once the agents' prompts say so, the agents call it, but most of the agent-cost saving measured so far comes from shorter prompts, fewer tool definitions and fewer turns, not from CodeLoupe's answers.