Code analysis and search, swift and sharp.
Vorpal indexes a codebase into a knowledge graph and answers questions about it: who calls this, what implements that, where is the code that does X. It is one binary with 49 tree-sitter grammars compiled in, a structural search and rewrite engine built on ast-grep, hybrid semantic search, and an MCP server so coding agents can use all of it.
Point it at a repository:
$ vorpal index .
parsed 355 files (0 replayed from cache) → 9795 nodes; refs: 10619 resolved, 1508 ambiguous, 7748 external, 8826 masked
index: ./.vorpal/index
$ vorpal graph callers resolve_import_path
resolve [Method] ./crates/resolve/src/resolver.rs
$ vorpal graph implementors FileExtractor
OutlineExtractor [Struct] ./crates/ingest/src/outline_extractor.rs
$ vorpal search "stat manifest change detection"
0.0167 FileStat [Struct] ./crates/ingest/src/manifest.rs
$ vorpal mcp # serve all of the above to agents over MCP (stdio)The output above is captured from running vorpal on its own repository.
Every release attaches one binary per platform. Download it and make it executable; there is no archive to unpack.
# macOS (Apple Silicon); other platforms in the table below
curl -L -o vorpal https://github.com/hyper-light/vorpal/releases/latest/download/vorpal-macos-arm64
chmod +x vorpal && sudo mv vorpal /usr/local/bin/
vorpal --help| Platform | Asset |
|---|---|
| macOS Apple Silicon | vorpal-macos-arm64 |
| macOS Intel | vorpal-macos-x64 |
| Linux x64 (glibc) | vorpal-linux-x64 |
| Linux ARM64 (glibc) | vorpal-linux-arm64 |
| Linux x64 (static/musl) | vorpal-linux-x64-musl |
| Linux ARM64 (static/musl) | vorpal-linux-arm64-musl |
| Windows x64 | vorpal-windows-x64.exe |
| Windows ARM64 | vorpal-windows-arm64.exe |
| Windows x86 (32-bit) | vorpal-windows-x86.exe |
npm install -g @hyper-light/vorpal-cli # installs the vorpal binary for your platformgit clone https://github.com/hyper-light/vorpal && cd vorpal
cargo build --release -p vorpal
sudo mv target/release/vorpal /usr/local/bin/ # or add to PATHMore detail (PATH setup, verifying, troubleshooting): docs/getting-started.md.
Run from your project root and the defaults line up:
$ cd my-project
$ vorpal index . # build ./.vorpal/index (incremental on re-runs)
$ vorpal search "parse http request" # hybrid semantic search
$ vorpal graph callers handle_request # who calls this?
vorpal index <dir>writes to<dir>/.vorpal/index;searchandgraphread./.vorpal/indexrelative to your shell. Index.from the project root and they match. Otherwise pass--index <dir>/.vorpal/indexto queries.
vorpal mcp is a Model Context Protocol server over stdio (revision 2026-07-28, with the
initialize handshake kept for older clients). It gives Claude, Codex, Cursor, and other
agents tools for callers, references, reachability, semantic and structural search, and
verbatim source. It builds the index if needed and keeps it current while it runs. What
it saves an agent against plain grep and read, in turns and tokens, is measured under
How does it compare?.
The short route, from your project root, writes the config for every client it finds:
vorpal mcp install # or --client claude-code|claude-desktop|codex|cursor|vscode|windsurfBy hand:
Claude Code
claude mcp add vorpal -- vorpal mcp --index /abs/path/to/project/.vorpal/indexClaude Desktop, in your MCP config:
{
"mcpServers": {
"vorpal": { "command": "vorpal", "args": ["mcp", "--index", "/abs/path/to/project/.vorpal/index"] }
}
}Codex CLI, in ~/.codex/config.toml:
[mcp_servers.vorpal]
command = "vorpal"
args = ["mcp", "--index", "/abs/path/to/project/.vorpal/index"]Cursor and other JSON clients use the same mcpServers block as Claude Desktop (Cursor reads
it from .cursor/mcp.json).
Use an absolute index path: MCP clients launch the server without a working directory. If
vorpalis not on the client'sPATH, use the binary's absolute path ascommand. Claude Code loads each MCP tool's schema in a turn of its own the first time a tool is used; the trade-offs of keeping them resident are in docs/mcp.md.
Tools exposed: index, health, schema, coverage, code_search, architecture,
compare_generations, impact, dead_code, node, graph (callers, callees, references,
importers, implementors, type_users, similar, observed), reachable, data_flow, query, structural_search,
rule_search, ast_dump, fetch_span, snippet, why, search. The whole listing is
under 12 KB on the wire (11.7 KB; a test gates it), because a client either loads each schema
in a model turn or carries the listing in every turn's input; the server's instructions
also carry the CLI one-liner for its index, so a client with a shell can answer a single
lookup in two turns with no schema load at all. Tools that return records
page with cursors and accept format: "lean" | "toon" | "ids"; graph callers and callees
rows carry the call-site line so "who calls X" and "what does X call" are one call each.
--profile scout|analysis|full limits the
tool set for read-only agents. Full descriptions and the wire contract:
docs/mcp.md.
The pattern engine and index API are also available as libraries:
pip install vorpal-py # Python → import vorpal_py
npm install @hyper-light/vorpal-node # Node.js (native)
npm install @hyper-light/vorpal-wasm # browser / portable→ Python quickstart · TypeScript/JS quickstart
| Command | What it does |
|---|---|
vorpal index [src] [--out DIR] [--verify] |
Build/refresh the knowledge-graph index |
vorpal search <query> [-k N] [--index DIR] |
Hybrid (name + semantic + graph) search |
vorpal graph <verb> [name] [--index DIR] |
callers callees refs importers implementors typeusers similar observed node reachable flows snippet schema dead coverage impact diff architecture |
vorpal query '<cypher>' [--index DIR] |
Cypher-shaped read-only graph queries (MATCH … WHERE … RETURN … LIMIT) |
vorpal run -p <pattern> [-l lang] [-r fix] |
One-off structural search/rewrite (default command) |
vorpal scan [-r rule.yml] [--format github] |
Run configured YAML rules across a project |
vorpal outline [paths] [--view signatures] |
File structure: symbols, members, imports/exports |
vorpal enable semantic-f16|semantic-f32 · disable · tune --queries FILE |
Install the neural encoder (274 MB / 547 MB); or measure every ranking tier on your own labelled queries and enable what wins |
vorpal mcp [--index DIR] |
Serve the MCP server over stdio |
vorpal test · new · lsp · grammars · completions |
Rule testing, scaffolding, LSP, grammar list, shell completions |
Every command with examples: docs/getting-started.md.
Numbers below are release builds of v0.8.3 on an Apple M5 Max (18 cores, 128 GB,
macOS 26.4.1, rustc 1.98.0), measured 2026-09-05. Times are wall-clock for the whole CLI
invocation including process start; cold times are the best of three runs. Every run
waited for a quiet machine first: two consecutive one-second top samples at least 88 %
idle, with nothing above half a core except the two macOS daemons that never fully idle
(fseventsd, WindowServer), whose load we recorded beside each result instead. Docker
Desktop stayed open throughout. Every dataset is pinned by commit. Indexing always builds
the full graph (calls, imports, types, data flow, near-clone pairs, request-to-route
links, co-change history), so each number covers the whole product, not a symbol table. Method and
history: docs/wip/BENCHMARKS.md.
vorpal index <source-tree> --out <index-dir>
Linux kernel @ 1590cf032971 (75,954 files parsed of 94,843 tracked, ~30 M LOC) |
|
|---|---|
| Cold index → 8,891,771 nodes | 10.9 s |
| Edit a function body, re-index | 0.9 s |
| Edit a comment only | 0.6 s |
| Add a function | 4.6 s |
touch one file (content unchanged) |
0.5 s |
| Nothing changed | 0.15 s |
The edit rows are medians of five saves to fs/read_write.c. A body edit or a comment
change replays only that file against the carried graph. Adding a definition changes the
file's signature and import sets, so the incremental lanes decline and the build replays
the whole reach graph; that is where the 4.6 s goes. The cold time is slower than the
8.2 s we published for v0.7.1 on 2026-09-03. Measured again on 2026-09-05 under the same
gate, within the same hour as these numbers, v0.7.1 builds this index in 10.7 s and
v0.8.2 in 11.4 s, so the difference is the machine on the day, not the code.
Fifteen other repositories, shallow-cloned at the pinned commit. "Files parsed" counts files a grammar handled, not everything tracked; the kernel row uses the same rule.
| Repo | Language | Files parsed | Nodes | Cold | Unchanged |
|---|---|---|---|---|---|
llvm/llvm-project d37814473 |
C++ | 86,124 | 1,444,028 | 7.4 s | 0.22 s |
ziglang/zig 738d2be9 |
Zig | 17,025 | 1,085,567 | 6.2 s | 0.04 s |
JetBrains/kotlin 9f27f51dd |
Kotlin | 75,448 | 795,719 | 2.4 s | 0.31 s |
kubernetes/kubernetes bce953e8 |
Go | 26,641 | 692,828 | 1.9 s | 0.10 s |
dotnet/roslyn 4cac4334 |
C# | 19,522 | 490,284 | 2.0 s | 0.06 s |
rust-lang/rust 5db7f4be8 |
Rust | 41,607 | 464,064 | 2.6 s | 0.09 s |
WordPress/WordPress c195362 |
PHP | 4,195 | 286,824 | 1.7 s | 0.02 s |
apache/spark 06539777 |
Scala | 11,512 | 253,753 | 1.5 s | 0.06 s |
apache/kafka 6e4c555 |
Java | 7,246 | 209,131 | 0.7 s | 0.03 s |
vercel/next.js 483f8420 |
TS/JS | 27,216 | 204,754 | 0.9 s | 0.22 s |
ghc/ghc 44d7788f |
Haskell | 15,837 | 178,259 | 0.6 s | 0.04 s |
python/cpython b86a41cbf63 |
Python/C | 3,841 | 162,945 | 0.9 s | 0.02 s |
rails/rails 4130768 |
Ruby | 3,952 | 49,635 | 0.3 s | 0.03 s |
neovim/neovim d423675 |
C/Lua | 1,476 | 40,507 | 0.2 s | 0.02 s |
vuejs/core d63616c |
Vue/TS | 626 | 11,191 | 0.1 s | 0.01 s |
This repository: 1,894 files parsed of 2,879 tracked → 79,817 nodes, 7.4 s cold¹, 0.02 s unchanged. The vendored tree-sitter runtime and 49 grammars are included in that count.
Disk: the kernel index is a 4.8 GB generation, most of it a parsed-product cache that makes the sub-second edits above possible; the search tiers a daemon warms on top of it bring the directory to 8.1 GB (table below). The previous generation is kept until the next commit, then swept. Indexer peak RSS on the kernel: 5.4 GB.
¹ One 33 MB generated parser.c sets the floor; everything else parses in parallel
underneath it. To re-run a pinned row, fetch by the full SHA
(git fetch --depth 1 origin <sha>); GitHub refuses abbreviated ones.
Yes. vorpal mcp watches the tree and re-indexes changed files as you save. Changes
apply incrementally, including to the semantic-search tier, so a save never re-parses the
tree. Round trips measured from the client side (medians of 30 calls, kernel index); the
save rows are medians of seven saves to fs/read_write.c on a scratch copy of the kernel,
polled every 20 ms until the daemon's answer showed the edit (range 1.9–4.9 s; each save
commits a new generation):
| Operation | Time |
|---|---|
Graph query (graph, node, …) |
0.13 ms |
| Hybrid search (default tier; per-tier table below) | 0.8 ms |
| Server start → answering queries on an existing index | immediate |
| First search after start (ranking tier warm-up, once) | 0.19 s |
| Save a file → answers include the change | 2.0 s (a body edit) · 2.2 s (a new function) |
Repositories with multi-megabyte source files get one more optimization in a long-lived
process (the MCP daemon, a watch loop, an SDK server calling indexBuild per save):
files over 1 MiB keep their parse state, so a save re-parses only the changed region and
re-walks only the edited definition. The result is checked byte-for-byte against a full
re-extraction on every row below.
| Edited file (per save) | Fresh | Incremental parse | + walk splice |
|---|---|---|---|
54 MB generated C (tree-sitter-julia parser), edit between definitions |
4.2 s | 1.9 s | 0.7 s |
| 54 MB generated C, edit inside its single 43 MB parse-table definition | 4.2 s | 1.9 s | 1.7 s |
17 MB generated C (tree-sitter-cpp parser), edit near the top |
1.35 s | 0.60 s | 0.21 s |
| 17 MB generated C, edit in the middle | 1.35 s | 0.60 s | 0.47 s |
1.4 MB hand-written C (CPython Parser/parser.c) |
107 ms | 37 ms | 17 ms |
The granularity is the enclosing definition: an edit inside one giant definition
re-walks that definition. Walk splicing currently ships for C; if any splice check fails,
the file falls back to a full walk. One-shot CLI builds are unaffected because nothing
is retained unless a file is parsed again. VORPAL_TREE_CACHE=0 disables retention,
VORPAL_WALK_REUSE=0 disables only the splice; _MIN and _BUDGET set the 1 MiB floor
and the 256 MiB budget.
vorpal search "socket buffer alloc" -k 10 --index <index-dir>
echo "semanticTier: learned" >> vorpalconfig.yml # train a ranking model on this corpus at the next index
vorpal enable semantic-f16 # install the 274 MB neural encoder (or semantic-f32, 547 MB)
vorpal tune --queries my-queries.txt # measure every tier on your queries; enable what wins
Out of the box, search fuses exact and token name matching, hashed name/signature/path embeddings, and graph in-degree. Nothing to download. Two optional tiers sit on top:
- Learned tier. A ranking model trained from your own corpus while the index warms.
No download. It improves results on every corpus we measure, mostly as recall. Select
it with
semanticTier: learnedinvorpalconfig.yml(orvorpal-index index --semantic-tier learned). - Neural encoder. CodeRankEmbed (MIT) reranks the top candidates at query time. The
f16 and f32 downloads produce the same embeddings to within rounding (cosine 1.000000
after conversion); they rank the same on CPython and this repo and differ by 0.001
NDCG@10 on the kernel, and otherwise differ only in disk. It also embeds referenced
definitions in the background, which lets it surface answers the name-based channels
never find. That fill runs on the GPU
when one is present (Metal, Vulkan, or DX12 through
wgpu; Apple, NVIDIA, AMD, Intel), otherwise on the platform BLAS or portable CPU code. Results do not depend on which built the embeddings.VORPAL_ENCODER_GPU=offforces CPU.
Graded retrieval on three corpora with the bundled labelled query sets (xtask/labels/:
54 / 54 / 55 queries across six classes from exact name to paraphrase; every grade cites a
source line in the .evidence.md files; NDCG@10 / MRR / recall@5; cargo xtask searcheval):
| Corpus (queries) | Default | + learned tier | + encoder (f32; f16 in a footnote) |
|---|---|---|---|
| Linux kernel, 8.9 M defs (54) | 0.306 / 0.310 / 0.312 | 0.315 / 0.304 / 0.361 | 0.295 / 0.290 / 0.302² |
| CPython, 163 K defs (54) | 0.306 / 0.291 / 0.333 | 0.341 / 0.322 / 0.389¹ | 0.351 / 0.331 / 0.426 |
| This repo, 79 K defs (55) | 0.400 / 0.393 / 0.400 | 0.430 / 0.427 / 0.455 | 0.455 / 0.448 / 0.500 |
Which tier to run is a per-repository decision. The encoder helps on CPython and this
repo but lowers the kernel's aggregate, mostly on the subset-of-a-name queries (0.540 →
0.438), because those answers live in subword identifiers that the encoder re-orders.
vorpal tune runs this measurement on your own queries and enables a tier only when it strictly improves the
mean and wins at least as often as it loses.
Two classes stay weak on every tier. Descriptive queries on the kernel score 0.07
because the right definitions rarely enter the candidate set. Paraphrase queries score
0 everywhere until the encoder's background embedding has filled, since nothing else
reads doc comments. The tables above are measured before that fill, so they are a floor.
--dense-budget-timeout 5m30s caps one fill round; <index>/dense.channel = off opts
out.
¹ The learned tier also runs a per-corpus BM25 check and enables BM25 when paired probes show a clear win. It enabled itself on CPython (38 wins to 17 losses over 512 paired probes) and on this repo (34 to 16), not on the kernel (17 to 10, under the gate's margin).
² f16 on the kernel: 0.296 / 0.290 / 0.302.
One-shot CLI (vorpal search, process start plus index mmap, page cache warm): kernel
0.20 s, CPython 0.01 s, this repo under 0.01 s. The daemon keeps the index warm.
Measured over 30 stdio round trips per tool from a client process, with the server's
resident memory sampled after every call:
| Index · tier | Search median | Search p95 | First search | Graph query | Peak RSS |
|---|---|---|---|---|---|
| Kernel · default | 0.8 ms | 2.1 ms | 0.19 s | 0.13 ms | 2.1 GB |
| Kernel · learned | 2.1 ms | 2.7 ms | 0.22 s | 0.18 ms | 2.4 GB |
| Kernel · learned + f16 | 36 ms³ | 319 ms | 0.69 s⁴ | 0.13 ms | 3.0 GB |
| Kernel · learned + f32 | 36 ms³ | 324 ms | 0.59 s⁴ | 0.13 ms | 2.9 GB |
| CPython · default | 0.3 ms | 0.7 ms | 5 ms | 0.13 ms | 110 MB |
| CPython · learned | 1.4 ms | 1.7 ms | 11 ms | 0.14 ms | 154 MB |
| CPython · learned + f16 | 36 ms³ | 253 ms | 0.37 s | 0.15 ms | 748 MB |
| CPython · learned + f32 | 35 ms³ | 256 ms | 0.29 s | 0.15 ms | 658 MB |
| This repo · default | 0.3 ms | 0.5 ms | 3 ms | 0.10 ms | 65 MB |
| This repo · learned | 1.3 ms | 1.4 ms | 6 ms | 0.12 ms | 79 MB |
| This repo · learned + f16 | 35 ms³ | 218 ms | 0.42 s | 0.13 ms | 652 MB |
| This repo · learned + f32 | 35 ms³ | 226 ms | 0.33 s | 0.13 ms | 561 MB |
³ Encoder medians cycle through a small query set, so most calls hit the 4,096-entry embedding cache. The p95 and first-search columns show the uncached cost: 0.2–0.3 s per new query at k = 10. Graph queries never touch the encoder. f16 halves the download but decodes to f32 in memory, so it is not smaller at run time. Encoder rows include the background embedding fill as it ships by default: complete on CPython (35,364 referenced definitions) and this repo (11,869), at the 10-minute cap on the kernel (36,096 with f16, 44,032 with f32).
⁴ Weights and index already in the page cache. The first process after a reboot pays a one-time page-in, about 4.8 s on the kernel (measured for v0.7.1).
Index on disk (one committed generation with its parsed-product cache and the warmed
search tiers): kernel 8.1 GB default / 8.5 GB with the learned model; CPython
210 / 280 MB; this repo 880 / 910 MB. Encoder weights: 547 MB (f32) or 274 MB (f16),
stored once under ~/.vorpal/models.
Against text search. Structural scan of the Linux kernel (63,775 C files):
vorpal scan --rule rule.yml ~/linux # kind: call_expression + regex: kmalloc
rg 'kmalloc\(' -t c ~/linux # comparison
| Tool | Time | What you get |
|---|---|---|
vorpal scan |
1.5 s (2.5 s the first time) | 6,819 call_expression nodes whose text matches kmalloc |
ripgrep |
0.8 s | 3,387 text lines containing kmalloc( |
The rule matches any call expression containing the pattern, so it also returns
kmalloc_array(...), devm_kmalloc(...), and every outer call that wraps one; an earlier
version of this row reported 42.6k matches from a broader rule. AST-matching 63,775 files
costs about twice a text grep once the parsed products are banked, three times on the
first run, which parses everything.
Against an agent's built-in tools. An agent already has grep and read, so we measured vorpal against them, on this repo and on the Linux kernel, with the v0.8.3 binary on 2026-09-05. Each row asks one question of a warm vorpal daemon (one MCP call, median of five after a first call) and of the ripgrep-plus-read pipeline behind Claude Code's Grep and Read tools. Wall time is the tool's own work. The last column is what the model then has to read.
| Question | vorpal | rg + read | Output the model reads |
|---|---|---|---|
This repo: callers of tool_result |
0.10 ms | 16 ms | 2 records with call sites vs 3 text lines |
This repo: callees of tool_result |
0.14 ms | no equivalent | 7 records with call sites |
This repo: what run_install reaches |
0.05 ms, 1 call | 6 ms, rg -A 75 |
4 records vs 76 lines (3 KB) |
This repo: source of render_toml |
0.05 ms | 39 ms, 2 commands | verified body vs 58 lines |
Kernel: callers of schedule_timeout_interruptible (page of 100) |
2.7 ms | 748 ms | 100 of 140 resolved records (23 KB) vs 164 lines (13 KB) |
Kernel: callers of vfs_read |
0.10 ms | 692 ms | 3 records with call sites vs 13 lines |
Kernel: callees of vfs_read |
0.11 ms | 763 ms, then the body | 4 records with call sites vs 41 lines |
Kernel: find schedule_timeout |
0.05 ms | 677 ms | 1 record vs 1 line |
Kernel: source of vfs_read |
0.08 ms | 713 ms, then a read | verified body vs 42 lines |
Kernel: what vfs_read reaches, depth 2 |
0.06 ms | no equivalent | 3 records |
On a small repo both are far under a model turn; the difference is round trips. On the
kernel every grep rescans 75,954 files (0.7 to 0.9 s) while the graph answers in
microseconds to milliseconds, and grep's lines include definitions, comments, and
macros the model must sift, where the graph returns resolved call edges with their
grades. A name with several definitions comes back as the list of candidates rather
than a merged answer: kmalloc has six in the kernel tree, and its macro form resolves
no call edges at all, so an earlier version of this table that counted those six
candidates as callers was wrong. The first call in a fresh daemon pays a cold open plus the
tree revalidation sweep (119 ms on this repo; 0.27 s on the kernel with the tree's
metadata cached, 2.8 s once when it was not on the kernel); a
kernel tree that changed since its generation pays a rebuild on that first call instead
(9.4 s measured with v0.8.2).
Claude Code hands the model a tool's structuredContent as compact JSON and drops the
text block (checked in the 2.1.261 transcripts), so format shapes what the model reads
through the structured half. For the three callers of vfs_read that is 753 B by default
(lean: name, kind, path, grade, and the call site with its line) and 355 B with
format: ids; each page puts its common directory in one base field. toon only
rewrites the text half, which this client never shows.
What that costs end to end. We tested Claude Code 2.1.261 with Opus 5 at high effort
(pinned with --effort high) on the same four questions, three ways. The first run could
only use Grep, Glob, and Read. The second could only use vorpal's MCP tools as Claude Code ships them, with
schemas deferred, so the first use of each tool costs a ToolSearch turn. The third
could also run the vorpal CLI from the shell; the server's instructions include the
exact command for its index, and the shell tool is never deferred. For that run, allow
the executable by the path the server prints, for example Bash(/path/to/vorpal:*).
Tokens count everything the model processed, cache reads included. Cost is what the API
billed, shown twice: cold, the first run of that arm on that corpus in a fresh hour, which
writes the prompt cache; and warm, the median of three runs straight after it, with the
cache holding the prefix. Turns, tokens, and wall are medians of the three warm runs.
Measured 2026-09-06 with v0.8.3.
| Question | Tools | Turns | Tokens | Cost cold | Cost warm | Wall |
|---|---|---|---|---|---|---|
This repo: who calls tool_result |
grep + read | 5 | 73 K | $0.212 | $0.081 | 9.4 s |
| vorpal MCP tool | 3 | 63 K | $0.178 | $0.054 | 5.9 s | |
| vorpal CLI via shell | 2 | 43 K | $0.236 | $0.028 | 5.3 s | |
This repo: what run_install reaches |
grep + read | 4 | 77 K | $0.224 | $0.136 | 11.8 s |
| vorpal MCP tool | 3 | 64 K | $0.161 | $0.045 | 7.0 s | |
| vorpal CLI via shell | 2 | 44 K | $0.140 | $0.040 | 7.9 s | |
Kernel: who calls vfs_read |
grep + read | 5 | 104 K | $0.197 | $0.200 | 16.6 s |
| vorpal MCP tool | 3 | 51 K | $0.138 | $0.042 | 6.9 s | |
| vorpal CLI via shell | 2 | 36 K | $0.124 | $0.029 | 6.1 s | |
Kernel: what vfs_read calls |
grep + read | 3 | 59 K | $0.116 | $0.053 | 8.2 s |
| vorpal MCP tool | 3 | 51 K | $0.107 | $0.046 | 5.7 s | |
| vorpal CLI via shell | 2 | 36 K | $0.093 | $0.026 | 5.8 s |
Each turn on Opus re-reads the whole context, about 17 K tokens here, so the turn count
sets the token column: two turns through the shell (the command and the reply), three
through the MCP tools (the schema load, one graph or reachable call, the reply), three
to six for grep, which decides how many searches to run and varies from run to run. Warm,
the bill follows the turns: grep $0.05 to $0.14 a question, the MCP tools $0.04 to $0.05,
the shell $0.03 to $0.04. Cold, the first question of the hour writes the prompt cache at
twice the input price, about 10 K to 21 K tokens depending on the arm, and that write is
most of a two- or three-turn bill; it is why the shell arm, the cheapest warm, is the most
expensive cold on the first repo question. Grep's warm runs still write their own tail,
the file contents it read, so on the kernel callers question its warm cost matched its
cold one. Wall time for a two-turn shell run is 5 to 8 s, of which vorpal's own work is
under 0.3 s; the rest is the model. Every vorpal run named the right callers, callees,
and reachable definitions. Grep gave the right file and line every time, but on the repo
callers question it named the enclosing function correctly in one run of six:
tools_call_multi came back as Router::call_tool or was left out. On the kernel
callees question grep's read also listed two inline
helpers, fsnotify_access and add_rchar, that the graph does not resolve as call edges.
On the run_install question grep found one transitive call, claude_desktop_config,
made inside a struct literal, that the graph does not record; the graph arms instead
carried three depth-2 records the resolver grades constrained, names from vendored
grammar JSON, which the model flagged as not real targets. Earlier versions of this
table, back to the 27-tool surface that took 8 turns and 161 K tokens on the kernel
callers question, are in docs/wip/BENCHMARKS.md.
Against the nearest tool. codebase-memory-mcp (cbm, v0.10.8-dev built from source
at 997d087) is also a single local binary with tree-sitter parsing, a typed code graph,
BM25, a Cypher subset, and an MCP server, so the same corpora, labelled queries, and
metrics run against both. Both indexed the same checkouts on the same machine. cbm ran
in its full mode (the only mode with semantic edges), timed through its scriptable
cli; memory is peak RSS over the whole process tree. This table is the v0.7.1
comparison from 2026-09-03, on that day's label sets and machine; we did not re-run cbm
for v0.8.3. The current vorpal numbers for the same rows are in the tables above.
| vorpal | codebase-memory-mcp | |
|---|---|---|
| Linux kernel cold index (75,954 files) | 8.2 s · 8.89 M nodes · peak RSS 5.6 GB · 7.6 GB on disk | 265 s · 8.53 M nodes / 16.0 M edges · peak RSS 70.3 GB · 15.9 GB SQLite |
| Kernel, nothing changed | 0.12 s | 12.5 s |
| CPython cold index (3,841 files) | 1.0 s · 162,945 nodes · 0.8 GB RSS · 200 MB | 36.2 s · 136,118 nodes · 6.6 GB RSS · 663 MB |
| This repo cold index (49 vendored grammar giants) | 7.4 s · 78,894 nodes · 12.2 GB RSS · 836 MB | 44.8 s · 66,141 nodes · 32.3 GB RSS · 291 MB |
| Search, kernel labels (NDCG@10 / MRR / recall@5) | 0.299 / 0.375 / 0.229 (default tier) | 0.116 / 0.104 / 0.167 (BM25) |
| Search, CPython labels | 0.137 / 0.208 / 0.250 default · 0.410 / 0.556 / 0.500 learned + encoder | 0.274 / 0.246 / 0.167 (BM25) |
| Search, this repo's labels | 0.571 / 0.560 / 0.550 default · 0.648 / 0.625 / 0.750 with the encoder | 0.479 / 0.500 / 0.450 (BM25) |
cbm semantic_query (keyword-vector mode), all three corpora |
— | 0.000 on every class |
| One search, one-shot CLI (kernel) | 0.2 s (daemon: 59 ms) | 3.3–5.5 s |
| Callers of a symbol, one-shot CLI (kernel) | 0.06 s (daemon: 0.1 ms) | 3.3–4.4 s |
| Ranking tiers | default · learned (trained per corpus) · neural encoder rerank (f16/f32), per-index tune |
BM25 · regex · static per-token vectors |
| Languages | 49 grammars | 162 grammars |
| Determinism | byte-identical generations, incremental = scratch (release-gated) | not claimed |
cbm ships 162 grammars to vorpal's 49, and its BM25 ranking beats vorpal's default tier
on CPython's descriptive queries. With the learned tier or encoder enabled, vorpal ranks
higher on all three corpora. cbm's semantic_query did not return a relevant definition
for any labelled query. cbm's memory use reflects its RAM-first indexing design.
Yes. Indexing the same tree twice produces byte-identical output: two independent cold builds commit the same content-addressed generation on every corpus above. Incremental builds converge to the same bytes as from-scratch builds; a release-gated battery checks scratch determinism plus six edit shapes across three repositories, and the kernel's one-shot edit is verified to the same generation id.
- Structural search and rewrite. Match code by AST pattern instead of regex:
vorpal run -p 'console.log($ARG)'. YAML rules, project scanning, rule testing, LSP, and interactive rewrite come from the ast-grep engine. - A code knowledge graph. Every definition is a node;
calls,imports,implements,of_type,references, and containment are edges, all derived from the AST rather than substring matching. - Resolution you can audit. References resolve with scope precedence and a confidence label. Anything that cannot be resolved is counted and reported, not guessed.
- Hybrid search. One query fuses exact and token name matching, embedding similarity, and graph in-degree (reciprocal rank fusion), with per-channel provenance on every hit.
- Incremental by construction. Per-file extraction is cached; a re-index re-parses only what changed and always re-links the whole graph, so renames and deletions never leave stale nodes.
- 49 languages in one binary. No plugins to install.
All 49 grammars are compiled into the binary: Astro, Bash, C, C++, C#, CMake, CSS, Dart, Dockerfile, Elixir, Erlang, Go, GraphQL, Haskell, HCL/Terraform, HTML, INI, Java, JavaScript, JSDoc, JSON, Julia, Kotlin, Lua, Make, Markdown, Nix, Objective-C, OCaml, Perl, PHP, PowerShell, Protobuf, Python, R, Ruby, Rust, Scala, Solidity, SQL, Svelte, Swift, TOML, TSX, TypeScript, Vue, XML, YAML, Zig. Vue, Svelte, and Astro single-file components are parsed with the embedded script, style, and frontmatter grammars. The relations each language supports are in the language matrix. Anything not extracted is absent, not guessed.
| Doc | What's in it |
|---|---|
| Getting started | Install, first index, every CLI command with examples |
| MCP setup | Connect vorpal to Claude, Codex, or any MCP client; the tool reference |
| Python · TypeScript/JS | Library quickstarts (patterns + index API) |
| Supported languages | The full matrix of what each of the 49 grammars extracts |
| Architecture | Storage format, memory model, concurrency, scaling roadmap |
| Index format | On-disk compatibility & migration policy |
parse (tree-sitter, 49 grammars)
→ extract definitions (YAML outline rules) + references (AST walk: calls/imports/types/impl)
→ intern blake3 path-qualified identity → dense node ids (dedup, incremental skip)
→ store columnar node segment (mmap, checksummed) + string heap + edge lists
→ resolve scope-precedence, confidence-labeled; approximate edges labeled, never faked
→ link resolved references become graph edges (CSR/CSC, both directions)
→ query name / graph / transitive closure / hybrid RRF search
Four rules hold throughout. Prefilters may only skip work that provably cannot match. Incrementality caches extraction, not conclusions, so the graph re-links from complete inputs every run. Edges are created only on grammar-proven evidence. Builds are deterministic. Design and the scaling roadmap: docs/wip/ARCHITECTURE.md.
cargo build -p vorpal # the main binary
cargo test --workspace # full suite
cargo clippy --workspace --all-targets -- -D warningsWorkspace layout, the extraction pipeline, and how to add a language are in docs/wip/ARCHITECTURE.md.
Vorpal's structural search engine began as ast-grep by Herrington Darkholme and contributors. The knowledge graph, semantic search, and MCP layers are original to vorpal.
MIT — © 2026 Ada Lundhe; portions © 2022 Herrington Darkholme (ast-grep). See LICENSE.