11 Benchmarks and Limits

Benchmarks & Limits

What was measured, how, and — with the same rigor — what was NOT. This page exists because a measurement without a stated boundary is marketing. Protocols: docs/architecture/ACCEPTANCE.md; raw artifacts: bench/results/.

---

Measurement doctrine (7 rules)

1. Baseline BEFORE the index, stored in bench/baseline/. A baseline captured afterwards is not a baseline.
2. Identical tasks for both arms — wording, files, done-criterion; only the toolset differs.
3. Tasks fixed in advance (bench/tasks.yaml); tasks invented after seeing answers don't count.
4. Every measurement is an artifact: JSON numbers + markdown interpretation. No artifact → no measurement.
5. Tokens counted by tiktoken, one fixed tokenizer per comparison.
6. Nobody else's percentages as targets — literature values are order-of-magnitude references, labeled as such next to every number.
7. Reproducible: the capture command is written into the artifact.

The seven metrics

Metric	Question	Status on the pilot
M1 navigation accuracy	first-query hit on file+line?	46% → 92% (12/13)
M2 impact completeness	can the graph predict co-changed files?	shipped artifacts: recall 0.771 (rev-0012, 20 commits) → 0.917 (rev-0025, 8 commits); with CPG 0.771→0.858; precision is low by design (0.02–0.07 — connectivity, not prediction, see caveat)
M3 context savings	tokens & calls per task	calls 28 (realistic 43) → 14; ~5 200 tokens of file reads saved (estimate)
M4 slice compression	tokens(slice)/tokens(surroundings)	measured on the pilot; forbidden to compare with the unconfirmed 68–70%/90.93% from secondary literature
M5 freshness	incidents of relying on stale facts	target 0 / second-kind incidents 0; stale_warnings coverage 100% of affected paths
M6 verifiability	links auto-confirmable?	100% for code-touching operations; 24.4% overall (schema facts have no lines by nature)
M7 update cost	full ≤15 min, incremental ≤3 min, p95 reads	full pilot index 71 s (CPG 64 s); schema 0.5 ms · symbol 0.7 ms · table_usage 0.9 ms · write_check 3.3 ms · hybrid search up to 1 s

Shipped artifacts (verifiable by anyone)

File	Contents
bench/results/M2_rev-0012.json	20 commits, depth-3 bidirectional graph walk: per-commit files_changed, primary_symbol, reachable, naive_recall (0.0 baseline), graph_recall, graph_precision; means + the LCIP reference orientation inside the artifact
bench/results/M2_rev-0025.json	8 commits on the later revision: recall mean 0.9167, median 1.0, zero-recall cases 0

The artifacts contain commit SHAs and relative paths of the author's private PHP corpus — the commits are not publicly resolvable; no secrets are present. The README states this explicitly (defect G-06 resolution).

The headline table (14 navigation tasks)

Indicator	Without RAGRAF	With RAGRAF	Notes
Tool calls	28 (realistically 43)	14	baseline computed in favor of rejecting the index (agent opens the right file first try); 43 = reviews up to 5 candidates
Calls per task	2.0 / 3.1	1.0
Candidates to review	up to 34 files	1–5 exact	rg finds files, doesn't localize: 34/44/64 candidates for three probe queries
Files read in full	14 (minimum)	0
First-query target hit	—	92% (12/13)	was 46% before improvements
Facts with path:line	—	100%	was 30%
Wall time, 14 tasks	0.32 s (rg only)	1.40 s	+1.1 s — the price of structure; honestly recorded

What the real development effort yielded

One feature (/agreements/ document flow), 46 logged operation calls, 0 runtime errors:

• 16 defects found or prevented: 4 before code ran (one by schema alone), 8 of the class «works, but wrongly» (broken approval routes, non-working field protection, over-broad permissions — invisible to compilers and «it opens» tests), the rest by measurement.
• 13 defects in RAGRAF itself found by the same work — including 5 that synthetic tests cannot show («code worked correctly but measured itself wrongly»: zero vector scores, RRF lifting noise, 11 K-token chunks, declared-but-empty schema layer, answers without path:line).
• The permission defect (any journal holder could mark_paid) was found once by a live-role run — and is now visible statically, before running, via policy_trace for any action.

Limits — the honest section

Limits of the measurements

1. Instrumental, not agentic. The harness counts calls and verifies links; it does not model reasoning. «Agent with index vs agent without» on real edit tasks is the next evidence level — not done (roadmap R-EVID-1).
2. Small sample. 14 navigation tasks + one feature. Enough to see a difference; not enough for statistics.
3. Tokens are estimates. Coefficients (4 chars/token, 45 chars/line) measured on two corpora; inference itself not measured. Every derived number is labeled «estimate».
4. Baseline favors the index-less arm (one file open per task). Both numbers (28/43) ship in the artifacts.
5. M2 precision is intentionally low (0.02–0.07): the shipped M2 measures connectivity of co-changed files (a lower bound of graph usefulness), not change prediction. Comparing it to LCIP's precision 69% would be dishonest; the artifact says so in reference_orient.

Limits of the product (as shipped)

Limitation	Detail	Roadmap
CFG/DFG extracted, not merged	444 464 control-flow + 930 567 data-flow edges sit in the export; only calls (540–603 confirmed) merge into the graph	R-CPG-2
slice ≠ true PDG slice	traversal over confirmed call links	R-CPG-2
Vectors unnormalized at rest	exact cosine computed at query time (works, costs)	R-PERF-4
Hybrid search up to 1 s	query embedding computed per call; no cache	R-PERF-3
Verifiability 24.4% overall	100% on code operations; schema has no lines by nature	accepted property, not a bug
Docblock contract in symbol only	code doesn't carry array_shape	R-CORE-5
Sequential AST parsing	docs claimed parallel; code is sequential	R-PERF-1
Windows-first	POSIX path works but unverified; glob case-sensitivity bites on Windows	R-PORT-1, R-CORE-1
PHP CPG syntax ceiling 8.2	php2cpg's bundled parser; 8.4 constructs parse with recovery/losses	watch upstream

What turned out to be a false alarm (methodology keeps this public)

The Shell.php «defect» (curl_multi_remove_handle() expects 2 arguments, 4 given on a line calling something else entirely) was stale opcache in a long-lived php -S process. Lesson shipped into the regimen: restart the stand after edits or disable opcache — otherwise hours go to hunting a nonexistent defect.

---

Next: FAQ →