01 Problem Statement

Problem Statement

Why «an agent + grep» is not enough — and what exactly RAGRAF solves. Every claim below is either a measurement (artifact linked) or a citation.

---

1. The three symptoms

Everyone who has tasked an LLM agent with an edit in an existing project knows them.

Symptom 1 — The agent cannot find the place

It answers confidently about a function; verification shows the function lives in another file, and the one described is a namesake. In a 392-file repository, a name grep yields dozens of matches — and the needed one is not first.

Measured on the pilot corpus: rg found files but did not localize: 34 candidates for SuperAdmin, 44 for z_e(, 64 for namespace BatteryService. Finding the file is easy; finding the place inside the file is not.

Symptom 2 — The agent does not know the schema

It writes a query with a column AuthorUserID because that is what the variable is called elsewhere in the same codebase. No such column exists. This surfaces at runtime, not while reading:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'AuthorUserID' in 'field list'

The journals used InitiatorID; UpdatedBy existed in exactly one table out of five. The code was already written when this appeared.

Symptom 3 — The agent does not know it is stale

The worst one. An index was built; the code was edited afterwards; now the agent receives facts about a state that no longer exists. From the answer alone you cannot tell «no such function» from «the function appeared after indexing». A decision is made on that answer — and it is wrong.

---

2. What the industry data says

Navigation is the dominant failure mode

The VulnGym study (cited in Positive Technologies' article on graphs in code audit): in tasks where the source-to-sink chain is longer than nine steps, the best agent found 22.58% of vulnerabilities, and in 56.5% of failures it either never found the needed file (26.8%) or found the file but not the line (29.7%).

More than half of the failures are not about analysis. They are about navigation. That is the part RAGRAF moves out of the model into an external index.

Free-form graph queries are an error generator

LLMxCPG (USENIX Security '25) measured base models generating CPGQL queries against a code property graph:

Model	Valid queries
Base model, config A	19 / 1278
Base model, config B	132 / 1278
Fine-tuned for query generation	1278 / 1278

The fine-tuned model is perfect because it was trained for exactly this. An arbitrary agent was not. Hence RAGRAF's architectural decision: no query language at all — a fixed operation set instead (Architecture → Principles).

Related literature the design leans on

• ALIBI — comment injections suppress detection in >90% of cases → free text is stored separately from structural facts (text_chunk vs node/edge).
• «Words Speak…» — phrasing shifts verdicts by 33.2% → answer wording is neutral by requirement (FT-23).
• Repoformer — the right not to search gives up to 70% speedup → an honest «not found» beats an approximate answer (P8).
• LCIP (arXiv:1512.07435) — measured impact-analysis ceiling: precision 69% / recall 79% / F1 55% (Java, mutation simulation) → used only as an order-of-magnitude reference for our M2 metric, never as a target to «beat» in prose.
• AdaRepair-Mem — «more memory ≠ better» → the drift ledger is archived and deduplicated, not accumulated forever.

---

3. Why existing tools do not close the gap

Alternative	What it gives	What it misses
rg / grep + file reads	universality, zero setup	localization (34–64 candidates); no structure; no freshness notion; every read costs context
LSP / IDE indexes	precise definitions and references	per-language servers; no cross-layer links (code ↔ DB schema ↔ routes); not designed for agent consumption; no provenance/staleness contract
Plain RAG (chunk everything, embed, retrieve)	semantic search	no who calls whom; no schema facts; silently stale after edits; similarity ≠ structure
Graph DB + Cypher/CPGQL	full expressiveness	the model cannot write valid queries (19–132/1278); a server to run; facts without an evidence contract
Joern / CodeQL directly	real CPG, taint analysis	JVM/CodeQL weight; query language problem again; no freshness ledger; no fixed contract for an agent

RAGRAF's position: take the structural precision of a graph and the semantic reach of vectors, wrap them in a fixed, verifiable, freshness-aware contract of 22 operations, and store everything in files (SQLite) with zero services. The optional Joern layer adds interprocedural reach — and its absence is reported, never hidden.

---

4. The problem RAGRAF itself must not create

An index is a cache. A cache that lies is worse than no cache:

1. Silent staleness — facts from last week served as current. → Immutable revision + drift ledger + stale_warnings in every envelope + auto-reindex guard.
2. Hallucinated links inside the index — a plausible edge «because the database said so». → Every edge carries evidence path:line, provenance (stated/derived), confidence. Heuristic edges are a separate kind and a separate answer block.
3. Unverifiable answers — the agent cannot check what it received. → 100% of code-touching facts carry path:line; metric M6 verifies links automatically.
4. Capability masquerade — «slice returned nothing» vs «slice is impossible in this mode». → capability: full|degraded in every envelope; slice refuses with exit code 3 and the reason; silent degradation is forbidden by design.
5. Secret leakage into the index — corpus configs and .env values ending up in vectors. → secrets_exclude is exclusion-only (never masked-and-stored); telemetry stores argument hashes + a 60-char digest, not full text.

These five anti-requirements shaped the architecture more than any feature. Each maps to a verifiable mechanism — see Architecture and the acceptance protocols (docs/architecture/ACCEPTANCE.md, metrics M5/M6, tests M3-T2).

---

5. What the solution demonstrably changed

From one real development effort (the /agreements/ document-flow section — journals of payments, contracts, vacations), all numbers from the call journal and test runs:

Outcome	Number
Tool calls, 14 navigation tasks	28 (realistically 43) → 14
Files read in full	14 → 0
First-query hit rate	46% → 92%
Fact verifiability (path:line)	30% → 100%
Defects found in the feature under development	9 (3 of them — broken approval routes and non-working field protection)
Defects found in RAGRAF itself by the same work	13 (5 of them invisible to synthetic tests: «works, but measures itself wrongly»)
Defects prevented before code was written	4 (one by schema, the rest by write_check / policy_trace)
Runtime errors in the session	0

The dominant defect class was «works, but wrongly» (8 of 16): a page opens, no exception is thrown — and the rejected document is a dead end, or any employee can mark a payment as paid. Neither a compiler nor an «it opens» test catches these; an exact schema in the index plus a static permission trace does.

Full case studies with code: docs/article/RAGRAF_ARTICLE.md §9, §11.

---

6. Scope — what RAGRAF is not

• Not a code rewriter. RAGRAF answers; the agent edits. There is no autonomous patching.
• Not a vulnerability scanner. Taint analysis is not a goal (the CPG layer makes it possible later; see roadmap R-CPG-3).
• Not a service. No daemons in the base install, no network endpoints except the loopback admin panel. Three SQLite files in <project>/.ragraf/.
• Not a truth oracle. Every answer lists its limits («static analysis; runtime dynamics not covered») — and the honest overall verifiability across all operations is 24.4%, because schema facts have no line numbers. We do not draw in references to prettify reports.

---

Next: Getting Started →