03 Architecture

Architecture

The condensed engineering view. The full normative document: docs/architecture/ARCHITECTURE.md; requirements: SPECIFICATION.md.

---

The one-paragraph model

RAGRAF snapshots a corpus into an immutable revision (rev-NNNN): a property/AST graph, an FTS5 lexicon, 384-dim vectors, a DB schema parsed from migrations, and optional interprocedural calls from Joern — all in three SQLite files inside <project>/.ragraf/. Reads go through a facade of 22 fixed operations (no query language), each answer wrapped in an envelope (index_rev, capability, drift, provenance, limits, stale_warnings, freshness). While the agent edits code, the index does not change: deltas accumulate in an append-only drift ledger, affected facts get marked stale, and a checksum guard reindexes automatically (or on reindex), archiving the ledger into a «what changed in the knowledge» report.

flowchart LR
    subgraph SRC["Corpus (read-only)"]
        F["source files"]
    end
    subgraph PIPE["Indexing pipeline"]
        S["scan<br/>sha256"] --> A["ast<br/>tree-sitter"] --> R["resolve"] --> SC["schema<br/>migrations"] --> C["cpg<br/>Joern (opt.)"] --> LV["lexvec<br/>FTS5+vec0"]
    end
    subgraph STORES[".ragraf/"]
        G[("graph.sqlite")]
        L[("lexvec.sqlite")]
        K[("control.sqlite")]
    end
    subgraph FAC["Facade"]
        E["22 operations<br/>+ envelope"]
    end
    F --> S
    A & R & SC & C --> G
    LV --> L
    S & LV --> K
    E --> G & L & K
    AG["LLM agent<br/>(MCP / CLI)"] --> E
    W["guard: corpus_hash"] -.->|stale → reindex| PIPE
    W -.->|freshness| E

Loading

Principles

Eight principles, each derived from a measured failure mode — not negotiable without an architecture revision:

#	Principle	Why
P1	The index is immutable within a revision	Otherwise facts of different moments mix in one answer — the birthplace of wrong conclusions
P2	Staleness is shown, never resolved to null (stale marks)	Staleness indistinguishable from an error is worse than no index
P3	No edge lives as a fact without evidence (path:line + confidence)	Protection against hallucinated links inside the index
P4	No query language for the agent — fixed operations only	LLMxCPG: base models write valid CPGQL in 19–132/1278 cases
P5	The graph is primary; vector & lexicon are supplementary	Vectors catch surface similarity, not interprocedural flows
P6	Free text is stored separately from structural facts	ALIBI: comment injections suppress detection >90%; phrasing shifts verdicts 33.2%
P7	Pilot small but end-to-end (6–10 files first)	Debugging a pipeline on 162 K lines is a time sink
P8	Precision over completeness at stage one	An honest «not found» + live_read beats an approximate answer (Repoformer: up to 70% speedup from the right not to search)

The three stores

File	Contents	Lifecycle
graph.sqlite	rev, file, node, edge (with provenance, evidence_path/line, confidence, sources, edge_uid), text_chunk, cfg_edge/dfg_edge (CPG), db_table/column/fk, web_route, invalidated, coverage	large, rebuildable
lexvec.sqlite	ft_chunk (FTS5, 4 tokenized content columns — the column count is a contract with bm25() weights), chunk_text (raw text, separate table on purpose), vec_chunk (vec0, FLOAT[384]), chunk_map, vec_meta	large, rebuildable
control.sqlite	rev_history, drift_entry (append-only, dedupe_key UNIQUE), drift_archive, calibration, stale_edge, telemetry usage_call	small, never deleted

Full DDL with the reasoning comments: ARCHITECTURE.md §4.

Identity = content

Determinism is a requirement (NFR-5), so every identifier is a content hash:

node_id   = sha1(kind | fqn | path | start_line)          ← line component mandatory
edge_uid  = sha1(src | dst | kind | evidence_path | evidence_line)
chunk_id  = sha1(content_sha256 | chunk_kind | model)     ← deliberately WITHOUT revision/path
corpus_hash = sha1(sorted list of (path, sha256))          ← the «input unchanged» signal

chunk_id's revision-independence is the main rebuild saving: unchanged chunks keep their embeddings across revisions. Building the same revision twice must produce identical identifier sets (test M1-T4).

Provenance: stated vs derived

Every edge carries provenance:

• stated — proven by an analyzer (exact declaration/call line, information_schema row). confidence = 1.0.
• derived — inferred (dynamic calls, name matches, embedding proximity). confidence from a calibratable table: 0.60 unambiguous heuristic · 0.40 ambiguous · 0.30 history/embedding inference.

calls_heuristic is a separate edge kind: heuristic edges are included (otherwise dynamic PHP has no links at all) but never silently mixed with proven ones — impact --mode hard returns only stated. Conflicts between sources are resolved by priority (information_schema > migrations > CPG > SCIP > tree-sitter) and recorded in coverage as conflict.*, never swallowed.

The drift ledger (why the index doesn't rot silently)

1. reindex --commit → rev-N+1 → ledger empty
2. agent edits      → compare/git-diff → drift_entry (add/modify/delete/rename/schema_change/slice_boundary)
3. any facade answer → stale_warnings for affected paths → agent: live_read or trust the index
4. reindex --commit → ledger archived → report «what changed in the knowledge»

Invariants: I1 the index doesn't change mid-revision · I2 ledger is append-only (errors are corrected by new entries) · I3 no silent reliance on an affected fact · I4 slice_boundary is recorded separately from modify (border changes are the one edit class that durably breaks slices) · I5 the archive answers «what changed in the knowledge», not «which files changed».

Five detection methods: hook:session-start, compare (sha256 + symbol diff), git-diff, db-schema-diff, manual. Deduplicated by (path, old_hash, new_hash, kind).

The freshness guard

Before serving, the facade can verify corpus_hash against the indexed one (15-second result cache; freshness --force bypasses). On divergence: per-file diff → automatic reindexing under a lock (file with pid + timestamp; a dead process's lock expires after 15 minutes; pid liveness is checked so a killed run doesn't block for 15 minutes). The guard's verdict (changed_since_index, reindexed, hashes) rides in every answer's freshness field. If the guard itself fails, data is still served — with stale: true and the reason. Two defects found while testing the guard (8.3-path relative_to silently skipping every directory; second-precision ORDER BY created_at making revision order undefined) are documented in the article §6 — both were of the «works, but wrongly» class.

Capability & degradation

Mode	Condition	Available	Contract
full	CPG built	everything incl. slice	—
degraded	CPG absent/failed	graph, symbols, schema, routes, lexicon, vector	rev_history.joern_used=0; every envelope carries capability: degraded; slice refuses with exit code 3 and the reason

No embedding model → lexical search works, envelope says so. No free port → the server reports the cause. Silent degradation is forbidden (src/ragraf/cpg.py:9-11).

Layers & processes

Components are separate processes, not libraries inside the agent:

Layer	Component	Tech
L5 integration	MCP server (stdio, JSON-RPC 2.0, 23 ragraf_* tools), project AGENTS.md regimen, skill	Python
L4 facade	22 operations + envelope + drift gate	src/ragraf/facade.py
L3 stores	graph / lexvec / control	SQLite 3.50.4 + FTS5 + sqlite-vec
L2 loading	normalize → schema → dedupe → idempotent revision load → invariant checks	indexer.py, revisions.py
L1 extractors	CPG (Joern+JDK) · symbols (SCIP/PHPStan where available) · syntax (tree-sitter, 16 languages) · SQL/DB (migrations, information_schema) · text (md/comments/docblocks)	extract/, cpg.py, schema_db.py
L0 sources	repository files · migrations · docs	—

Performance contract (NFR-4) and reality

Operation	Target	Notes
symbol / references / callers	≤ 200 ms p95	index-backed (fqn, src/dst)
impact (depth 3)	≤ 1 s p95	recursive CTEs
search (hybrid)	≤ 1 s p95	FTS5 + KNN scan; measured mean ≈ up to 1 s — embedding computation dominates; no query-embedding cache yet (roadmap R-PERF-3)
slice	≤ 2 s p95	inside Joern
Full index, 162 K-line pilot	≤ 15 min	measured 71 s on a 323-file corpus incl. CPG (64 s of it); parsing is sequential in the shipped code — the documented «parallel parse (8 cores/16 threads)» is a target, roadmap R-PERF-1
Reindex after an edit	minutes	embedding reuse by chunk_sha256; CPG rebuilds fully (no incremental CPG exists in Joern)

Schema-tier operations are effectively free: measured means schema 0.5 ms · symbol 0.7 ms · table_usage 0.9 ms · write_check 3.3 ms.

Security & secrets posture

• Secrets are excluded, never masked-and-stored: secrets_exclude files never reach text_chunk, vectors or logs (FT-39/40, test M3-T2 on patterns).
• db.dsn_from indexes the fact «this symbol references config», without values.
• Telemetry stores sha1 of arguments + a 60-char digest, not full query text.
• Admin panel: loopback-only bind, POST-only mutations, token via secrets.compare_digest, project-config writes backed up (.bak.<ts>) and validated. Known gap: GET /api/token hands the token to any local caller and the page auto-fetches it — treat the token as protection against accidents, not adversaries, until R-SEC-1 lands (Host/Origin validation + manual token entry).
• No shell=True except the documented Joern .bat launcher path (hand-quoted; hardening tracked as R-SEC-4).

Where the numbers come from

Every «measured» claim in the docs points at an artifact: bench/results/*.json (shipped: M2 rev-0012 & rev-0025), the call journal usage_call (per-corpus), and stage timings in index_status. Estimates (tokens ≈ chars/4, lines ≈ chars/45) are labeled «estimate» everywhere they appear — the coefficients themselves were measured on two corpora.

---

Next: Operations Reference → · Deep dive: ARCHITECTURE.md (normative, with full DDL)