08 CPG Joern Layer

CPG & Joern Layer

The optional interprocedural layer: what it adds, what it costs, and the twelve things its documentation does not tell you. Deep source: docs/implementation/CPG_INTEGRATION.md — a primary-source study where every fact carries a confidence level.

---

Why this layer exists

tree-sitter gives per-file structure: symbols, intra-file calls, SQL literals. It cannot see interprocedural reach — what actually calls what across files, with resolved names. Joern's code property graph (AST+CFG+DFG+PDG) can. On the pilot, adding the layer lifted M2 recall 0.771 → 0.858 (median 1.000, zero-recall cases 1 → 0) and unlocked slice.

Without it the product works: capability=degraded, slice refuses with exit code 3 and the reason, everything else functions. Degradation is reported in every envelope — never silent.

Installation

# automated: Temurin JDK 21 (Adoptium API) + Joern CLI (GitHub release),
# SHA-256 verified, ~5-6 GB, idempotent
pwsh -File scripts/install-joern.ps1 -Download

# air-gapped: place joern-cli.zip into .tools/downloads/ and JDK into .tools/jdk/
pwsh -File scripts/install-joern.ps1

Discovery order used by the pipeline (cpg.py:find_java etc.): RAGRAF_JAVA_HOME/RAGRAF_JOERN_HOME env → .tools/jdk, .tools/joern-cli → PATH. Heap limit: index.joern_heap_gb (default 8) or RAGRAF_JOERN_HEAP_GB, passed as -J-Xmx8g.

Versions verified in production: Joern 4.0.99, Temurin JDK 21 (docs.joern.io names 19 as tested, «newer might work» — 21 is what RAGRAF measured; note joern-slice has a documented Circe classpath issue on Java >17, which is one reason the shipped pipeline exports instead of slicing in-process).

The precondition nobody documents: external PHP

php2cpg contains no PHP parser. It shells out to a php executable with a bundled php-parser.phar (nikic/PHP-Parser 4.15.10, syntax up to PHP 8.2). Without php in PATH the frontend does not degrade — it refuses to start. The pipeline therefore runs a dedicated find_php() and puts the php directory first in PATH (the frontend invokes literally php; a full path to php.exe does not work).

Consequences for modern code: PHP 8.4 constructs (property hooks, asymmetric visibility) are absent from the 4.x grammar. --with-recovery is always passed, so expect an incomplete AST, not a crash — and count the losses in the coverage metric (Failed to process '<file>' lines in the Joern log are warnings under which files silently drop).

The twelve run-verified facts (documentation does not contain them)

Seven of these caused silent data loss — the code didn't crash, the result was zero or distorted. This table is the operational core of the integration:

#	Discovery	Symptom	Resolution in RAGRAF
1	php2cpg requires external php in PATH	frontend never starts	find_php() + PATH prepending
2	php2cpg rejects --exclude	Unknown option → RuntimeException: XTypeStubsParserConfig	filter noise paths AFTER export (is_noise_file)
3	Script mode (--script) is broken on JDK 21	no predef (importCpg, cpg not found)	abandoned scripts; everything via joern-export
4	GraphSON export OOMs even at 18 GB	Required array length 2147483640 + 8 is too large	neo4jcsv: streaming CSVs, 223 MB, 27 s
5	--repr=cpg writes a file per method and dies	731 files, 739 MB, FileSystemException	--repr=all + filter while parsing
6	JAVA_OPTS never reaches the JVM	OOM at any value	limit as argument -J-Xmx18g (form without =)
7	Joern paths don't match ours	leading dot + backslashes → 0 matches	normalize_joern_path
8	Joern writes methods with ->, not ::	App\Core\Config->get unmatched	normalize_joern_name → 411 full-name links
9	Unbounded AST walk captures everything	the <global> stub (present in every file) owned 179 330 of 183 357 calls	stop traversal at nested METHOD nodes
10	The export's vertex array is vertices, not nodes	0 vertices against 83 542 parsed edges	key @value.vertices.item; properties UPPER-CASE under properties
11	Most CALL nodes are language operators	50 K of 183 K were <operator>.assignment	_is_noise_call → 17 822 real calls
12	PHP builtins need separate classification	13 008 of 17 822 «unresolved» were array_push, is_array	_is_builtin_target: 7 819 recognized; honest remainder 4 985

The lesson: such defects are caught only by cross-checking independent paths (CPGQL «worked in theory»; the <global> walk was «formally correct»). Any CPG-related change must be validated by comparing against tree-sitter counts, not by «it ran».

Name resolution contract

Indexer._make_fqn_resolver: exact full-name match → method-name+file match → refusal. The third outcome is principled: in framework-less PHP, short names (get, run) belong to dozens of classes — binding «by name» would fabricate exactly the links principle P3 forbids. dispatchType (STATIC_DISPATCH vs DYNAMIC_DISPATCH) separates proven from heuristic edges.

Pipeline shape & costs (pilot measurements)

cpg stage:  joern-parse (--language PHP) → cpg.bin (18.6 MB, ~50 s / 323 files)
export:     joern-export --repr all --format neo4jcsv (223.8 MB, 132 files, ~27 s)
parse:      streaming CSV via ijson (the hand-rolled JSON parser LOST NODES on a real 95 MB file)
merge:      confirmed calls into graph.sqlite
totals:     193 442 meaningful vertices · 3 348 516 edges (444 464 CFG, 930 567 data-flow)
            → 17 822 real calls → 603 confirmed links (411 by full name, 192 by name+file)
            full indexing with CPG: 71 s (CPG = 64 s of it)

• Exports are pruned before each indexing (keep current + 2): auto-reindex once accumulated 37 export dirs = 1.5 GB (article defect №8).
• The Joern server stops automatically when the indexer closes (indexer.py:1166-1183). After manual experiments, check for stray multi-GB java processes holding port 9500.
• No incremental CPG exists — every reindex rebuilds it fully; that is why the guard's «fast» mode skips CPG below 20 accumulated edits.
• CFG/DFG are extracted but only CALLS are merged into the graph today; a true PDG slice is roadmap R-CPG-2. slice currently traverses confirmed call links.

Degradation matrix

Condition	capability	slice	Everything else
Joern+JDK+php present, export parsed	full	works	works
Joern missing / JVM broken / timeout / OOM	degraded	refuses, exit 3, reason in index_status stage cpg	works
Export returned code 1 but cpg.bin exists	degraded	refuses	works (known unlocalized defect — «next on empty iterator»; recorded, not hidden)

Never present degraded as «the index is broken» — it is an honest capability mark with a recorded reason.

---

Next: Admin Panel →