Skip to content

Knowledge Graph Pipeline

Jump to bottom
Chris Sweet edited this page Jun 25, 2026 · 5 revisions

type: concept up: "Wiki-LLM-Pattern" status: implemented tags: [kg, sparql, retrieval]

Knowledge Graph Pipeline

The frontmatter and body cross-references in the wiki feed an optional knowledge-graph pipeline that exposes the wiki as a SPARQL-queryable graph. Shipped in PR #14 as scripts/kg/. See Implementation-Status for the broader real-vs-aspirational classification and Wiki-Corrections-Log for the 2026-06-12 entry recording the transition of this page from aspirational to implemented.

What it does

scripts/kg/build-graph.sh walks the wiki, extracts frontmatter and body links, and produces a typed-edge graph as both JSON-LD and Turtle, plus a SHACL conformance report. The pipeline runs in-process Python using rdflib + pyshacl: no Java, no Fuseki, no subprocess.

  • Frontmatter typed edges (extends:, supports:, criticizes:, source:, up:, related:) become typed predicates resolving under the LA3D ontology (https://la3d.github.io/llm-wiki-colab/ontology#).
  • Body cross-references ([text](Page-Name)) become mentions edges, weighted via RDF-star.
  • Materialisation (inverse edges, area inheritance, hub flagging) runs as SPARQL CONSTRUCT queries against the loaded graph.
  • SHACL validation produces validation-report.ttl.

Outputs land in scripts/kg/build/ (gitignored): graph.jsonld, graph.ttl, graph-weights.ttl, graph-full.ttl, validation-report.ttl.

The typing comes from Typed-Edges-and-Frontmatter; the operations come from Three-Operations.

When to use it

Use the KG for topology questions:

  • "Which pages cite finding X?"
  • "What concepts does this experiment depend on, transitively?"
  • "Find all pages tagged Y that criticizes: a parent of Z."

Use direct file reads for content questions: "What does page X actually say?"

The right pattern: use the KG to discover where to look, then use file tools to read what the chosen pages say. Reserve grep for non-wiki code or content searches that span many files.

Rebuilding

Run ./scripts/kg/build-graph.sh after each ingest, or periodically. The script is idempotent. Flags: --wiki=PATH for a custom wiki location, --refresh-spec to re-fetch the LA3D ontology/SHACL spec (cached in .cache/), --stats for extractor coverage, --help for the full list.

SPARQL endpoint (optional)

The default mode is in-process: queries run against the loaded graph object directly. No server, no HTTP. For multi-client query, agent-write via SPARQL UPDATE, federated queries, or web dashboards, Apache Jena Fuseki can host the produced graph-full.ttl and rdflib talks to it via SPARQLStore. This is opt-in. The future agent-memory feature is expected to depend on the Fuseki path because cross-session writes need SPARQL UPDATE and an HTTP surface.

Distribution to derived projects

scripts/kg/ is not yet listed in ALWAYS_FILES in either sync script (see Sync-Flow). The pipeline exists in the template repo and is exercised by the test harness, but derived projects do not automatically receive it on update-from-template.sh. The likely path is shipping it as an opt-in feature under features/ (see the in-progress feature-flag framework, RFC #13) rather than promoting to ALWAYS_FILES.

Why typed edges matter here

Without the typed-edge contracts, the KG collapses to a directed graph of mentions between pages. That is useful but flat. With typed edges, the same query can ask not just "which pages connect to X" but "which pages support a claim that X". Edge typing is what turns the KG from a navigation aid into a reasoning substrate.

See also

Clone this wiki locally