docs(adr): importer chunking contract + code-graph ontology convergence#641
Conversation
…y convergence Two ADRs to close out the open architectural questions surfaced by the Graphify experiment (PR #602): - ADR 0002 (importer chunking contract): clients chunk, daemon publishes the limits. Pins CHUNK=5000 quads/write + ROOT_CHUNK=1000 URIs/promote as the de-facto values importers should use (matching what .dkg/scripts/scan-code.mjs has been using in production). Documents 413 handling as a normal recoverable error, and explicitly rejects a server-side "/api/import/bulk" endpoint (would hide failure modes, duplicate manifest logic, defeat resumability). - ADR 0003 (code-graph ontology convergence): canonical URI shapes for urn:dkg:code:{package,file,module}:* and the producer-scope rules so monorepo scans, external-repo scans, and future importers don't fragment the graph. Deprecates the graphify:* namespace introduced in PR #602 in favour of urn:dkg:code:* + owl:sameAs reconciliation for graphs that have already promoted the legacy URIs. Both ADRs cite the canonical helpers in scripts/lib/ontology.mjs and cross-reference each other. They unblock the importer SKILL.md and manifest library in the follow-up PR. Related to #596 (graphify import fixes) and complements #636 (bug report), #640 (WM persistence fix). Co-authored-by: Cursor <cursoragent@cursor.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| urn:dkg:code:module:<moduleName> | ||
| ``` | ||
|
|
||
| - `<pkgName>` and `<relPath>` MUST be `encodeURIComponent`'d per path |
There was a problem hiding this comment.
🔴 Bug: This text says Code.uri.file percent-encodes <relPath> per path segment (src/a%20b.ts), but the current helper in scripts/lib/ontology.mjs encodes the whole relPath (src%2Fa%20b.ts). If a new importer follows this ADR, it will mint different file URIs than existing producers and fragment the graph again. Either update the ADR to match the current helper output, or change the helper and migrate callers in the same PR.
There was a problem hiding this comment.
Addressed on this branch: ADR 0003 now documents the monorepo helper whole-string encoding separately from the parent scanner per-segment encoding, so new importers do not mint a third URI shape.
| `AGENTS.md` (§ "Query gotchas #4"). | ||
| - `<pkgName>` is **the source's natural package handle in the producer's | ||
| scope**, not always a public npm name: | ||
| - For DKG monorepo packages: the workspace package name without scope |
There was a problem hiding this comment.
🔴 Bug: Declaring the monorepo canonical <pkgName> as the unscoped workspace name (dkg-storage, dkg-agent, dkg-cli) conflicts with the URIs produced today. Existing importers/seeds pass the full package name from package.json (for example @origintrail-official/dkg-storage) into Code.uri.package/file. Treating the unscoped form as canonical would create a second namespace for the same packages/files and break joins with already-indexed data. Please keep the current scoped name here or document a concrete migration path in the same change.
There was a problem hiding this comment.
Addressed on this branch: the monorepo canonical package handle now stays the scoped package.json name, with examples using @origintrail-official/dkg-storage, @origintrail-official/dkg-agent, and @origintrail-official/dkg-cli.
|
|
||
| ```sparql | ||
| SELECT ?file ?path WHERE { | ||
| ?file owl:sameAs* ?canonical . |
There was a problem hiding this comment.
🟡 Issue: This SPARQL example uses owl:sameAs* without declaring the owl: prefix, so it won't parse if someone copies it into a standard SPARQL client. Add PREFIX owl: <http://www.w3.org/2002/07/owl#> or use the full IRI in the property path.
There was a problem hiding this comment.
Addressed on this branch: the SPARQL example now declares PREFIX owl before using owl:sameAs*.
ADR 0003 made two factual claims about the canonical code-graph URI
shape that did not match what the existing `scripts/lib/ontology.mjs`
helper actually does, plus a SPARQL example that wouldn't parse in a
strict client. Codex flagged all three. Fixed by aligning the ADR to
the deployed reality:
- Encoding section now documents the two encodings actually in use
(monorepo helper does whole-string encodeURIComponent; parent-workspace
scanner does per-segment + literal slashes) and explains why they
cannot collide (disjoint `<pkgName>` slices). Calls out future
unification as a separate ADR rather than an in-passing flip — the
cost of changing the helper now is migrating every `urn:dkg:code:*`
URI already in WM/SWM/VM.
- `<pkgName>` for monorepo packages updated to the scoped
package.json `name` field (`@origintrail-official/dkg-storage`)
matching what `import-code-graph.mjs`, `seed-dkg-code-project.mjs`,
and the other existing importers pass today. The previous ADR text
("unscoped — e.g. `dkg-storage`") would have minted a parallel
namespace.
- §Reconciliation SPARQL example now declares `PREFIX owl:` so it
parses in standard clients.
- Updated the producer-ownership table to show the `<pkgName>` shape
explicitly per producer.
Co-authored-by: Cursor <cursoragent@cursor.com>
Implements the resumable-import pattern from ADR 0002 and gives every
agent / human importer a single canonical reference for chunked bulk
writes against a DKG node.
scripts/lib/manifest.mjs:
- buildInitialManifestTriples / createImportManifest write the import
graph as an RDF assertion in the project's `meta` sub-graph.
- markPartitionStatus appends status events (append-only — no
DELETE/INSERT needed; latest event wins on read).
- loadImportManifest + pendingPartitions resolve the current per-
partition status in one SPARQL round-trip, so an interrupted import
can resume from the next pending partition without re-doing work.
- Itself respects the ADR 0002 chunking contract by reusing
DkgClient.writeAssertion.
scripts/lib/__tests__/manifest.test.mjs:
- Six unit tests covering URI encoding, ontology constants, and the
buildInitialManifestTriples / pendingPartitions pure helpers. The
daemon-roundtrip behaviour stays covered by the repro suite in
scripts/repro/.
packages/cli/skills/dkg-importer/SKILL.md (NEW):
- The agent-readable manual for bulk imports. Codifies the chunking
contract from ADR 0002 (CHUNK=5000, ROOT_CHUNK=1000), gives
worked examples in TypeScript + Python, walks through the manifest
pattern, and links to ADR 0003 for canonical URIs.
- Includes anti-patterns ("don't bump MAX_BODY_BYTES to 1GB",
"don't invent a graphify:* URI scheme"), HTTP error handling,
and a one-page cheat sheet.
packages/cli/skills/dkg-node/SKILL.md:
- One-line cross-reference under the assertion-write tool table
pointing agents to dkg-importer/SKILL.md when they're about to
push >5k quads in one go.
Stacked on docs/importer-adrs (#641). Related to #596, #636, #640.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
||
| ``` | ||
| POST /api/assertion/create { name, subGraphName, contextGraphId } | ||
| POST /api/assertion/:name/write { quads: [...] } ── one or more times |
There was a problem hiding this comment.
🔴 Bug: The example payloads here omit the required contextGraphId on both /api/assertion/:name/write and /api/assertion/:name/promote. Those routes validate contextGraphId today, so an importer that follows this ADR literally will send invalid requests. Include contextGraphId in the write/promote bodies to keep the contract aligned with the actual API.
There was a problem hiding this comment.
Addressed on this branch: the assertion create/write/promote examples now include contextGraphId on every request body, including write and promote.
|
|
||
| 4. **`<relPath>` is the path relative to the package root, not the repo | ||
| root.** A monorepo file at `packages/storage/src/adapters/oxigraph.ts` | ||
| has `<pkgName>=dkg-storage` and `<relPath>=src/adapters/oxigraph.ts`. The |
There was a problem hiding this comment.
🔴 Bug: This example switches back to the unscoped dkg-storage handle, but the ADR just established that monorepo producers must use the scoped package.json name (for example @origintrail-official/dkg-storage), which is also what Code.uri.file() encodes today. Leaving the unscoped form here recreates the parallel namespace this ADR is trying to eliminate. Update this example, and the matching owl:sameAs examples below, to use the scoped package name.
There was a problem hiding this comment.
Addressed on this branch: the examples and owl:sameAs reconciliation targets now use scoped package names so they match Code.uri.file output.
| ```sparql | ||
| PREFIX owl: <http://www.w3.org/2002/07/owl#> | ||
|
|
||
| SELECT ?file ?path WHERE { |
There was a problem hiding this comment.
🔴 Bug: This SPARQL example omits a GRAPH clause even though DKG sub-graph data is stored in named graphs. As written, this query will return zero rows against the real code/decisions data and make the reconciliation pattern look broken. Wrap the triple patterns in GRAPH ?g { ... } (or the specific named graph) so the example is executable in this repo's query model.
There was a problem hiding this comment.
Addressed on this branch: the SPARQL reconciliation example now wraps the triple patterns in GRAPH ?g so it runs against named-graph-backed code/decisions data.
0002 (chunking):
- Show `contextGraphId` (and `subGraphName`) in the write/promote API
examples — the daemon's request validator rejects calls that omit
them with HTTP 400, so an importer following the earlier draft
literally would never get past the first write. Added a paragraph
noting the requirement explicitly.
0003 (ontology):
- §Decisions item 4: use the scoped `@origintrail-official/dkg-storage`
form for `<pkgName>`, matching what `Code.uri.package`/`Code.uri.file`
emit today. Earlier draft showed the unscoped form, which would have
recreated the parallel namespace the ADR aims to eliminate.
- §Reconciliation owl:sameAs examples: use the scoped, percent-encoded
form (`%40origintrail-official/dkg-storage/...`) for the same reason,
with a note explaining the `%40` and pointing at scripts/lib/ontology.mjs.
- §Reconciliation SPARQL example: wrap the pattern in `GRAPH ?g { ... }`.
DKG sub-graph data lives in named graphs; the unwrapped query returns
zero rows, making the reconciliation pattern look broken. Cross-
references AGENTS.md §Query gotchas for context.
Co-authored-by: Cursor <cursoragent@cursor.com>
| ## Context | ||
|
|
||
| Three things in the tree (and one in the next-door tree at | ||
| `/Users/aleatoric/dev/.dkg/scripts/scan-code.mjs`) currently produce RDF |
There was a problem hiding this comment.
🟡 Issue: This ADR hard-codes an author-local absolute path (/Users/aleatoric/...) as the reference for the parent scanner. That path is not portable and will be meaningless to other contributors reading the doc. Replace it with a repo-relative or generic description of the external scanner location.
There was a problem hiding this comment.
Addressed in 592c6cc: replaced the author-local absolute path with a portable description of the shared workspace scanner at .dkg/scripts/scan-code.mjs outside this repo.
|
|
||
| The canonical pattern is a small RDF manifest assertion that the importer | ||
| maintains as it works (specified separately in | ||
| [`packages/cli/skills/dkg-importer/SKILL.md`](../../packages/cli/skills/dkg-importer/SKILL.md) |
There was a problem hiding this comment.
🟡 Issue: This ADR links to packages/cli/skills/dkg-importer/SKILL.md, but that file is not present in this PR or the current tree, and the next line also references scripts/lib/manifest.mjs, which is missing too. Either land the referenced artifacts in the same change or rewrite this section as future work without a live relative link so readers do not hit dead references.
There was a problem hiding this comment.
Addressed in 592c6cc: rewrote the importer skill and manifest helper references as future artifacts without a live relative link to files that are not in this PR.
Implements the resumable-import pattern from ADR 0002 and gives every
agent / human importer a single canonical reference for chunked bulk
writes against a DKG node.
scripts/lib/manifest.mjs:
- buildInitialManifestTriples / createImportManifest write the import
graph as an RDF assertion in the project's `meta` sub-graph.
- markPartitionStatus appends status events (append-only — no
DELETE/INSERT needed; latest event wins on read).
- loadImportManifest + pendingPartitions resolve the current per-
partition status in one SPARQL round-trip, so an interrupted import
can resume from the next pending partition without re-doing work.
- Itself respects the ADR 0002 chunking contract by reusing
DkgClient.writeAssertion.
scripts/lib/__tests__/manifest.test.mjs:
- Six unit tests covering URI encoding, ontology constants, and the
buildInitialManifestTriples / pendingPartitions pure helpers. The
daemon-roundtrip behaviour stays covered by the repro suite in
scripts/repro/.
packages/cli/skills/dkg-importer/SKILL.md (NEW):
- The agent-readable manual for bulk imports. Codifies the chunking
contract from ADR 0002 (CHUNK=5000, ROOT_CHUNK=1000), gives
worked examples in TypeScript + Python, walks through the manifest
pattern, and links to ADR 0003 for canonical URIs.
- Includes anti-patterns ("don't bump MAX_BODY_BYTES to 1GB",
"don't invent a graphify:* URI scheme"), HTTP error handling,
and a one-page cheat sheet.
packages/cli/skills/dkg-node/SKILL.md:
- One-line cross-reference under the assertion-write tool table
pointing agents to dkg-importer/SKILL.md when they're about to
push >5k quads in one go.
Stacked on docs/importer-adrs (#641). Related to #596, #636, #640.
Co-authored-by: Cursor <cursoragent@cursor.com>
…mport Adds a short "Empirical motivation" section between the Consequences and Rejected alternatives lists, with concrete data from importing this repo's Graphify output (26,960 nodes / 63,003 edges) on rc.10: - Hand-rolled importer from dkg-node SKILL.md alone: ~330 LOC, half of which is cap-discovery ceremony. - Same workload using rc.10's existing scripts/lib/dkg-daemon.mjs DkgClient: ~half the LOC, gaps remaining are the ones #642 and #643 close. - Four of 17 partitions exceeded the 10 MB gossip cap; total wall- clock in halve-and-retry: ~15 minutes on top of legitimate work. - Write latency degrades ~200x from baseline once SWM grows past ~100k triples; that cost is what motivates #643 but requires the chunking discipline this ADR formalises. Keeps the headlines so the contract's "why these numbers" question has a concrete answer for reviewers; the full per-partition data lives in the rc.10 test artifacts referenced in the PR description. Co-authored-by: Cursor <cursoragent@cursor.com>
Implements the resumable-import pattern from ADR 0002 and gives every
agent / human importer a single canonical reference for chunked bulk
writes against a DKG node.
scripts/lib/manifest.mjs:
- buildInitialManifestTriples / createImportManifest write the import
graph as an RDF assertion in the project's `meta` sub-graph.
- markPartitionStatus appends status events (append-only — no
DELETE/INSERT needed; latest event wins on read).
- loadImportManifest + pendingPartitions resolve the current per-
partition status in one SPARQL round-trip, so an interrupted import
can resume from the next pending partition without re-doing work.
- Itself respects the ADR 0002 chunking contract by reusing
DkgClient.writeAssertion.
scripts/lib/__tests__/manifest.test.mjs:
- Six unit tests covering URI encoding, ontology constants, and the
buildInitialManifestTriples / pendingPartitions pure helpers. The
daemon-roundtrip behaviour stays covered by the repro suite in
scripts/repro/.
packages/cli/skills/dkg-importer/SKILL.md (NEW):
- The agent-readable manual for bulk imports. Codifies the chunking
contract from ADR 0002 (CHUNK=5000, ROOT_CHUNK=1000), gives
worked examples in TypeScript + Python, walks through the manifest
pattern, and links to ADR 0003 for canonical URIs.
- Includes anti-patterns ("don't bump MAX_BODY_BYTES to 1GB",
"don't invent a graphify:* URI scheme"), HTTP error handling,
and a one-page cheat sheet.
packages/cli/skills/dkg-node/SKILL.md:
- One-line cross-reference under the assertion-write tool table
pointing agents to dkg-importer/SKILL.md when they're about to
push >5k quads in one go.
Stacked on docs/importer-adrs (#641). Related to #596, #636, #640.
Co-authored-by: Cursor <cursoragent@cursor.com>
…ivation Brings in: - #641: round-3 codex (portable scanner path; importer skill/manifest framed as future work) + empirical motivation subsection in ADR 0002 (~330 LOC hand-rolled importer, ~15 min wall-clock in halve-and-retry). - #642: round-3 codex (createImportManifest idempotency on 'already exists'; timestamp tie-breaker on equal-ms; loud failure on empty manifest bindings) + new §1.1 in dkg-importer/SKILL.md documenting the three hard caps and their verbatim error strings (10 MB /write body, 256 KB /promote body, 10 MB gossip). Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # packages/cli/skills/dkg-importer/SKILL.md # scripts/lib/__tests__/manifest.test.mjs # scripts/lib/manifest.mjs
Adds a "What changed in round 3" section to INTEGRATION_NOTES_GRAPHIFY.md summarising the per-PR updates folded into integration after running the stack against a real Graphify codebase import on rc.10: - #641: empirical motivation in ADR 0002 + portable scanner ref + dead-link cleanup - #642: §1.1 caps reference with verbatim error strings + manifest hardening (idempotent create, ms tie-breaker, loud empty bindings) - #643: SMALL_BODY_BYTES correction + full assertion identity for uniqueness + lease-expiry shutdown + §10 empirical appendix Notes that rc.10 import test artifacts are kept local, not committed. Co-authored-by: Cursor <cursoragent@cursor.com>
Implements the resumable-import pattern from ADR 0002 and gives every
agent / human importer a single canonical reference for chunked bulk
writes against a DKG node.
scripts/lib/manifest.mjs:
- buildInitialManifestTriples / createImportManifest write the import
graph as an RDF assertion in the project's `meta` sub-graph.
- markPartitionStatus appends status events (append-only — no
DELETE/INSERT needed; latest event wins on read).
- loadImportManifest + pendingPartitions resolve the current per-
partition status in one SPARQL round-trip, so an interrupted import
can resume from the next pending partition without re-doing work.
- Itself respects the ADR 0002 chunking contract by reusing
DkgClient.writeAssertion.
scripts/lib/__tests__/manifest.test.mjs:
- Six unit tests covering URI encoding, ontology constants, and the
buildInitialManifestTriples / pendingPartitions pure helpers. The
daemon-roundtrip behaviour stays covered by the repro suite in
scripts/repro/.
packages/cli/skills/dkg-importer/SKILL.md (NEW):
- The agent-readable manual for bulk imports. Codifies the chunking
contract from ADR 0002 (CHUNK=5000, ROOT_CHUNK=1000), gives
worked examples in TypeScript + Python, walks through the manifest
pattern, and links to ADR 0003 for canonical URIs.
- Includes anti-patterns ("don't bump MAX_BODY_BYTES to 1GB",
"don't invent a graphify:* URI scheme"), HTTP error handling,
and a one-page cheat sheet.
packages/cli/skills/dkg-node/SKILL.md:
- One-line cross-reference under the assertion-write tool table
pointing agents to dkg-importer/SKILL.md when they're about to
push >5k quads in one go.
Stacked on docs/importer-adrs (#641). Related to #596, #636, #640.
Co-authored-by: Cursor <cursoragent@cursor.com>
1 participant
Summary
Two ADRs to close out the architectural questions surfaced by the Graphify experiment (PR #602):
ADR 0002 — Importer chunking contract: Clients chunk, daemon publishes the limits. Pins
CHUNK=5000quads/write +ROOT_CHUNK=1000URIs/promote as the de-facto budgets importers should use. Documents HTTP 413 as a normal recoverable error, and explicitly rejects a server-side `/api/import/bulk` endpoint (would hide failure modes, duplicate manifest logic, defeat resumability).ADR 0003 — Code-graph ontology convergence: Canonical URI shapes for `urn:dkg:code:{package,file,module}:` + producer-scope rules so monorepo scans, external-repo scans, and future importers don't fragment the graph. Deprecates the `graphify:` namespace introduced in PR docs: document Graphify codebase import experiment #602 in favour of `urn:dkg:code:*` + `owl:sameAs` reconciliation for graphs that have already promoted the legacy URIs.
Both ADRs cite the canonical helpers in `scripts/lib/ontology.mjs` and cross-reference each other. They unblock the importer SKILL.md and manifest library in a follow-up PR.
Test plan
Related: #596, #636 (bug report), #640 (WM persistence fix).
Made with Cursor
Update — round 3 (2026-05-25): empirical motivation
Adds a short Empirical motivation subsection to ADR 0002 with concrete numbers from importing this repository's Graphify output (26,960 nodes / 63,003 edges) on rc.10:
dkg-node/SKILL.mdalone needed ~330 LOC, roughly half of which is cap-discovery ceremony (413 halving, 500-gossip switch to per-root batching, hand-rolled wallet-scoped CG ids, hand-rolled JSON resume state).scripts/lib/dkg-daemon.mjsDkgClientcuts the importer to ~half the LOC; the gaps that remain are the ones PR feat(importer): manifest library + dkg-importer SKILL.md #642 and PR docs(specs): RFC for async WM→SWM promote queue #643 close.entities: "all". Total wall-clock spent in halve-and-retry: ~15 minutes on top of legitimate write/promote work.Full per-partition data lives in the rc.10 test artifacts; this ADR keeps the headline numbers so reviewers see "why these constants" up front.
Also addresses the two remaining round-3 codex line comments inline: portable scanner reference (replaces the author-local absolute path) and dead-link cleanup for the
dkg-importer/SKILL.md/manifest.mjsreferences (now framed as future implementation work, since they ship in the dependent PR #642).