Skip to content

v10.0.19

Choose a tag to compare

@branarakic-agent branarakic-agent released this 28 Sep 12:53
· 2928 commits to main since this release
v10.0.19
42d7a92

DKG v10.0.19

Cores keep what they acknowledge, and stay responsive on large graphs.

DKG v10.0.19 turns the StorageACK into a finality gate: a Core signs a public StorageACK only when the acknowledged data is guaranteed to reach its Verifiable Memory, and it keeps its copy until then. Cores that had switched off the peer-sync reconciler promote acknowledged data again, and graphs acknowledged while promotion was off are backfilled. Several multi-second main-thread stalls on large mainnet Cores are gone. Chain log reads now fit each RPC provider's limits, and Knowledge Asset write routes validate their input.

No smart-contract, ABI or deployment-registry changes are required. Update through the normal npm path, but read Upgrading from 10.0.18 first: several changes affect Core operators.

npm: @origintrail-official/dkg@10.0.19 · dist-tags latest, testnet, mainnet

Highlights

The StorageACK is a finality gate. A Core verifies the data before it signs a public StorageACK. It also checks that its chain-driven VM reconciliation is running and records the graph as core-hosted. Only then does it store the copy and sign. A Core that cannot promote declines with a clear reason, and publishers retry transient declines. Public updates get the same guarantee through a new pending-update lane.

Acknowledged data reaches Verifiable Memory again. Since 10.0.14, switching off the peer-sync reconciler also stopped VM promotion on Cores. VM reconciliation now has its own switch (vmReconcilerEnabled, default on). An ACK promotion audit backfills graphs acknowledged while promotion was off. Acknowledged copies are kept past the SWM TTL until they are promoted.

No more multi-second stalls on large Cores:

  • Looking up a Knowledge Asset's Context Graph now reads one indexed row instead of every registration, removing 8–11 s stalls after finalizations and repairs.
  • Walking a large graph's registrations takes well under a second instead of minutes.
  • RFC-64 catalog replay costs about 2.35 s of CPU per connecting peer instead of 18.7 s.

Chain reads that fit the provider. Every log scanner splits ranges to each RPC provider's eth_getLogs span cap and remembers it, so a Base mainnet node that was down for over an hour catches up on the default public RPC set. Requests that never left the node's own RPC queue no longer count as provider failures. RPC URLs, and any API keys in them, no longer reach logs or error messages.

Safer inputs and status. Knowledge Asset write routes return 400 for malformed terms instead of storing the wrong triple. Catch-up status and public node status no longer name private Context Graphs to other callers. /api/status reports VM-promotion state and main-thread stalls (eventLoopDelay).

Upgrading from 10.0.18

Change Impact Action
Knowledge Asset write routes answer 400 for a malformed term POST /api/knowledge-assets and POST /api/knowledge-assets/{name}/wm/write now reject relative or RFC 3987-invalid IRIs, unquoted or malformed literals and a graph that is not a bare absolute IRI. Before, such a write failed with a store error or stored the triple under a different term than the caller sent. Send absolute IRIs, blank nodes and well-formed quoted literals. Check agents, scripts and MCP tools that build terms by string concatenation, and treat a 400 from these routes as a client error to fix, not a transient one.
A publishing Core counts its own StorageACK toward the Core quorum The publishing Core stores and verifies its copy through the same handler as remote Cores before signing. With the mainnet minimum of three distinct Core identities, a Core publish needs its own ACK plus two remote Core ACKs; an Edge publish still needs three remote Core ACKs. Edge peers cannot serve StorageACKs and are excluded from ACK collection Upgrade Cores one at a time and check that each registers the StorageACK handler. Monitor ACK quorum and decline logs during the canary rollout
Chain-driven VM reconciliation has its own switch, vmReconcilerEnabled (env DKG_VM_RECONCILER_ENABLED, default on) syncReconcilerEnabled: false / DKG_SYNC_RECONCILER_ENABLED=0 now only stops the periodic peer-sync reconciler. A Core that set it resumes promoting acknowledged public data to Verifiable Memory on upgrade None to restore promotion. Keep syncReconcilerEnabled as it is if you still want the peer-sync reconciler off. Do not set vmReconcilerEnabled: false on a Core
Cores with VM reconcile disabled will decline all storage ACKs for public Context Graphs A Core signs a public StorageACK (publish or update) only when the data is guaranteed to reach its Verifiable Memory. With VM reconciliation off it declines every one with CORE_VM_PROMOTION_DISABLED, which publishers treat as final for that Core. While it cannot commit yet (starting up, chain or store unavailable, an older version of the asset still awaiting promotion) it declines with CORE_TEMPORARILY_UNAVAILABLE and a message starting VM promotion unavailable:, which every deployed publisher, 10.0.18 included, retries. Curated catalog ACKs are unaffected After the upgrade check GET /api/status: vmPromotion.storageAckGate is "ready" and vmPromotion.storageAckHandler is "registered"; vmPromotion.storageAckDeclinesLastHour counts declines per reason. The startup log warns when a Core cannot promote to VM
Cores decline legacy (not graph-scoped) public ACK requests A Core cannot keep a copy of a legacy root-entity publish that it could later promote, so it declines those requests with CORE_VM_PROMOTION_DISABLED. Every default publish and update path (API, CLI, MCP, adapters) has sent graph-scoped requests since 10.0.7. Still legacy: raw-lift jobs queued before 10.0.7 or restored with the legacy raw-lift import, and publishFromSharedMemory called without contentScopeVersion. These can no longer collect ACKs from upgraded Cores Run publishers on 10.0.7 or later. Finish or discard legacy raw-lift jobs before the Cores upgrade, and publish that data again through the current publish API
Graphs acknowledged while VM reconcile was off are backfilled From 1 to 6 minutes after start, then every 15 minutes, the Core records the public graphs holding the StorageACK copies it signed as core-hosted, at most 32 per pass. The reconciler then walks each graph's chain registrations and promotes the Knowledge Assets the chain registered to it, including ones this Core never held and fetches from peers. On a Core that ran with the peer-sync reconciler off, this can be a large catch-up. Copies of publishes that never landed are not promoted; they expire as described under Fixed Before upgrading, check with the node-operator token that GET /api/context-graph/subscriptions reports subscription rehydration enabled and no dormant core-hosted rows under rehydration.dormantReasons (a dormant row's graph is declined). Upgrade Cores one at a time, and start the next only when the previous one reports vmPromotion.audit.backfillPending 0 and staleUnpromotedCopies near 0 (watch the ACK promotion audit log line). The catch-up runs in the background RPC class and store lane; DKG_VM_RECONCILE_CONCURRENCY (graphs at once, default 2) and DKG_VM_RECONCILE_ORDINAL_CONCURRENCY (registrations per graph at once, default 5) throttle it further
Mode-only replenishing approval configs use a relative ceiling When chain.approvalPolicy.mode is replenishing and neither sizing field is set, the implicit ceiling changes from a flat 1000 TRAC to 20 times the triggering publish cost. The same ceiling sizes the Context Graph registration deposit approval: when a registration reverts for too low an allowance, the node now approves the ContextGraphs contract for 20 times the on-chain registration deposit, where it approved 1000 TRAC (or the deposit, when larger). This can lower or raise the standing allowance depending on publish cost and the deposit; startup emits a warning for this exact legacy shape Set targetAllowance: '1000000000000000000000' to retain the former flat 1000 TRAC ceiling for publishes and the registration deposit (a larger deposit is still approved in full), or set targetAllowanceMultiple explicitly to adopt relative sizing
Curated and local-only Context Graph metadata leaves ontology, which a rollback to 10.0.18 does not undo On start, and before each store discovery pass, a node moves the on-chain id binding, definition and name of each curated or private: true graph that earlier builds wrote to the shared ontology graph into the graph's own _meta, and no longer writes or gossips a curated graph's binding to ontology. Such rows for graphs the node does not hold are deleted, a bare binding only once the chain proves its slot curated. The VM publish registration guard of 10.0.18 reads a binding from ontology only: it passes a graph whose _meta records it as registered, as the registering node's does, or that has an ontology binding. So on a node rolled back to 10.0.18 after running 10.0.19, publishing into a curated graph with an open publish policy that the node did not register can fail the guard with CG_NOT_REGISTERED (... is not registered on-chain). Nothing on 10.0.18 restores the row: its chain discovery writes a curated binding only on the graph's curator, registration refuses a registered graph, and upgraded peers no longer carry the row, so only ontology sync from a peer still on 10.0.18 can bring it back Do not roll back a node that publishes into curated graphs it did not register. The error's advice to run dkg context-graph register does not help there. If one was rolled back, upgrade it to 10.0.19 again: the binding is still in the graph's _meta, which 10.0.19 reads, so publishing resumes after the restart without a resync

Known issues

  • An on-demand subscription can stay partly synced: after
    dkg subscribe <id> without --save (the CLI default since 10.0.13), the
    VM reconciler fails to save the subscription's cursor on every sweep and
    logs Cannot acknowledge join approval ... durable subscription intent or host state is missing, so Knowledge Assets the initial fetch missed are
    never fetched. Subscribe with --save to avoid it. A Core that hosts a
    graph that is also subscribed on demand is affected the same way. A fix is
    in review.
  • A node that holds the shared ontology system graph can bind on-chain id
    claims its own chain does not confirm
    : dkg context-graph list can show
    one on-chain id under several graph names, and subscribing to a public
    Context Graph by its name hash can create a row keyed by the hash, whose
    Verifiable Memory fetches then come back empty. Subscribing by the graph's
    cleartext id avoids the name-hash path. A fix is in review.

What's new

Changed

  • Route-based request deadlines in clients (#2768). The CLI, the MCP server and both adapters wait 30 s for reads and 240 s for long Knowledge Asset mutations. A long mutation that times out is reported as outcome unknown, not failed: check dkg ka history before you retry. DKG_API_READ_TIMEOUT_MS and DKG_API_LONG_TIMEOUT_MS override the deadlines.
  • The StorageACK is a finality gate, and public update ACKs are durable too. See the highlights and the upgrade table.
  • Curated catalog ACKs guarantee that a verified catalog commitment is stored in <cg>/_catalog, which is what random sampling proves for curated Knowledge Assets.
  • The VM reconcile walk yields to foreground work: it runs in the background RPC class and store lane.

Added

  • ACK promotion audit on Cores. It records public graphs with signed ACKs as core-hosted and promotes ledgered copies still not in VM. It reports VM promotion watchdog: ... and gauges for stalled ACKs, backfills and retries.
  • GET /api/status:
    • syncLifecycle.vmReconcilerEnabled;
    • a vmPromotion block (storageAckGate, storageAckHandler, storageAckDeclinesLastHour, core-hosted graphs, last audit);
    • an eventLoopDelay block, with an Event loop blocked warning when a stall reaches 2 s.

Fixed

  • Cores read the chain far less while catching up on VM promotion: local state is checked first.
  • A request that never left the node's RPC queue is no longer counted as a provider failure (RPC_REQUEST_GOVERNOR_QUEUE_FULL).
  • One damaged shared-memory head no longer stops Verifiable Memory promotion for a whole Context Graph.
  • An Edge can publish to Verifiable Memory in a public Context Graph that another node registered.
  • Much less CPU per connecting peer for RFC-64 catalog replay.
  • Subscriptions for a Context Graph the chain reports unknown are no longer retried every 30 seconds.
  • Remote queries no longer pass the per-graph access check with a graph id named like a built-in object property.
  • Knowledge Asset write routes answer 400 for a malformed term.
  • The SWM TTL no longer deletes an acknowledged copy before promotion, and an ACK or share can no longer overwrite a copy the Core still owes to VM.
  • dkg knowledge-asset create/write --input-file keeps literal values intact.
  • Random Sampling resolves the challenged Context Graph by its chain name commitment, and proof repair fetches from peers again (#2741).
  • Chain log reads fit each RPC provider's eth_getLogs limits, and RPC read errors no longer quote a configured RPC URL.
  • A curated StorageACK stores only a genuine catalog (CATALOG_ROOT_MISMATCH otherwise).
  • dkg status no longer reports a healthy store as UNREACHABLE on a cold daemon.
  • The API opens promptly after a restart on a node with many catalogued Context Graphs.
  • Catch-up status and the sync scope in public node status no longer name private Context Graphs to other callers.
  • Looking up a Knowledge Asset's Context Graph, and walking a graph's registrations, no longer block the node for seconds.

The full entries are in CHANGELOG.md.

Full Changelog: v10.0.18...v10.0.19