Repository navigation
v10.0.24
DKG v10.0.24
A patch for queued publications into private Context Graphs. A queued
publication into a private graph stayed non-terminal for minutes after its
transaction was final. It waited for the placement of its asset in the
author's catalog, behind the placements before it, and each placement read
and verified the whole catalog three times and told peers about the new head
while it held the catalog lock. The publication is now terminal once
finality is observed, the verified content is stored and the placement is
recorded and requested. A placement no longer reads and verifies the whole
catalog for each asset, and peers are told about a new head outside the
catalog change. A publication also no longer waits, behind work of other
graphs, for the retirement of the marker its share left. A new asset for a
full author catalog is refused once and reported; its placement was retried
for ever. After a start, a chain index that could not start is started
again, a graph whose reconcile pass was refused or overtaken is asked again
within seconds, and the checks of saved subscriptions no longer take the
node's whole chain request budget. Reads that prove a graph's registration
state no longer keep a transaction waiting until its deadline. A read made
with an agent token is authorized against the Context Graph it asks about,
and a node serves its ontology graph only once every binding in it is
classified. Random-sampling proof and challenge transactions are sent with
more gas headroom, a vector search over Working Memory covers the asking
agent's own entries, and a member whose metadata fetch failed right after a
live approval no longer needs a restart, which fixes a 10.0.23 known issue
(#3109).
This release also fixes what a community report found between Edge nodes
(#3198). An asset whose text holds a tab no longer blocks its author's
catalog, and a node on Blazegraph accepts catalog rows with dates and
numbers. A share into a sub-graph reaches the peers that follow a public
graph through a relay, and a request to such a peer no longer fails when its
dial ends in a closed connection. A new subscription to a public graph asks
every peer it selected, gets the graph's definition and reports synced
within seconds. An Edge keeps its relay reservations on the relays its
network lists.
An Edge node can now hand its EVM signing to an external signer, so that it
holds no operational private keys; this is off by default (#3177).
No smart-contract, ABI or deployment registry changes. Message formats
are unchanged; a node now announces only the newest head of a catalog
(#3130) and runs the libp2p identify push protocol (#3145). A catalog
projection may now hold a literal with a tab or another control character;
a node on an earlier version does not apply a catalog head with such a row
(#3209).
npm: @origintrail-official/dkg@10.0.24 · dist-tags latest, testnet, mainnet
Upgrading from 10.0.23
| Change | Impact | Action |
|---|---|---|
| A queued publication into a private Context Graph is terminal before its catalog placement has run (#3127) | The job is terminal once finality is observed, the verified content is stored and the placement is recorded and requested. The author's catalog can list the asset later. A synchronous publish no longer waits for a placement attempt either. Placements that are still owed are counted as pending, with oldestPendingAgeMs, under rfc64Catalog.finalizedPrivatePlacementQueue in GET /api/status |
None. Read pending to see whether placements are still owed |
| A new catalog head is delivered after the catalog change, and only to peers the node's own policy admits (#3130) | The warn line RFC-64 catalog head announce failed for F/T peer(s) keeps its text, but T counts eligible peers only, and a head with no eligible peer writes nothing at warn level. Peers that the policy check could not answer for get a warn line of their own, RFC-64 catalog head delivery could not check U/T peer(s) .... rfc64Catalog.catalogHeadDelivery in GET /api/status counts hand-offs and peers by outcome since the node started |
None. A refused peer no longer counts as a failed one. uncheckedPeers growing while deliveredPeers stands still means the policy check answers for nobody |
| A new asset for a full author catalog is refused once, parked and reported (#3143) | The asset is published and finalized as before. Its placement is no longer retried at intervals of up to 60 s. The node writes a catalog_full warning when a catalog first refuses and then at most one an hour, for a refused placement and for a refused share-time projection each on its own, and counts scopesAtCap and parkedPlacements under rfc64Catalog.authorCatalogCapacity in GET /api/status. catalog_private_repair_failed and catalog_repair_failed are not written for this reason. A parked placement still counts as owed in finalizedPrivatePlacementQueue |
Watch authorCatalogCapacity. The change adds no capacity; see Known issues |
| A node keeps the verified state of its author catalogs in memory (#3133) | At most 64 catalog scopes (one author in one graph) and 96 MiB in all, least recently used first out. Nothing is persisted. The first placement into a catalog after a start reads and verifies the whole catalog, as every placement did | None. DKG_RFC64_CATALOG_MUTATION_MEMORY=0 switches the memory off; see docs/use-dkg/run-node.md |
| The wait of a queued publication after finality is logged and shown in status (#3120, #3136) | Two new log lines: async_publish_post_finality, one per finalized queued job, at info when the wait after finality is 10 s or more and at debug otherwise, and rfc64_catalog_placement_wait for a catalog placement that takes 5 s or more. New block rfc64Catalog.finalizedPrivatePlacementQueue in GET /api/status. The execution, recovery and placement lines of one job carry [from:<jobId>] |
None. The lines and the block are additive |
| A vector search over Working Memory covers the asking agent's own entries (#3150) | The local vector store is migrated at start, and its existing rows are kept. Working Memory embeddings written before the upgrade have no recorded writer and no longer appear in an agent's vector search. Shared Working Memory and Verifiable Memory keep their graph-wide scope | None. A search by the node operator still reads the embeddings that have no recorded writer |
| Random-sampling transactions are signed with a higher gas limit (#3117, #3122) | A proof is signed with its gas estimate plus 50 %, where it had no headroom, and a challenge creation with its estimate plus the larger of 50 % and 2,500,000 gas, where it had 50 %. Unused gas is refunded, but the wallet must hold the limit times the fee cap before the transaction is accepted: about 2.83M gas for a challenge creation, where it was 0.49M. A creation that still runs out of gas burns its whole limit | None for a funded wallet. The wallet that signs random-sampling transactions must be able to pay the higher limit up front |
| A chain index start that cannot complete for now is tried again (#3116) | The start is repeated after 5 s and then at doubling waits of up to 60 s, until the index attaches. The node logs [chain] one-log chain index start deferred (retrying, next attempt in 5s): <error> once, says every 30 minutes that the start is still deferred, with the latest failure, and logs once more when the index comes up. A line that begins [chain] one-log chain index disabled: is still written for a failure another attempt cannot change |
None. A node that logged the disabled line for a transient failure no longer needs a restart to get its index |
| A Verifiable Memory reconcile pass that was refused locally or overtaken is asked again (#3123, #3135) | Two new info lines: VM reconcile for "<graph>" is waiting for local RPC admission; asking again shortly: <the refusal> and VM reconcile for "<graph>" was overtaken by a change to the graph's subscription or binding; asking again shortly. For these two causes the warning VM reconcile for "<graph>" failed; retrying on the periodic sweep: ... is written only once asking again has stopped; a graph is asked again within a window of five minutes |
None. No setting is added |
| The checks of saved subscriptions that the activation cap left dormant are paced after a start (#3119) | On a node with several hundred saved subscriptions that the chain does not confirm, the last one is checked about three quarters of an hour after the start, where it was a few minutes. A graph the chain confirms waits for the rows before it only when its saved row has no chain id and nothing asks for it: 44 minutes in the measured run, where it was 3. One WARN summary when a pass has no row left, and every five minutes while it runs, replaces one WARN per row; the per-row line is written at debug level. A row whose chain id the chain does not know shows as deactivated on GET /api/context-graph/subscriptions at once |
None. No setting is added and no saved row is changed. An explicit subscribe does not wait for these checks |
| A missing asset version view names the chain endpoints that failed (#3099, #3106) | GET /api/status gains chain.versionSnapshot. The recovery deferral, the conflict of an exact-asset fetch (Knowledge Asset <ual> has no coherent on-chain version snapshot: <why>, the HTTP 409 body of the operator asset fetch) and the Verifiable Memory refresh check (no coherent chain view confirms the copy: <why>) end with the cause: each endpoint that gave no view, by its position in the configured list and its host. The remains pending warning is logged once per asset when a reason appears or changes, then as one summary per reason every 5 minutes |
None. The earlier text is the start of each message. A line that begins Operator action needed: names an endpoint to fix or replace |
| A subscribe or unsubscribe that fails on an authority outage returns an operation id (#3151) | The response carries an operation id, and exactly one server log line matches it. The subscription error code, the generic response body and the retry headers are unchanged | None. Use the operation id to find the line in the node's log |
| A slow identity probe is retried once before the peer is suppressed (#3154) | A probe that ends on a transport deadline is retried once with a response deadline of 15 s, where the first deadline is 3 s. Later suppression of that peer is capped at 3 s while it has an open connection. The validation of the signed proof, quarantine and the back-off for disconnected peers are unchanged | None |
| A failed asynchronous share reports the transition that was stored (#3155) | The worker takes the terminal or retrying transition of a failed attempt from the write that stored it, where it read the job's outcome in a separate step that could fail or see a later attempt. Bookkeeping that could not be resolved is reported as uncertain, not as a fatal failure. The operator diagnostics and the documentation (docs/use-dkg/knowledge-asset-lifecycle.md) are updated with it |
None |
The commit of a node built from source can end in -dirty (#3153) |
GET /api/status reports <commit>-dirty for a build from a checkout with local changes or untracked files. A release slot in which the git updater has set stale package copies aside counts as such a checkout (#3169). commitShort carries no suffix. Nodes installed from npm are not affected |
None. Tooling that compares commit between nodes should compare commitShort or strip the suffix |
| The managed Oxigraph store reports its retained write-ahead log; a maintenance reopen is available and off by default (#2571, #3195) | By default nothing changes except new log lines. A managed store (store.backend: oxigraph-server) keeps its write-ahead log for as long as the store process runs. The node measures it once a minute and writes [oxigraph] <size> of write-ahead log is retained and it grows until the store is reopened; the maintenance reopen is off. ... when the log first reaches 4 GiB and again each time it has doubled. The reopen runs only when store.options.walRestartThresholdBytes is set to a positive integer, a size in bytes; any other value is ignored with a log line. With it on, the node refuses all store work once the log reaches that size, waits for the work in progress, reopens the store and then leaves it alone for an hour. In our tests one cycle refused store work for about 75 to 95 seconds, and store-backed API calls answered 503 with outcome: "not_started" meanwhile. Apart from the log: an update that runs past its client deadline (store.options.clientTimeoutMs, 30 s by default) now restarts the managed store, as a query already did, which interrupts the writes in progress on that store |
None. Operators of long-running nodes with a managed store may set the option, or restart the node when the report appears; see docs/use-dkg/storage-sparql-http.md. Retry an interrupted write under the same Knowledge Asset name or idempotency key |
| A read made with an agent token is authorized against the Context Graph it asks about (#3172) | Daemon read routes check a request made with an agent token against the Context Graph it names. A request for a graph that agent may not read answers 403, Not authorized to read this context graph. When the node cannot establish the answer it answers 503 with Retry-After and the code CONTEXT_GRAPH_READ_AUTHORITY_UNAVAILABLE, and the request can be sent again. A listing of publisher jobs made with an agent token names its Context Graph: contextGraphId on GET /api/publisher/jobs, --context-graph on dkg publisher jobs. The node operator's token and a node with authentication disabled read as before |
Integrations that read with agent tokens should expect 403 where a graph is outside the agent's access, and should name the graph when they list publisher jobs |
The ontology graph is served to peers only when every Context Graph binding in it is classified (#3157) |
A node whose ontology graph holds bindings that it cannot classify from its own data does not serve that graph to peers (sync, changelog sync, remote queries) after a start, until later passes have classified them against the chain, at most 16 new bindings a pass. On test nodes that took from one minute to about three hours. A pass that fails or is cut short closes the graph again, and there is no time limit. The node warns at start Not serving the ontology graph to peers until the context graph metadata relocation reaches every candidate. After a pass that left some it logs Context graph metadata relocation is incomplete: <n> deferred, <m> unclassified candidate(s), where it logged Context graph metadata relocation stopped early: .... It logs Context graph metadata relocation reached every candidate; serving the ontology graph to peers again when it serves again. Other graphs are served as before |
None. If the last line does not come, restore the node's chain reads; see docs/use-dkg/run-node.md |
| A publication waits one second at most for the retirement of its Shared Working Memory marker (#3179) | tailLegacySwmRetireMs on the async_publish_post_finality line now shows what the publication waited: the retirement when it ended within a second, about one second otherwise. The retirement goes on after that and is repeated when it did not get its turn. New lines: Legacy SWM boundary retirement <asset> did not get its turn; repeat <n> in <s> s and Legacy SWM boundary retirement <asset> ended on repeat <n>: <outcome> at info, and Legacy SWM boundary retirement <asset> is not repeated any more (<reason>); its marker stays until the asset is published again at warn |
None. No setting is added |
| Reads that prove a graph's registration state take turns with other chain requests (#3194) | In the foreground class of the node's chain request budget, such a read and an ordinary request take turns when both are waiting, and background work may take one turn between them, within its own budget. When only one of the two kinds is waiting, admission is exactly as before. A transaction no longer waits out its 10 s deadline behind such reads. Beside one ordinary read a second, a pass of such reads takes about 10 % longer | None. No setting is added, and the budget itself is unchanged |
| A deadline spent waiting for the node's own request budget is reported as local capacity (#3171) | The line <step> waited <n>ms for local RPC admission and was not sent now reads <step> exhausted its <n>ms deadline waiting for local RPC admission with no endpoint request in flight. It is also written when earlier requests of the same step had been answered; that case read <step> timed out after <n>ms and counted against the endpoint. Its error code is RPC_REQUEST_GOVERNOR_QUEUE_FULL. GET /api/status gains chain.rpcRequestTiming and chain.rpcRequestGovernor |
None. Tooling that matches the old text should match for local RPC admission. To size the budget see docs/use-dkg/rpc-request-budget.md |
| Chain adapter initialization accepts a missing optional Hub contract only when the Hub says it is missing (#3162) | A lookup of an optional contract that fails for another reason (a timeout, a throttled endpoint, an answer the node cannot decode) fails the initialization, and the next caller starts it again. 10.0.23 went on without that contract. The default endpoints of the three main networks answer a missing contract in the form the node accepts | None on the default endpoints. A node on another provider that does not finish its chain initialization after the upgrade needs an endpoint that passes on the Hub's revert data |
| A refused query that only aggregates answers one row (#3181) | A SELECT whose projection holds only aggregate calls, for example SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }, answered no row when the node refused the read, or when the view resolved no graph. It now answers the one row such a query has over no data: 0 for COUNT, SUM and AVG, an empty string for GROUP_CONCAT, unbound for MIN, MAX and SAMPLE. Other queries keep the answer without rows |
None. A client that took "no row" from a count for "refused or empty" should read the value |
| A node runs the libp2p identify push protocol (#3145) | A node announces a change of its protocol list to the peers it is connected to. A requester whose record of a connected peer lacks the sync protocols asks that peer directly before it passes it over: a probe of at most 3 s a protocol that sends no application data. A peer on an earlier version does not take the push and is probed the same way | None |
| A Core that serves the batch stream answers a request it cannot serve with a refusal (#3137) | ASSET_MISSING, DENIED or BUSY on the stream, where the stream was reset. The codes exist in 10.0.23, so a requester on that version reads them. New lines on such a Core: Exact batch responder refusal stage=<stage> code=<code> and Exact batch inbound opened peer=<suffix> at info, and Exact batch inbound failed peer=<suffix> stage=<stage> error=<name> ... at warn, where <suffix> is the last eight characters of the peer id |
None |
| Catalog status keeps a promised head pending (#3161) | phase under rfc64Catalog.contextGraphs[] in GET /api/status reads known-incomplete, where it could read complete, while a head that a peer promised is not applied. expectedCatalogHeadDigest is null when two heads compete at one version |
None. Tooling that waits for complete can wait longer, for a state that was not complete before either |
Every assertion write records an AssertionWritten activity (#3164) |
Seven rows or more for each write in the Context Graph's _meta graph, in the same insert as the payload. Assertion history gains written events; the lifecycle state is unchanged |
None. metadata.provenanceEvents: false leaves them out, as it does the other lifecycle events |
The dashboard shows Node unavailable when its node does not answer (#3160) |
The demo data that filled the dashboard after failed status checks is gone. The view replaces every route until the page is reloaded | None. Use Retry connection once the node answers again |
A cg-shared-v1 catalog projection may hold a literal with a tab or another control character (#3209) |
Such a literal had no form the projection check accepted, so an author that shared an asset with one could not publish its catalog row, nor any later row for that graph's root. A node on an earlier version does not apply a catalog head that contains such a row; it did not receive these rows before either. Merkle leaves and roots are unchanged | Upgrade the nodes that should receive such assets. An author with a stuck row publishes it with its next share into that graph's root; a restart alone does not |
| A receiver accepts a catalog row when its store returns a literal in its own lexical form (#3209) | A node on Blazegraph refused a row with a whole-second xsd:dateTime, an integral xsd:double or xsd:float, or a short xsd:duration, with catalog reconciliation failed ... cause=NQUADS_BYTE_LIMIT_EXCEEDED or exact SWM post-read differs, and the graph stayed at catalog-reconciliation-failed. The read-back is now compared in the canonical term form |
None. A graph that was stuck applies the rows with the next catalog head it receives |
| A share into a public Context Graph also goes to the peers that follow the graph through a relay (#3208) | A peer that asked this node for the graph's shared memory or for its catalog heads is a follower for 30 minutes and is sent each share directly, also over a relayed connection that has at least 16 MiB and 5 minutes of its allowance left. The fan-out line gains followers=<n>; GET /api/peer-info reports allowance for a relayed connection |
None |
| An Edge keeps its relay reservations on the relays its network lists (#3206) | A node with a relay list no longer reserves a slot on a relay outside it, and its circuit fallback tries every listed relay within the time the first four took. New line at info, at most once per peer in 10 minutes: Relay reservation skipped: peer=<suffix> offers a relay but is not in this node's relay set. While a slot is free the node keeps looking for a relay, as it did before whenever it was short |
None for an Edge. An operator of listed relays restarts at most one of a network's listed relays at a time, so that no Edge is short of relays |
| A subscribe asks every peer it selected for the graph's shared memory (#3205) | A catch-up job fetched from three peers and ended the rounds of the others without a request. It now waits for the metadata pool, at most ten seconds a transfer. A new subscription of a public graph that is not registered on chain reports synced within seconds where it stayed unsynced, and its first write is accepted. One job sends more metadata requests than before |
None |
| The peer that served a graph's seed is asked first for its definition (#3212) | For a public graph that is not registered on chain. An empty answer from a peer that does not hold the graph is logged as Peer <suffix> holds no definition of "<graph>" at info, where it read Rejected curator metadata snapshot ... missing an unambiguous public definition at warn |
None. Tooling that counted the warning sees fewer of them |
| A sender's Sender Key epoch for a private graph is replaced once after the upgrade (#3203) | One setup message per recipient, once per graph and sender agent, the first time the node shares into the graph. Shares that overlap get consecutive message indexes | None |
| A catch-up job waits for the graph's policy to become active (#3217) | Once per job and for at most 10 s, only for a graph whose public policy the node has accepted and not yet activated. New line at info for a job that waited: Catch-up for "<graph>": the selected SWM lane became active after <n> ms, or is still not active after <n> ms |
None |
| An Edge node can sign its EVM transactions with an external signer (#3177) | Off by default: without a chain.signer block nothing changes. With chain.signer.type: "remote" the node signs through standard Ethereum JSON-RPC (eth dialect, Web3Signer-compatible, or Clef) over HTTP(S) or a unix socket, verifies every result and retries nothing; it still builds, records and broadcasts every transaction. Author and payer keys can both live in the signer. With a signer, the Core role, publisher.enabled and the mock chain are refused at start, and admin-key operations and dkg set-ask fail with an explicit error. POST /api/agent/register accepts address for an agent whose key the signer holds |
None, unless you want it: see docs/use-dkg/external-signer.md |
| The query-catalog read can be limited to one catalog and answers up to 2 MiB (#3235) | POST /api/profile/query-catalog/read accepts catalogSlug (a nonempty string of at most 512 characters; anything else answers 400 QUERY_CATALOG_INVALID_FILTER) and then reads that catalog only. The serialized response limit is 2 MiB, up from 1 MiB, because the response carries canonical items and legacy bindings together |
None |
Known issues
- RFC-64 catalog replay loop: replays that come back incomplete repeat on
every reconnect. Keeprfc64Catalog.rollout.killSwitchon wherever it is
set. - A node updated from git can report its build as
-dirty(#3169): a
source build is reported as<commit>-dirtywhen its checkout holds
untracked files, and the git updater can leave an untracked folder in
a release slot by itself, when it sets stale package copies aside. No
tracked file differs from the commit in that case, and the updater
keeps working. Nodes installed from npm are not affected. - An author's catalog for a Context Graph holds 1,024 assets (#3134): the
catalog is one signed bucket of at most 1,024 rows. A further new asset
of that author in that graph is published, finalized and stored as
before, but the catalog does not list it. Its placement is refused once
and parked, and the node reports it: acatalog_fullwarning when a
catalog first refuses and then at most one an hour, for the placement
and for the share-time projection each, and
rfc64Catalog.authorCatalogCapacityinGET /api/status. A newer
version of an asset the catalog holds is still placed, and a removal
frees a row. A member on the default configuration keeps receiving such
assets, with a delay: in a test on a public testnet a member of a
private graph with 1,050 assets from one author held all of them, and in
later tests on that graph a further verified asset reached the member
hours after its confirmation. By the receiver's rules, a member that
takes a private graph through catalogs stops receiving an author's
updates once that author's catalog cannot list one of the author's
finalized assets; this is read from the code and was not reproduced.
Capacity beyond one bucket is tracked in #3134. - Under a burst of publications the author's catalog can trail them by
minutes (#3128): a publication no longer waits for its placement, and
placements still run one at a time, in a fixed order. In a test on a
public testnet with the catalog at 700 to 880 rows one placement took 6
to 21 s at the median, and the catalog trailed a burst of 60
publications by up to about five minutes in one run and by under 20 s in
another.pendingandoldestPendingAgeMsunder
rfc64Catalog.finalizedPrivatePlacementQueueinGET /api/statusshow
the backlog. - A finalized publication into a private graph can stay out of the
author's catalog (#3128): a publication is terminal once its placement
is recorded, and in three cases, all older than this release, a
confirmation ends with nothing recorded. The catalog lane cannot be
resolved at the confirmation, which is logged as
RFC-64 catalog authority was unavailable; the graph has no catalog
lane at that moment, which writes no line; or the write of the placement
marker fails, which is logged as
finalized-private placement was not recorded, so nothing owes it. The
asset then reaches the catalog only through the projection that its
share requested, which the node retries when it fails. A fourth case
concerns one version: when an assertion is re-opened for editing twice
while the placement of its published version is still queued, and the
node then restarts, that version cannot be placed. Every attempt writes
catalog_private_repair_failedwithkind: integrityand
stage: seal, at most once a minute, and the placement stays counted
inpendinguntil the catalog holds a later version of the asset. Both
are tracked in #3128. - A head replay to a peer that connects during a burst of publications
can be rejected (#3129): a replay is rejected when a catalog change
overlaps it, before its snapshot or while it is sent, and during a burst
that is most replays. The requester asks again. Such a peer converges
when the burst pauses, through a later replay or the next delivered
head. - A queued publication can still wait several seconds after finality on a
node with peers connected (#3128): the wait in the retirement of the
asset's marker is gone (#3179). In the test of that change on a public
testnet, what was left was the clear of the published asset's shared
graph, 0.6 to 1.5 s at the median and about 10 s at the most, the
lifecycle stamp, up to 2.9 s, and the admission of the recovery turn, up
to 5.8 s. With no peer connected a publication waited 0.9 s at the median
after finality. The fieldstailPublishedGraphClearMs,
tailLifecycleStampMsandrecoveryAdmissionMsof the
async_publish_post_finalityline show them (#3136). There is no fix yet. - After a start a node may not serve its
ontologygraph for hours
(#3157): a node whoseontologygraph holds Context Graph bindings that
it cannot classify from its own data keeps the graph from peers until it
has classified them against the chain, at most 16 new bindings a pass. On
test nodes that took from one minute to about three hours after a start,
longest on a node whose store was under pressure. There is no time limit:
a node whose chain reads keep failing does not serve the graph. A peer
that asks is told to try again later, and other graphs are served as
before. - Under load a subscriber of a public Context Graph may hold a newly
verified asset later than before (#3179): a publication no longer waits
for the retirement of its marker, so a publish job ends sooner. In tests
under load on a public testnet a subscriber held newly verified assets
later, counted from their chain confirmation, than without the change. The
assets arrived in every run. - The write-ahead log of a managed Oxigraph store grows until the store is
reopened (#2395): the maintenance reopen is off by default, so only a
restart of the node or a recovery of the store truncates the log. In our
tests the log grew by 0.25 to 0.6 GiB an hour on nodes with a light write
load, and by 2.9 and 4.9 GiB an hour on two nodes under a mixed publish
and sync load. A large log takes disk space and lengthens the next start,
which replays it. The node reports the log from 4 GiB on. Restart the node
at a time of your choosing, or set
store.options.walRestartThresholdBytesand accept about a minute and a
half of refused store work for each cycle. - A subscriber of a public Context Graph can receive a new asset minutes
late (#3052): while a node applies a catalog update for a graph it
refuses Shared Working Memory writes for that graph. A sync round that
overlaps the update writes none of the snapshots it fetched, and the node
does not pull that graph again at once: the assets arrive with a later
round or through the catalog, in our tests 6 to 7 minutes after they were
shared. An asset that was finalized in the meantime is then fetched on its
own, and those fetches can fail and be retried minutes apart: on a graph
of a few hundred assets that was being published to, single assets took up
to 15 minutes. Nothing is lost, and this is not new in this release. - A member that restarted before its empty private graph was proven ready
stays not ready (#3109): the readiness proof of an empty registered
private graph now runs again when the member's copy of the metadata
becomes authoritative (#3108). A node that restarted after that moment and
before the proof succeeded keeps the graph subscribed and not writable
until the member subscribes again or content arrives. Members that reached
this state on an earlier version are in the same position after the
upgrade. Subscribe again. - A read by a newly approved member of an unregistered private graph can
answer a retryable 503 (#3064): before it reads such a graph, the
member's node has to accept a proof that the graph's name is not
registered on chain. One slow or discarded run of that proof no longer
fails a read (#3074), but in the seconds after an approval a read can
still answer 503 withCONTEXT_GRAPH_READ_AUTHORITY_UNAVAILABLE. Send the
read again. - A member of private graphs that are not registered on chain reads the
chain much more often than other nodes (#3064, #3184): the node proves
very often that the name of such a graph is not registered: one periodic
refresh asks the chain about every such graph on each pass. These reads no
longer keep a transaction waiting until its deadline, because such a read
and an ordinary request now take turns when both are waiting (#3194), and
a read that accepts bounded freshness no longer adds scans of its own
(#3183). The reads themselves are as frequent as before. They still use a
large part of the node's chain request budget, up to every other permit
when ordinary requests are waiting, so other chain work on that node is
slower. - Recovery through the batch stream depends on the Cores that serve it:
only Cores with the switch set serve the stream. A Core that is busy
answersBUSYand is asked again after a short pause, and a stream that
breaks is opened once more with the same Core (#2979, #2987); a stream that
stalls without closing is not detected. After a busy attempt, and after a
broken one on a Core that has already served the graph, the Core keeps its
turn, but between two stream attempts the node can probe peers without the
stream. Each such probe is bounded to two minutes while a stream-capable
Core is connected, three times out of four (#2978), and large assets still
take minutes each there. Serve the stream from more than one Core that
holds the graph. - The catalog of an unregistered private graph can stay unverified on a
member (#2993): after the approval, data and reads reach a member Core
within seconds and a member Edge after about a minute, but the member's
catalog status reachescompleteonly after about eight minutes on a
Core and may not reach it on an Edge, because the curator gives up its
replay to the approved peer before the member is ready. Delivery and reads
do not depend on it. These timings were measured before #3036, which
retries an incomplete private catalog within a short budget; they were not
measured again. - A node on an earlier version does not learn the protocols a peer
registers after their connection opened (#2854): a node on this release
announces such protocols to its connected peers, and it asks a connected
peer directly before it passes the peer over (#3145). A node on 10.0.23 or
earlier does neither. It learns a peer's protocols when the connection
opens, and two such nodes that connected before either had registered its
handlers recover when they reconnect, as before. - A member added to a private Context Graph only by wallet address does not
receive its Shared Working Memory until it joins (#2862): the member needs
the graph's metadata, which the curator sends on an approved join request.
Have the member send a signed join request; a pre-authorized member is
approved automatically. - A restarted member Edge may not reconnect to its private graph's curator
Edge if the curator is unreachable (#2865): approval now carries a
validated listener hint and recovery retries the curator connection
(#3058). This does not make an unavailable curator reachable. If automatic
discovery cannot establish the connection, connect to a reachable curator
multiaddr withPOST /api/connect. - Copies already stale before the upgrade may wait for another update
(#2866): the update lane initially replays about 500 blocks. Older stale
copies, and refresh targets given up after 24 hours, need the next update
or an explicit asset fetch. The rate-limited backfill is not in this
release. - The successor of an abandoned shared update draft is shared under the same
number (#2964): peers' share gate, the RFC-64 catalog and curator
confirmation (swmAwaitCuratorAck) do not replace a draft with another of
the same number, so peers may keep showing the earlier draft until the
update is published, and the replacement replaces the private payload
sealed under that number. - A catalog authority refresh can wait for the next five-minute pass on a
node with constant store writes (#2968): a refresh that is composed while
the node's authority facts change is composed again, up to six more times
within about four seconds. On a node whose unrelated writes never pause,
every attempt can be discarded, and a newly approved member then receives
the catalog only after the next periodic pass. - Some lifecycle pointer stamps are still only logged when they fail
(#2962): the Verifiable Memory stamps after a confirmed publish, and the
shared pointer stamp ofpublishAsyncbetween its share and its publish. - A share from one member of a public graph to another member it has no
connection with is not delivered (#1611, #3211): the owner receives it
and nothing carries it on. Members that reach each other only through the
owner's relays are in this layout. - A request over a relayed connection still sets up a second circuit that
is closed at once (#3211): the request then goes out on the open
connection (#3216), but each one costs a circuit on a relay. - A subscriber that does not get a graph's definition is not asked again
until the next subscribe call (#3207): when no connected peer holds it,
or the request fails for a reason other than the one #3216 removes. - A subscription to a public graph without content stays unsynced when one
of the peers it asked failed in transport (#3207): such a graph is marked
synced only when every asked peer answered, and the job is not repeated.
It becomes synced when content arrives or with the next subscribe call;
until then a write answersCONTEXT_GRAPH_NOT_WRITABLE. - Blazegraph can return the delete character U+007F as another character
(#3209): a catalog row whose literal holds it is refused on such a node. - Overlapping first shares into a private graph set up one epoch each, and
a recipient that misses more than 64 shares of an epoch refuses the later
ones until the epoch is replaced (#3202). - A catalog filter given a slug as a read returns it finds nothing when the
slug needed encoding (#3235, #3239): a read returns a catalog's slug
percent-encoded, as its IRI stores it, and the filter encodes the value
again. Slugs made from a name by the node ([a-z0-9-]) are not affected.
Pass the slug as it was written.
Fixed
-
A queued publication into a private Context Graph is terminal once its
catalog placement is recorded (#3081, #3127): a confirmed publication
into a private graph stayed non-terminal until the first attempt at
placing its asset in the author's catalog had ended. Placements run one
at a time on a node, so a job waited behind the placements of the jobs
before it, for work that does not decide whether the publication
succeeded. The job is now terminal once finality is observed, the
verified content is stored, the placement marker is stored and the
placement is requested. The catalog supervisor owns the attempt, its
retries and its drain at shutdown as before, and a marker that was not
placed before a restart is placed after it. A synchronous publish answers
at the same point. A recovery pass finalizes every proven job, where each
waited behind the placement of the one before it, and the finalize of a
held failed job no longer keeps the queue's node-wide claim lock while a
placement runs. In a test on a public testnet, with one private graph,
queued publications three at a time and the author catalog at 700 to 880
rows, a burst of 60 publications took 1,313 s on the previous code, and a
publication waited 816 s at the median after finality (1,122 s at the
90th percentile). That run was the bad case, in which one placement held
the catalog lock. With this change and #3133, #3136, #3141 and #3143 the
same burst took 430 to 630 s, and a publication waited 2.7 to 9.8 s at
the median (5.6 to 21 s at the 90th percentile). In a later run with
#3130 as well, 30 publications into a full catalog all finalized,
waiting 0.8 s at the median after finality (1.9 s at the 90th
percentile, 3.8 s at the most). -
A catalog placement no longer reads and verifies the whole catalog for
each asset (#3072, #3081, #3133): placing one confirmed asset read and
verified the whole applied catalog twice, for the coverage check and
again under the catalog lock, and verified every row a third time while
it produced the successor. The node now keeps the verified state of a
catalog's applied head in memory. It uses that state only while the
durable applied-head record is unchanged, and it reads and verifies the
head and its delegation from the durable store on every use. A row whose
exact bytes were verified earlier in the same process is not verified
again. A confirmation that is observed a second time, or whose row is
already in the catalog, is answered once the head's history and that
row's bundle have been read back, without the bundles of all the other
rows. The signed head, directory node and bucket are byte for byte what
the previous code signs for the same inputs. In the change's benchmark on
one machine, a placement into a catalog of 700 rows took 7.83 s with the
memory switched off, which is the earlier behaviour, and 2.04 s with it;
at 1,024 rows it took 11.8 s and 3.00 s. The first placement into a
catalog after a start costs what it did. On the previous code, in a test
on a public testnet, a burst of 60 publications took 287 s with the
catalog at 30 to 90 rows and 969 to 1,129 s at 390 to 690 rows. -
A new catalog head is delivered outside the catalog change, and only to
peers the node's own policy admits (#3081, #3130): a catalog change
committed its head and then waited, still holding the catalog's lock,
until its peers had been told, one peer at a time and each with a budget
of 10 s. A lane with no configured peers addressed every connected peer.
For a private graph most of those are not members: the node refused each
one locally, sent it nothing and logged the head as a failed announce at
warn level. The change now hands the committed head to a delivery owner
and returns. One owner per catalog asks the policy about every peer and
then sends the newest head in waves of 16, under one budget for all the
sends. A refused peer is left out and is not a failure. A peer the
policy check could not answer for is sent nothing, counted on its own
and logged at warn. A newer head replaces one that is still waiting, as
long as it stays within 1,024 versions of the head sent before it: a
receiver fetches the head it is told about and proves the lineage across
the versions it skipped, which receivers do since 10.0.15. Receivers
older than 10.0.15 accept only the direct successor of the head they
hold and were not tested with this release. A head that was not
delivered reaches a peer with the catalog's next head or through
replay. With one peer that never answers, two changes of one catalog
were placed after 10.5 s and 20.8 s before, and after 0.4 to 0.5 s and
0.65 to 0.75 s now. -
A head replay to a connecting peer no longer holds the catalog locks
while it sends (#3081, #3129, #3141): a node replays its catalog heads
to a peer on every new connection, on the peer's request and after a
join approval. The replay took the lock of every catalog it replayed and
sent head after head inside those locks, so a peer that did not answer
held every change of those catalogs for up to 10 s a head. A replay now
prepares its manifest under the locks and sends it after they are
released. When the sends have ended, the inventory is compared once more
and the replay is rejected as before if it moved. In a test on a public
testnet a placement waited up to 392 s for the catalog lock or the state
read under it before the change. With the change the longest such wait
in two bursts of 60 publications was 28 s in one and 0 s in the other. A
replay that a catalog change overlaps is still rejected; see Known
issues. -
A new asset for a full author catalog is refused once, parked and
reported (#3134, #3143): an author's catalog for a Context Graph is one
signed bucket of at most 1,024 rows. The placement of a further new
asset failed inside the successor producer, was reported as
catalog_private_repair_failedwith kindunknownand neither graph nor
author, and was retried for ever at intervals of up to 60 s. Every retry
read and verified the whole catalog, and every publication after the
catalog had filled added one more such placement. The asset is still
published and finalized as before. The node now refuses its row before
it builds anything, keeps the placement's marker and parks it: a pass
over parked placements reads one row a catalog and attempts nothing.
Parked placements are attempted again when the catalog shows a free row,
and one of them every hour. A newer version of an asset the catalog
holds is placed as before. The refusal is written as acatalog_full
warning and counted inGET /api/status. This adds no capacity; see
Known issues. In a test on a public testnet 170 queued publications took
a catalog of about 880 rows across the cap. All 170 finalized, with a
median wait of 3.2 s after finality, and the 26 placements past the cap
stayed parked and were not retried. -
A vector search over Working Memory covers the asking agent's own
entries (#2652, #3150): embeddings of Working Memory now record which
agent wrote them, and a vector search filters Working Memory by that
identity before it ranks. Shared Working Memory and Verifiable Memory
keep their graph-wide scope. The local vector store is migrated at
start, and its existing rows are kept. Working Memory embeddings written
before the upgrade have no recorded writer and no longer appear in an
agent's vector search; a search by the node operator still reads them.
The authorization checks of a Context Graph and of its Shared Working
Memory, and the text search, behave as before. -
A chain index start that could not complete is started again (#3115,
#3116): at start a node builds its chain index from the deploy block of
every indexed contract. Each read of that search has a 4 s deadline that
also covers its wait in the node's own request queue. One read still
queued at its deadline, or a head probe that no endpoint answered, ended
the start with a line that begins
[chain] one-log chain index disabled:, and the node ran without the
index until its next restart. A start that fails for a reason another
attempt can change (the node's own request capacity, a timeout, a
throttled or failing endpoint, a network error) is now repeated after
5 s and then at doubling waits of up to 60 s, until the index attaches.
It logs one line when it is deferred, one every 30 minutes while it
stays deferred, and one when the index comes up. A failure another
attempt cannot change ends the start as before, with the same line. -
Random-sampling proof and challenge transactions are sent with more gas
headroom (#3117, #3122): a proof transaction was sent with its bare gas
estimate. The proof settles the node's stake to the time of its block,
and an estimate taken on the block that holds the node's challenge skips
that work, so the same call needs more gas from the next block on. Such a
transaction was mined and reverted, and the next tick sent the proof
again: no proof was missed, and each case cost one failed transaction. On
four nodes this happened in 3 of 12, 7 of 12, 5 of 13 and 8 of 9 proof
periods of one morning. A proof is now signed with its estimate plus
50 %. A challenge creation already had that headroom, but what a creation
costs depends on which Context Graphs the draw of its block lands on, and
the block it is mined in can need several times the estimate. It is now
signed with its estimate plus the larger of 50 % and 2,500,000 gas. In
23.6 days of one test network's public history 557 of 5,336 challenge
creations failed, every one out of gas. Replayed on a local copy of that
network's state, 38 of 300 creations failed with the old limit and one
with the new, in a block where no limit succeeds. In 9.3 days of one
main network's history none of 4,791 creations failed. -
A graph whose reconcile pass was refused by the node's own request
limiter is asked again within seconds (#3028, #3123): a Verifiable
Memory reconcile pass that met a chain read the node's own request
limiter had refused ended there, and its graph waited for its turn in
the periodic sweep, with its chain-event nudges dropped meanwhile. The
sweep goes round the bound graphs four at a tick, so on a node with 40
graphs that turn came ten minutes later. It happened above all right
after a start, when background reads are held back: a member that had
missed one asset while it was down still lacked it nine minutes after
its start. The graph is now asked again 2.5 s later, one graph at a
time, for up to five minutes; after that the pass ends as it did and the
sweep retries it. A read that was sent and then failed or timed out, and
an answer from the chain, are handled as before. -
A graph whose reconcile pass was overtaken by an update of its
subscription is asked again (#3135): a reconcile pass works on the
graph's subscription row as it was when the pass began and stops when
the row has been stored again. Ordinary work does that on a running
node: a subscribe request for a graph the node already holds, catch-up,
or a sync round. The pass then ended as failed, with the warning
VM reconcile for "<graph>" failed; retrying on the periodic sweep: VM reconcile queue is closed for node shutdown
on a node that was not shutting down, and the graph got no further pass
until the periodic sweep, minutes later. Such a pass now leaves the
graph to be asked again 2.5 s later, on the row as it is then. In a test
with a publisher and a subscriber node, 39 of 40 published assets were
in the subscriber's Verifiable Memory at the first check with the fix,
and 11 of 20 without it. -
The checks of saved subscriptions after a start no longer take the
node's whole chain request budget (#3118, #3119): after a start a node
checks the saved subscriptions that its activation cap left dormant. It
checked them back to back, and on a node with several hundred such rows
that took the whole chain request budget for minutes after every start,
while the node's other chain reads waited or were refused. A check that
activates nothing is now followed by a pause of at least 5 s, at least
four times the time the check took and at most 30 s. A row that
something asked for is checked first, then rows with a saved chain id,
then the rest. In a test process with 571 saved rows, 565 of them not
confirmed by the chain, the checks made 10 chain reads a second for two
minutes before the change and 1.3 a second after it, and the node's
ordinary foreground reads waited up to 88 s for admission before and at
most 1 ms after. The last row was checked 44 minutes after the start,
where it was 2 min 39 s. -
Rolling activation of saved subscriptions keeps its cap and its
wakes (#3125, #3156): rolling activation could exceed its cap when
another request took the last slot during an authority read, and it
could miss a wake that was requested as a pass ended. The pass now
checks the cap again after its reads, and a wake that arrives as a pass
ends is kept. The pacing of the checks uses a monotonic clock, so a
change of the wall clock cannot lengthen a pause. -
Join metadata is fetched again when the fetch after a live approval
fails (#3109, #3110): when a join is approved on a running node, the
member fetches the curator's metadata in at most two attempts and then
asks for a broadcast catch-up, which does not carry private metadata. If
both fetches failed in transport, nothing fetched the metadata again
until the node restarted, and every write of the member was answered
CONTEXT_GRAPH_NOT_FOUND. The member now goes on with the bounded
recovery that start-up already uses: one attempt at once, then after
15 s, 30 s, 1 min and 2 min, and every 5 min from there, until the
metadata is confirmed, the approval is withdrawn or the node stops.
Metadata that the node refused because it names another registration
than the one its own row holds is not fetched again. Fixes a 10.0.23
known issue. -
Gossip reconcile and catalog responsibility keep their state when an
authority read gets no answer (#3042): two places that act on an
authority check reduced it to yes or no, so a chain read that got no
answer counted as a refusal. In the Shared Working Memory gossip
reconcile a subscription the node holds now stays, and one it does not
hold is not made. For the catalog of a private graph a responsibility
the node holds as a member stays. The graph is asked again shortly, one
graph every 2.5 s. A refusal and every other answered result take effect
at once, as before. -
The Shared Working Memory gate no longer drops a valid envelope because
another graph's metadata changed (#2968, #3011): the gate took its
snapshot of a graph's metadata again when a node-wide revision had
moved, and gave up after two attempts. A metadata write to any other
graph moves that revision, so on a busy node both attempts saw a change,
the gate failed with
metadata authority kept changing while resolving its SWM gateand the
envelope was dropped. The gate now compares the graph's own
authority-facts revision. A change to the graph itself still takes a
fresh snapshot. -
Concurrent callers share one chain adapter initialization (#3043):
every caller that found the chain adapter uninitialized ran its own
initialization of about 18 chain reads, in its own request class. After
a reset of the contract bindings, every caller that arrived before the
first one had finished repeated the reads, and the copies waited behind
each other in the node's own request queue. One initialization now
serves all the callers that wait for it, in the foreground class. On a
local chain a node ran 8, 12 and 19 initializations at once in three
starts before the change and one in each start after it, and the first
registration after the reset took 1.8, 3.0 and 10.2 s before and 0.11 to
0.13 s after. -
A read that another endpoint serves no longer spends the first
endpoint's re-probe (#3022, #3152): when the first chain endpoint
refuses a kind of read, that read is served by another endpoint. Such a
read used up the re-probe that was due for the first endpoint, although
the first endpoint was not asked. The re-probe is now used only when the
first endpoint is tried, so it stays available to the next ordinary
read. -
A dial by peer id alone has a deadline (#2864, #3149): the fallback
that finds a peer by its id passed on only the caller's signal, which
switched off the transport's default limits, and a Verifiable Memory
reconcile could wait on it without end. The fallback now runs under a
deadline, and the caller's wait ends there even when the transport
ignores the cancellation. A dial that completes after its deadline does
not count as a success. -
A slow identity probe is retried once before the peer is suppressed
(#2860, #3154): a peer whose identity responder was busy for longer
than the 3 s deadline went into exponential suppression before it could
prove its network. A probe that ends on a transport deadline is now
retried once with a response deadline of 15 s, and later suppression is
capped at 3 s while the peer has an open connection. The validation of
the signed proof, quarantine and the back-off for disconnected peers
are unchanged. -
A subscribe or unsubscribe that fails on an authority outage returns
an operation id (#2843, #3151): the response now carries an operation
id, and exactly one server log line matches it. The subscription error
code, the generic response body and the retry headers are unchanged. -
A failed asynchronous share reports the transition that was stored
(#2896, #3155): after a failed attempt the worker read the job's outcome
in a separate step, which could fail or see a later attempt. The queue
now returns the exact terminal or retrying transition with the write
that stored it, and bookkeeping that could not be resolved is returned
as an uncertain result, not as a fatal one. -
A publication no longer waits for the retirement of its Shared Working
Memory marker (#3081, #3179): after a confirmed publication the node
clears the asset's copy in Shared Working Memory and retires the marker
that the share left for it. The retirement took its turn on one node-wide
chain, shared with the retirements and the receiver leases of every graph.
A receiver lease is held while the node proves and commits a catalog
change from a peer, which can take tens of seconds, and the publication
waited for the retirement. So a publication into one graph waited behind
work on other graphs, for up to 30 s. At that bound the retirement gave
up: nothing was retired, and nothing came back for the marker. The
retirement of one asset now waits only behind work of its own graph, and
it answers at once when no marker of the asset is outstanding. Its caller
waits one second at most. The retirement goes on without the caller, and
one that did not get its turn or failed is repeated: after 30 s, then at
doubling pauses of up to ten minutes, ten times at most. A node that stops
ends the retirements that still wait and waits for one that is at work.
What is still owed at a stop is dropped; a marker that is left goes with
the next confirmed publication of its asset, or with a catalog projection
that covers the asset. In a test on a public testnet, with bursts of 30
queued publications into a private graph on a node with peers connected, a
publication waited 25 to 32 s at the median after finality before the
change and 2 to 4 s with it. What still waits is under Known issues. -
Reads that prove a Context Graph's registration state take turns with
other chain requests (#3064, #3194): the node's own chain request budget
admitted a read that proves a graph's registration state ahead of every
ordinary request, without a bound. On a node where such reads arrive as
fast as the budget admits them, nothing else was admitted. A transaction
then waited for a permit behind reads that had each arrived after it,
until its 10 s deadline, and was not sent: in 10.0.23 the line is
<step> waited 10000ms for local RPC admission and was not sent, which
#3171 rewords. When both kinds are waiting they now take turns: neither
waits for more than one admission of the other, and background work may
take one turn between them, within its own budget. Whenever only one of
the two kinds is waiting, admission is exactly as before. In a workload of
such reads issued back to back beside one ordinary read a second, on the
default budget, a pass of the reads took 165 s where it took 150 s, about
10 % longer, and an ordinary read was admitted after about 0.2 s where it
had waited out its deadline and was not sent. The change does not make
these reads fewer; see Known issues. -
A bounded read of an unregistered graph name is answered without a
provider scan (#3064, #3183): some reads of a graph's name accept
bounded freshness; the read-authority check behind queries and Shared
Working Memory sync is one. When such a read found the name not
registered, it paid for a scan of the chain endpoint almost every time:
about nine requests a read whenever the chain had moved since the last
one. Such a read is now answered from the chain log the node already
follows. Where that log cannot answer, on a node without its chain index
or with an index that has fallen behind, one scan a tick serves the
bounded reads, where each read scanned. A read that asks for the live
state is unchanged, and a registration is seen as soon as the node's index
has committed it. This takes one source out of the reads that #3194 is
about. It is not what ended the waiting of transactions there, and the
live reads are as frequent as before. -
A subscribe that changes nothing no longer stops a member's first
store (#3182): a member that has just been approved for a private graph
fetches the graph's published assets, authenticates each against the chain
and stores it. The store runs under the graph's subscription record and
stops when that record has been replaced. Every subscribe call installed a
new record, also a call that changed nothing, and so did a catch-up result
that only updated readiness. A client that repeated its subscribe while it
waited stopped the store that way, with the error
Context graph binding for <graph> changed during graph-scoped store. The
verified asset was dropped, the catch-up reported the graph unreachable,
and the member got the asset later by another route. A subscription write
that changes nothing now keeps the stored record, and a store continues
when the record was stored again with the same admission, lifetime, chain
binding and metadata state. A record that was unsubscribed, that was
deleted and subscribed again, or that differs in one of these four still
stops the store. A Verifiable Memory reconcile pass is no longer overtaken
by such a subscribe either (#3135). -
A refused query that only aggregates answers the row it has over no
data (#3181): a read the node refuses is answered with an empty result
of the query's form. For aSELECTwhose projection holds only aggregate
calls, such asSELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }, that was no
row, although such a query has one row also when nothing matches: a count
of 0. A client that asked for a count had nothing to read. The node now
answers that row, as it does for a graph that holds nothing:COUNT,
SUMandAVGare 0,GROUP_CONCATis an empty string, andMIN,MAX
andSAMPLEare unbound. The same holds for a view that resolves no
graph, for example the Working Memory view of an agent with no draft. The
row is made from the query text; the query is not run. It is made only
when theWHEREgroup needs stored data to have a solution. A query that
has solutions without data (inlineVALUES,BIND, anOPTIONALon its
own) keeps the answer without rows, and so does a query withGROUP BY,
HAVINGor an expression around an aggregate. For those a refused read
still differs from a read of an empty graph. -
Daemon read routes authorize an agent token against the Context Graph it
asks about (#3172): a read route of the daemon now checks a request made
with an agent token against the Context Graph the request names. A request
for a graph the agent may not read is refused with 403. When the node
cannot establish the answer, the request is answered 503 with
Retry-Afterand can be sent again. A listing of publisher jobs made with
an agent token names its Context Graph:GET /api/publisher/jobstakes
contextGraphId, anddkg publisher jobstakes--context-graph. The
node operator's token and a node with authentication disabled read as
before. -
The
ontologygraph is served to peers only when every Context Graph
binding in it is classified (#3157): a node keeps itsontologygraph
from sync, changelog sync and remote queries until a pass has classified
or moved the Context Graph metadata that earlier builds left in it
(#2787). That now also holds for a binding the node cannot classify from
its own data. The pass at start makes no chain reads and leaves such a
binding unclassified, and the graph stays closed until later passes have
classified every binding against the chain, at most 16 new ones a pass.
What a pass has proven is kept for the next one, so a large graph finishes
over several passes. A later pass that fails, is cut short or leaves a
binding unclassified closes the graph again. There is no time limit: a
node whose chain reads keep failing does not serve the graph. See Known
issues anddocs/use-dkg/run-node.md. -
A node tells its connected peers about protocols it registers after they
connected (#2854, #3145): a node records a peer's protocols when the
connection is identified, and a node registers its sync handlers after its
transport has started. A peer that connected in between recorded the node
without sync support and passed it over, for sync and for the fetch of a
single asset, although the connection was live. This was seen on a main
network after a Core restarted. A node now also runs the libp2p identify
push protocol, so a change of its protocol list reaches the peers it is
connected to. A peer on an earlier version does not take the push. So a
requester whose record of a connected peer lacks the sync protocols now
asks that peer directly before it passes it over, with a bounded probe
that sends no application data. The readiness checks at the start of a
catch-up run four at a time, where they ran one after another. -
A batch stream request that cannot be served is answered with a
refusal (#3137): a Core that serves the exact-asset batch stream reset
the stream, before the first byte of its answer, when the requested asset
had no confirmed export on that Core, when it refused the request, or when
it could not read the graph's authority or its store at that moment. The
requester saw a broken stream in each case. The Core now answers on the
stream, with refusal codes the stream already has:ASSET_MISSINGfor an
asset without a confirmed export,DENIEDfor a request it refuses, and
BUSYwhen the authority cannot be read, the store is under pressure or
the Core is not serving that graph for now. A refusal can follow assets
that were already sent: those stay committed, and the asset in progress is
not acknowledged. Malformed metadata, a root that does not match and a
source that changes during the export still end the stream. The Core logs
each refusal by stage and code, and a failed inbound stream by stage,
without request data. -
A deadline spent waiting for the node's own request budget is not
counted against the endpoint (#2949, #3171): a step of several chain
requests under one deadline, in a publish for example, could have one
request answered and then spend the rest of its deadline waiting for a
permit of the node's own request budget. That ended as a timeout of the
endpoint: the node failed over and de-preferred an endpoint that had
answered. A deadline that runs out while a request waits for a permit and
no request is in flight now ends as local capacity, with the code
RPC_REQUEST_GOVERNOR_QUEUE_FULL. No endpoint is blamed and none is tried
next, because every endpoint shares that budget. A request that is in
flight when the deadline runs out times out as before. The line reads
<step> exhausted its <n>ms deadline waiting for local RPC admission with no endpoint request in flight.
A step that had sent nothing at all was already taken for local capacity,
and its line read
<step> waited <n>ms for local RPC admission and was not sent. -
Catalog status keeps a promised head pending when its provider stops
answering (#2826, #3161): the per-graph catalog status in
GET /api/statuslost a head that a peer had promised in a replay once
that peer stopped answering. When the promised head did not hold more rows
than the applied one, because it replaced rows or removed the last one,
the graph readcomplete, with the old catalog version and the digest of
the head already applied as the expected one, although a newer signed head
was owed. Promised heads now count as pending. Such a graph reads
known-incompleteuntil the head is applied, andapplyingonly while a
live target is pending. The expected version, digest and row count come
from the newest head of each author's catalog, and the expected digest is
null when two heads compete at one version. -
The Context Graph enumeration finishes one run of ids before it asks for
the next (#3026, #3158): the enumeration reads Context Graph ids from
the chain in runs of 32, with one aggregate request a run and single reads
for a run whose aggregate request fails. When an early id could not be
read, the requests for later runs had already been sent, and their answers
were discarded. Each run, with its single reads, is now resolved before
the next is requested. The ids returned, the pinned block and the bound of
four readers are unchanged. -
A refusal for a full catalog keeps the remembered catalog state
(#3134, #3159): the node forgets the verified state of a catalog when a
change to that catalog fails (#3133). The refusal of a new asset for a
full catalog (#3143) counted as such a failure, so the next placement into
that catalog read and verified every row again: once an hour, and once
after a restart. The refusal is decided from the row count of the state
the change has just read, before anything is built or written, so the
state is now kept. Every other failure forgets it as before. The saving
was not measured on a catalog at the cap. -
The dashboard shows
Node unavailablewhen the node does not answer
(#1133, #3160): when the page's first status checks failed, the dashboard
filled in demo data, while health, balances, graphs and integrations kept
looking live. Every route now shows aNode unavailableview with a
Retry connection button, which reloads the page. A status answer keeps
the dashboard, and a 401 keeps the token prompt. -
Shares into a private Context Graph that overlapped took the same message
index, and recipients refused later shares (#3203). Recipients answered
with a transient rejection and loggedSender Key message nonce does not match ratchet state; past 64 skipped messages they refused every later
share of the epoch. The assets still arrived once finalized. Present since
10.0.0. -
An asset whose text holds a tab or another control character blocked its
author's catalog for that graph (#3198, #3209). The author logged
catalog_repair_failedwithkind: unknownon every attempt, and neither
that asset nor any later one shared into the graph's root reached other
nodes. -
A node on Blazegraph refused catalog rows with common typed literals
(#3198, #3209): whole-second date-times, integral doubles and floats, and
short durations, which that store returns in another lexical form of the
same value. -
Two Edge nodes that reach each other only through a relay did not deliver
a share into a named sub-graph to each other (#3198, #3208). Share
recipients of a public graph were the peers subscribed to its gossip topic,
and gossip does not run over a relayed connection. -
A request to a peer reached only through a relay failed whenever its dial
ended in a closed connection (#3198, #3216), withRemote closed connection during opening. Sync requests, which are sent once, never used
the relayed connection that was already open. A subscriber's request for a
graph's definition to the one peer that holds it failed in most tries. -
A subscribe to a public Context Graph asked only three peers for its
shared memory (#3198, #3205), since 10.0.17. With four or more peers the
subscription of a graph without content never reportedsyncedand
refused writes withCONTEXT_GRAPH_NOT_WRITABLE. -
A subscriber could leave the peer that holds a graph's definition out of
the peers it asked (#3198, #3212), on a node with more than eight
connected peers. -
An Edge could move a relay reservation to a relay outside its network's
list and then not get it back (#3206), after a listed relay restarted;
the node then announced an address that other nodes of its network refuse. -
A new subscription whose graph definition arrived within a second or two
never reportedsynced(#3198, #3217): its catch-up asked every peer
before the graph's policy was active on the node, and nothing repeated it.
A catch-up job now waits for that, once, for at most ten seconds. -
A member of a private Context Graph stopped publishing its catalog after
the roster changed again (#3220): for example when a second member was
approved. The member's metadata then held two roster generations, its
authority for the graph stayed incomplete withRFC-64 private roster metadata has conflicting generations, and the curator received none of
that member's writes into the graph's root. The newest generation is now
read. -
A local-model chat failed when a request exceeded the model's context
(#3235): when the model endpoint rejects a request for its context size
before inference, the chat now retries without the earlier conversation
history, and then from the current tool results without tool schemas; at
most twice, and completed tool calls are not repeated. A domain profile can
namerequiredTools; the chat then refuses to start when one of them is not
available.
Changed
- A publish into a private graph resolves its recipients and its curated
context once (#3069, #3086): each Verifiable Memory publish resolved
the curated context twice, for payload and for chunk encryption, and
concurrent resolutions of a private roster repeated every per-agent key
lookup. One context is now resolved per publish attempt, with one policy
probe and one sender-key setup pass where there were two of each. Key
lookups are shared between concurrent callers and remembered for one
second; the memo is bypassed while a key or route write is pending.
After a sender-key setup the recipient authority is read once more, and
the resolution starts over when the roster or a route changed meanwhile.
An authority that still moves after three setups fails the attempt with
a typed authority failure, which a queued Knowledge Asset publish
records asauthority_unavailableand retries. - A missing asset version view names the endpoints that failed (#3097,
#3099, #3106): after a publish is confirmed the node reads the asset's
version from the chain at one pinned block, from the first endpoint and
then from the others. An endpoint that failed that read was passed over
without a word, and when no endpoint served it, recovery deferred on
every tick with the same message and no cause. The read now keeps which
endpoint failed, by its position in the configured list and its host, at
which step, and a closed class for why. No URL and no provider error
text is kept. The recovery deferral, the exact-asset fetch and the
Verifiable Memory refresh check append that cause to their messages. The
remains pendingwarning is logged once per asset when a reason appears
or changes and then as one summary per reason every 5 minutes, where it
was logged on every tick. When no endpoint has served the read for 5
minutes and at least 12 deferrals, one line that begins
Operator action needed:says what is not finalizing, why, and what to
do. What each caller decides is unchanged. - A node built from source keeps the identity of its build (#2852,
#3153): a source build captures the commit it is built from before it
compiles and stores it only when the build succeeds, and status keeps
reporting that commit after the checkout has moved on. A build from a
changed or unidentifiable tree is not certified: itscommitin
GET /api/statusends in-dirty, also when the tree only holds
untracked files. A failed compilation keeps the earlier outputs and
their metadata. When the auto-updater builds from source, it clears
stale build metadata in its inactive slot first, and a failed clean-up
or build leaves the active slot untouched. The devnet scripts rebuild
stale executables. - The dashboard says when a size is a preview (#1015, #1026, #3147,
#3148): a memory layer that was capped or unavailable looked like a
complete graph size. Rows and the total now show a loaded preview, with
+on an entity count that is a lower bound and~on a triple count
that can still change, and tooltips explain the cap and the
uncertainty. The triple counts of Working, Shared Working and
Verifiable Memory are labelled as stored triples, and the Verifiable
Memory count as data plus provenance. - The managed Oxigraph store reports its retained write-ahead log, and a
maintenance reopen is available, off by default (#2395, #2571, #3195): a
managed store (store.backend: oxigraph-server) keeps its write-ahead log
for as long as the store process runs; only a reopen, such as a restart of
the node, replays and truncates it. The node now measures the retained log
once a minute and reports it in its log when it first reaches 4 GiB and
again each time it has doubled; a confirmed reopen of the store starts
these reports over. By default nothing else changes. A maintenance reopen
is available and off: it runs only when
store.options.walRestartThresholdBytesis set to a positive integer, a
size in bytes. Any other value is ignored with a log line, and a node with
the option set says at start which way it runs. With the reopen on, the
node refuses all new store work once the log reaches that size, waits for
the work in progress to end and for 30 s without any, reopens the store
and then leaves it alone for an hour. In our tests one cycle refused store
work for about 75 to 95 seconds, and store-backed API calls answered 503
withoutcome: "not_started"meanwhile. Also part of this change: the
managed store is restarted when an update, and not only a query, runs past
its client deadline. A catalog compare-and-swap whose confirming read
fails is reported with an unknown outcome and invalidates cached
revisions, a change that commits and then fails in its bookkeeping is
flagged for reconciliation, and an abandoned read keeps its claim on a
restart until a restart is confirmed. Described in
docs/use-dkg/storage-sparql-http.md. - Chain adapter initialization accepts only a typed absence of an optional
Hub contract (#2913, #3162): initialization looks up several contracts
that a Hub may lack and goes on without the ones that are missing. It took
a failed lookup for a missing contract unless the node itself had
interrupted the lookup, so a lookup that failed on the endpoint left the
node without that contract until the next initialization. Only the Hub's
own answer that the name is not registered now counts as absence. Any
other failure fails the initialization, and the next caller starts it
again. The default endpoints of the three main networks answer a missing
contract in the form the node accepts. A missing required contract fails
initialization as before. - Catalog production yields the main thread, and a confirmed placement
reads one inventory row (#3134, #3175): the producer of a catalog's
successor verified the predecessor and built the replacement bucket in one
turn of the main thread. It now takes its rows aside and gives the thread
a turn between the two. A confirmed placement that has work to do read the
author's whole local inventory to find one asset; it now reads that row by
its key. The row is a hint: the seal and content checks still decide a
placement. The signed objects are byte for byte what they were, and the
capacity of a catalog is unchanged. - The refresh of a Verifiable Memory copy after an update is a fetch
policy of its own (#2882, #3166): a refresh carried its need for a fresh
responder session as a separate flag through Shared Working Memory hosting
and durable sync. It is now one policy,vm-update-refresh, beside the
ordinary and the challenge-pinned ones: an ordinary fetch resumes its
session, and every peer attempt of a refresh starts a new snapshot. What a
node does is unchanged. For callers of the agent API,
forceFreshExactSessionis still accepted and is deprecated, a request
that selects assets no longer takes the whole-graph changelog lane, and a
requester context of the older positional form refuses a refresh. - A Knowledge Asset update event keeps one identity from the chain to the
refresh queue (#2885, #3168): the update event of an asset now carries
one typed identity, made of its block, block hash, log position and
transaction as far as the chain adapter reports them, through the adapter,
the replay filter of the event lane, the lifecycle callback and the
Verifiable Memory refresh queue. Before, each of these read the fields for
itself. Two updates with the same root, an update whose block was
replaced, and an event that comes again with a block position it lacked
the first time are each delivered. An exact replay is still suppressed,
and the other event lanes keep their replay handling. - Publisher wallet funding is assessed for each candidate wallet (#2869,
#3167): the check that picks a funded publisher wallet now keeps, for each
candidate, whether it is funded, not funded or could not be read. When no
candidate is funded at the first look, the candidates are re-priced one
after another, and a wallet that was topped up while an earlier candidate
was re-priced is read again before its own decision. The order of
selection, the codesNO_FUNDED_PUBLISHER_WALLETand
PCA_FUNDING_UNKNOWNand their messages are unchanged. - Test and CI plumbing (#3131, #3132): the timing baseline from which
CI plans the ten agent test shards is refreshed. 264 of the 644 agent
test files had no measured time and were planned at 60 s each, so a
branch that added a test file could move the heavy files into one shard,
which was then cancelled at its 10-minute limit although no test had
failed.
Added
- External signer for EVM signing (#3177): an Edge node can delegate the
signing of its transactions to an external signer, for example an HSM-backed
one, and hold no operational private keys. Authors and payers are separate
keys; payers covered by a Publishing Conviction Account need no TRAC, and a
node whose author keys live in the signer can register a curated context
graph backed by a PCA, into which every agent of the PCA can publish.
Configuration and limits:docs/use-dkg/external-signer.md. - Timing of a queued publication's wait after finality (#3081, #3120,
#3136): oneasync_publish_post_finalityline per finalized queued job,
at info when the wait is 10 s or more and at debug otherwise. It names
the path the job took (inline,detached,held-failedor
interrupted) and splits the wait intotailMs,recoveryAdmissionMs,
recoveryRetryMs,proofMs,recoveryMsandterminalWriteMs, which
sum tototalMs.tailMs, the executor's work after the confirmation,
is split again by step:tailPublishMs,tailReceiptWriteMs,
tailPublishedGraphClearMs,tailLegacySwmRetireMs,
tailRemainingSwmClearMs,tailLifecycleStampMs,tailGraphIdReadMs,
tailFinalizationGossipMs,tailShareMarkerClearMs,
tailCatalogObserverMsandtailRestMs. A part whose step did not run
prints-. Onerfc64_catalog_placement_waitline is written for a
catalog placement that takes 5 s or more, withrequestMs,queueMs
andattemptMs, and the attempt by phase (coverageMs,assetMs,
stateMs,successorMs,casMs,otherMs). ItsannounceMs,peers,
failedPeersanddeniedPeersare always 0, because a head is no
longer delivered inside an attempt (#3130). The lines of one job carry
[from:<jobId>]. The marks are kept in memory, so a restart loses them. - Author catalog blocks in
GET /api/status(#3120, #3127, #3130,
#3143): three blocks underrfc64Catalog, with counts only and no
graph, author, asset or peer.finalizedPrivatePlacementQueuecounts
the placements that are owed (depth,pending,oldestPendingAgeMs,
waiters,oldestWaiterAgeMs,passRunning,lastPassDurationMs,
cooldownSkips).authorCatalogCapacitycounts the catalogs that
refused a new row for lack of room and the placements parked for them
(scopesAtCap,parkedPlacements).catalogHeadDeliverycounts head
deliveries since the node started (handoffsQueued,handoffsNobody,
handoffsNotQueued,reportedHeads,deliveredPeers,failedPeers,
refusedPeers,uncheckedPeers,unconfirmedPeers,supersededHeads,
undeliverableHeads,checkpointCapacityExceeded). A parked placement
is still owed, so a full catalog shows in the first block as a backlog
that does not drain:pendingminusparkedPlacementsis what waits
for an attempt. chain.versionSnapshotinGET /api/status(#3097, #3099): the
endpoints that failed the asset version read, by position and host. An
endpoint that reads pass over stays on record while they succeed on
another one.DKG_RFC64_CATALOG_MUTATION_MEMORY(#3133): set to0, it switches
off the in-memory catalog state, and every placement reads and verifies
the durable catalog as before. Described indocs/use-dkg/run-node.md.- An out-of-gas hint on a reverted write (#3117, #3122): a receipt
carries neither the gas limit nor revert data, so the error of a write
that was mined and reverted gave no hint of the cause. When at most 1/32
of the gas limit was left, the message now ends
it used <n> of its <m> gas limit, so it probably ran out of gas. V10
publish and update transactions have their own receipt handling and do
not get the hint. chain.rpcRequestTimingandchain.rpcRequestGovernorin
GET /api/status(#2949, #3171):rpcRequestTimingholds histograms
since the node started, for the foreground and the background class:
admissionWaitfor waits that got a permit of the node's own request
budget,admissionDeadlineWait,admissionCancelledWaitand
admissionRejectedWaitfor waits that ended on a deadline, on the
caller's cancellation or on a refusal, andendpointLatencywith
endpointFailures.rpcRequestGovernoris the budget's current log
window: its rates, the tokens available, the queue depths and the counts
of admitted, deferred, rejected and cancelled requests. Reading status
sends no chain request.docs/use-dkg/rpc-request-budget.mdis new: it
says how to sizechain.rpcRequestBudgetand how to read these fields.- An
AssertionWrittenactivity for every assertion write (#674,
#3164): a successfulassertionWriterecords an activity in the Context
Graph's_metagraph, in the same store insert as the payload: the
writer, the time, the assertion, every Working Memory graph that received
payload and the number of payload quads in the call. Repeated writes keep
separate activities. Assertion history returns them aswrittenevents;
the lifecycle state of the assertion does not change. With
metadata.provenanceEvents: falsethey are not written. dkg okf verify --sub-graph-name(#1419, #3165): with
--sub-graph-name <name>,dkg okf verifyreads that sub-graph's Shared
Working Memory for every integrity count and for the missing-concept
query, asdkg okf exportdoes. Without the option it reads what it did.
Described indocs/use-dkg/okf.md.
Full Changelog: v10.0.23...v10.0.24