Skip to content

v10.0.24

Choose a tag to compare

@branarakic-agent branarakic-agent released this 14 Oct 19:31
· 1138 commits to main since this release
v10.0.24
fffef7d

DKG v10.0.24

A patch for queued publications into private Context Graphs. A queued
publication into a private graph stayed non-terminal for minutes after its
transaction was final. It waited for the placement of its asset in the
author's catalog, behind the placements before it, and each placement read
and verified the whole catalog three times and told peers about the new head
while it held the catalog lock. The publication is now terminal once
finality is observed, the verified content is stored and the placement is
recorded and requested. A placement no longer reads and verifies the whole
catalog for each asset, and peers are told about a new head outside the
catalog change. A publication also no longer waits, behind work of other
graphs, for the retirement of the marker its share left. A new asset for a
full author catalog is refused once and reported; its placement was retried
for ever. After a start, a chain index that could not start is started
again, a graph whose reconcile pass was refused or overtaken is asked again
within seconds, and the checks of saved subscriptions no longer take the
node's whole chain request budget. Reads that prove a graph's registration
state no longer keep a transaction waiting until its deadline. A read made
with an agent token is authorized against the Context Graph it asks about,
and a node serves its ontology graph only once every binding in it is
classified. Random-sampling proof and challenge transactions are sent with
more gas headroom, a vector search over Working Memory covers the asking
agent's own entries, and a member whose metadata fetch failed right after a
live approval no longer needs a restart, which fixes a 10.0.23 known issue
(#3109).

This release also fixes what a community report found between Edge nodes
(#3198). An asset whose text holds a tab no longer blocks its author's
catalog, and a node on Blazegraph accepts catalog rows with dates and
numbers. A share into a sub-graph reaches the peers that follow a public
graph through a relay, and a request to such a peer no longer fails when its
dial ends in a closed connection. A new subscription to a public graph asks
every peer it selected, gets the graph's definition and reports synced
within seconds. An Edge keeps its relay reservations on the relays its
network lists.

An Edge node can now hand its EVM signing to an external signer, so that it
holds no operational private keys; this is off by default (#3177).

No smart-contract, ABI or deployment registry changes. Message formats
are unchanged; a node now announces only the newest head of a catalog
(#3130) and runs the libp2p identify push protocol (#3145). A catalog
projection may now hold a literal with a tab or another control character;
a node on an earlier version does not apply a catalog head with such a row
(#3209).

npm: @origintrail-official/dkg@10.0.24 · dist-tags latest, testnet, mainnet

Upgrading from 10.0.23

Change Impact Action
A queued publication into a private Context Graph is terminal before its catalog placement has run (#3127) The job is terminal once finality is observed, the verified content is stored and the placement is recorded and requested. The author's catalog can list the asset later. A synchronous publish no longer waits for a placement attempt either. Placements that are still owed are counted as pending, with oldestPendingAgeMs, under rfc64Catalog.finalizedPrivatePlacementQueue in GET /api/status None. Read pending to see whether placements are still owed
A new catalog head is delivered after the catalog change, and only to peers the node's own policy admits (#3130) The warn line RFC-64 catalog head announce failed for F/T peer(s) keeps its text, but T counts eligible peers only, and a head with no eligible peer writes nothing at warn level. Peers that the policy check could not answer for get a warn line of their own, RFC-64 catalog head delivery could not check U/T peer(s) .... rfc64Catalog.catalogHeadDelivery in GET /api/status counts hand-offs and peers by outcome since the node started None. A refused peer no longer counts as a failed one. uncheckedPeers growing while deliveredPeers stands still means the policy check answers for nobody
A new asset for a full author catalog is refused once, parked and reported (#3143) The asset is published and finalized as before. Its placement is no longer retried at intervals of up to 60 s. The node writes a catalog_full warning when a catalog first refuses and then at most one an hour, for a refused placement and for a refused share-time projection each on its own, and counts scopesAtCap and parkedPlacements under rfc64Catalog.authorCatalogCapacity in GET /api/status. catalog_private_repair_failed and catalog_repair_failed are not written for this reason. A parked placement still counts as owed in finalizedPrivatePlacementQueue Watch authorCatalogCapacity. The change adds no capacity; see Known issues
A node keeps the verified state of its author catalogs in memory (#3133) At most 64 catalog scopes (one author in one graph) and 96 MiB in all, least recently used first out. Nothing is persisted. The first placement into a catalog after a start reads and verifies the whole catalog, as every placement did None. DKG_RFC64_CATALOG_MUTATION_MEMORY=0 switches the memory off; see docs/use-dkg/run-node.md
The wait of a queued publication after finality is logged and shown in status (#3120, #3136) Two new log lines: async_publish_post_finality, one per finalized queued job, at info when the wait after finality is 10 s or more and at debug otherwise, and rfc64_catalog_placement_wait for a catalog placement that takes 5 s or more. New block rfc64Catalog.finalizedPrivatePlacementQueue in GET /api/status. The execution, recovery and placement lines of one job carry [from:<jobId>] None. The lines and the block are additive
A vector search over Working Memory covers the asking agent's own entries (#3150) The local vector store is migrated at start, and its existing rows are kept. Working Memory embeddings written before the upgrade have no recorded writer and no longer appear in an agent's vector search. Shared Working Memory and Verifiable Memory keep their graph-wide scope None. A search by the node operator still reads the embeddings that have no recorded writer
Random-sampling transactions are signed with a higher gas limit (#3117, #3122) A proof is signed with its gas estimate plus 50 %, where it had no headroom, and a challenge creation with its estimate plus the larger of 50 % and 2,500,000 gas, where it had 50 %. Unused gas is refunded, but the wallet must hold the limit times the fee cap before the transaction is accepted: about 2.83M gas for a challenge creation, where it was 0.49M. A creation that still runs out of gas burns its whole limit None for a funded wallet. The wallet that signs random-sampling transactions must be able to pay the higher limit up front
A chain index start that cannot complete for now is tried again (#3116) The start is repeated after 5 s and then at doubling waits of up to 60 s, until the index attaches. The node logs [chain] one-log chain index start deferred (retrying, next attempt in 5s): <error> once, says every 30 minutes that the start is still deferred, with the latest failure, and logs once more when the index comes up. A line that begins [chain] one-log chain index disabled: is still written for a failure another attempt cannot change None. A node that logged the disabled line for a transient failure no longer needs a restart to get its index
A Verifiable Memory reconcile pass that was refused locally or overtaken is asked again (#3123, #3135) Two new info lines: VM reconcile for "<graph>" is waiting for local RPC admission; asking again shortly: <the refusal> and VM reconcile for "<graph>" was overtaken by a change to the graph's subscription or binding; asking again shortly. For these two causes the warning VM reconcile for "<graph>" failed; retrying on the periodic sweep: ... is written only once asking again has stopped; a graph is asked again within a window of five minutes None. No setting is added
The checks of saved subscriptions that the activation cap left dormant are paced after a start (#3119) On a node with several hundred saved subscriptions that the chain does not confirm, the last one is checked about three quarters of an hour after the start, where it was a few minutes. A graph the chain confirms waits for the rows before it only when its saved row has no chain id and nothing asks for it: 44 minutes in the measured run, where it was 3. One WARN summary when a pass has no row left, and every five minutes while it runs, replaces one WARN per row; the per-row line is written at debug level. A row whose chain id the chain does not know shows as deactivated on GET /api/context-graph/subscriptions at once None. No setting is added and no saved row is changed. An explicit subscribe does not wait for these checks
A missing asset version view names the chain endpoints that failed (#3099, #3106) GET /api/status gains chain.versionSnapshot. The recovery deferral, the conflict of an exact-asset fetch (Knowledge Asset <ual> has no coherent on-chain version snapshot: <why>, the HTTP 409 body of the operator asset fetch) and the Verifiable Memory refresh check (no coherent chain view confirms the copy: <why>) end with the cause: each endpoint that gave no view, by its position in the configured list and its host. The remains pending warning is logged once per asset when a reason appears or changes, then as one summary per reason every 5 minutes None. The earlier text is the start of each message. A line that begins Operator action needed: names an endpoint to fix or replace
A subscribe or unsubscribe that fails on an authority outage returns an operation id (#3151) The response carries an operation id, and exactly one server log line matches it. The subscription error code, the generic response body and the retry headers are unchanged None. Use the operation id to find the line in the node's log
A slow identity probe is retried once before the peer is suppressed (#3154) A probe that ends on a transport deadline is retried once with a response deadline of 15 s, where the first deadline is 3 s. Later suppression of that peer is capped at 3 s while it has an open connection. The validation of the signed proof, quarantine and the back-off for disconnected peers are unchanged None
A failed asynchronous share reports the transition that was stored (#3155) The worker takes the terminal or retrying transition of a failed attempt from the write that stored it, where it read the job's outcome in a separate step that could fail or see a later attempt. Bookkeeping that could not be resolved is reported as uncertain, not as a fatal failure. The operator diagnostics and the documentation (docs/use-dkg/knowledge-asset-lifecycle.md) are updated with it None
The commit of a node built from source can end in -dirty (#3153) GET /api/status reports <commit>-dirty for a build from a checkout with local changes or untracked files. A release slot in which the git updater has set stale package copies aside counts as such a checkout (#3169). commitShort carries no suffix. Nodes installed from npm are not affected None. Tooling that compares commit between nodes should compare commitShort or strip the suffix
The managed Oxigraph store reports its retained write-ahead log; a maintenance reopen is available and off by default (#2571, #3195) By default nothing changes except new log lines. A managed store (store.backend: oxigraph-server) keeps its write-ahead log for as long as the store process runs. The node measures it once a minute and writes [oxigraph] <size> of write-ahead log is retained and it grows until the store is reopened; the maintenance reopen is off. ... when the log first reaches 4 GiB and again each time it has doubled. The reopen runs only when store.options.walRestartThresholdBytes is set to a positive integer, a size in bytes; any other value is ignored with a log line. With it on, the node refuses all store work once the log reaches that size, waits for the work in progress, reopens the store and then leaves it alone for an hour. In our tests one cycle refused store work for about 75 to 95 seconds, and store-backed API calls answered 503 with outcome: "not_started" meanwhile. Apart from the log: an update that runs past its client deadline (store.options.clientTimeoutMs, 30 s by default) now restarts the managed store, as a query already did, which interrupts the writes in progress on that store None. Operators of long-running nodes with a managed store may set the option, or restart the node when the report appears; see docs/use-dkg/storage-sparql-http.md. Retry an interrupted write under the same Knowledge Asset name or idempotency key
A read made with an agent token is authorized against the Context Graph it asks about (#3172) Daemon read routes check a request made with an agent token against the Context Graph it names. A request for a graph that agent may not read answers 403, Not authorized to read this context graph. When the node cannot establish the answer it answers 503 with Retry-After and the code CONTEXT_GRAPH_READ_AUTHORITY_UNAVAILABLE, and the request can be sent again. A listing of publisher jobs made with an agent token names its Context Graph: contextGraphId on GET /api/publisher/jobs, --context-graph on dkg publisher jobs. The node operator's token and a node with authentication disabled read as before Integrations that read with agent tokens should expect 403 where a graph is outside the agent's access, and should name the graph when they list publisher jobs
The ontology graph is served to peers only when every Context Graph binding in it is classified (#3157) A node whose ontology graph holds bindings that it cannot classify from its own data does not serve that graph to peers (sync, changelog sync, remote queries) after a start, until later passes have classified them against the chain, at most 16 new bindings a pass. On test nodes that took from one minute to about three hours. A pass that fails or is cut short closes the graph again, and there is no time limit. The node warns at start Not serving the ontology graph to peers until the context graph metadata relocation reaches every candidate. After a pass that left some it logs Context graph metadata relocation is incomplete: <n> deferred, <m> unclassified candidate(s), where it logged Context graph metadata relocation stopped early: .... It logs Context graph metadata relocation reached every candidate; serving the ontology graph to peers again when it serves again. Other graphs are served as before None. If the last line does not come, restore the node's chain reads; see docs/use-dkg/run-node.md
A publication waits one second at most for the retirement of its Shared Working Memory marker (#3179) tailLegacySwmRetireMs on the async_publish_post_finality line now shows what the publication waited: the retirement when it ended within a second, about one second otherwise. The retirement goes on after that and is repeated when it did not get its turn. New lines: Legacy SWM boundary retirement <asset> did not get its turn; repeat <n> in <s> s and Legacy SWM boundary retirement <asset> ended on repeat <n>: <outcome> at info, and Legacy SWM boundary retirement <asset> is not repeated any more (<reason>); its marker stays until the asset is published again at warn None. No setting is added
Reads that prove a graph's registration state take turns with other chain requests (#3194) In the foreground class of the node's chain request budget, such a read and an ordinary request take turns when both are waiting, and background work may take one turn between them, within its own budget. When only one of the two kinds is waiting, admission is exactly as before. A transaction no longer waits out its 10 s deadline behind such reads. Beside one ordinary read a second, a pass of such reads takes about 10 % longer None. No setting is added, and the budget itself is unchanged
A deadline spent waiting for the node's own request budget is reported as local capacity (#3171) The line <step> waited <n>ms for local RPC admission and was not sent now reads <step> exhausted its <n>ms deadline waiting for local RPC admission with no endpoint request in flight. It is also written when earlier requests of the same step had been answered; that case read <step> timed out after <n>ms and counted against the endpoint. Its error code is RPC_REQUEST_GOVERNOR_QUEUE_FULL. GET /api/status gains chain.rpcRequestTiming and chain.rpcRequestGovernor None. Tooling that matches the old text should match for local RPC admission. To size the budget see docs/use-dkg/rpc-request-budget.md
Chain adapter initialization accepts a missing optional Hub contract only when the Hub says it is missing (#3162) A lookup of an optional contract that fails for another reason (a timeout, a throttled endpoint, an answer the node cannot decode) fails the initialization, and the next caller starts it again. 10.0.23 went on without that contract. The default endpoints of the three main networks answer a missing contract in the form the node accepts None on the default endpoints. A node on another provider that does not finish its chain initialization after the upgrade needs an endpoint that passes on the Hub's revert data
A refused query that only aggregates answers one row (#3181) A SELECT whose projection holds only aggregate calls, for example SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }, answered no row when the node refused the read, or when the view resolved no graph. It now answers the one row such a query has over no data: 0 for COUNT, SUM and AVG, an empty string for GROUP_CONCAT, unbound for MIN, MAX and SAMPLE. Other queries keep the answer without rows None. A client that took "no row" from a count for "refused or empty" should read the value
A node runs the libp2p identify push protocol (#3145) A node announces a change of its protocol list to the peers it is connected to. A requester whose record of a connected peer lacks the sync protocols asks that peer directly before it passes it over: a probe of at most 3 s a protocol that sends no application data. A peer on an earlier version does not take the push and is probed the same way None
A Core that serves the batch stream answers a request it cannot serve with a refusal (#3137) ASSET_MISSING, DENIED or BUSY on the stream, where the stream was reset. The codes exist in 10.0.23, so a requester on that version reads them. New lines on such a Core: Exact batch responder refusal stage=<stage> code=<code> and Exact batch inbound opened peer=<suffix> at info, and Exact batch inbound failed peer=<suffix> stage=<stage> error=<name> ... at warn, where <suffix> is the last eight characters of the peer id None
Catalog status keeps a promised head pending (#3161) phase under rfc64Catalog.contextGraphs[] in GET /api/status reads known-incomplete, where it could read complete, while a head that a peer promised is not applied. expectedCatalogHeadDigest is null when two heads compete at one version None. Tooling that waits for complete can wait longer, for a state that was not complete before either
Every assertion write records an AssertionWritten activity (#3164) Seven rows or more for each write in the Context Graph's _meta graph, in the same insert as the payload. Assertion history gains written events; the lifecycle state is unchanged None. metadata.provenanceEvents: false leaves them out, as it does the other lifecycle events
The dashboard shows Node unavailable when its node does not answer (#3160) The demo data that filled the dashboard after failed status checks is gone. The view replaces every route until the page is reloaded None. Use Retry connection once the node answers again
A cg-shared-v1 catalog projection may hold a literal with a tab or another control character (#3209) Such a literal had no form the projection check accepted, so an author that shared an asset with one could not publish its catalog row, nor any later row for that graph's root. A node on an earlier version does not apply a catalog head that contains such a row; it did not receive these rows before either. Merkle leaves and roots are unchanged Upgrade the nodes that should receive such assets. An author with a stuck row publishes it with its next share into that graph's root; a restart alone does not
A receiver accepts a catalog row when its store returns a literal in its own lexical form (#3209) A node on Blazegraph refused a row with a whole-second xsd:dateTime, an integral xsd:double or xsd:float, or a short xsd:duration, with catalog reconciliation failed ... cause=NQUADS_BYTE_LIMIT_EXCEEDED or exact SWM post-read differs, and the graph stayed at catalog-reconciliation-failed. The read-back is now compared in the canonical term form None. A graph that was stuck applies the rows with the next catalog head it receives
A share into a public Context Graph also goes to the peers that follow the graph through a relay (#3208) A peer that asked this node for the graph's shared memory or for its catalog heads is a follower for 30 minutes and is sent each share directly, also over a relayed connection that has at least 16 MiB and 5 minutes of its allowance left. The fan-out line gains followers=<n>; GET /api/peer-info reports allowance for a relayed connection None
An Edge keeps its relay reservations on the relays its network lists (#3206) A node with a relay list no longer reserves a slot on a relay outside it, and its circuit fallback tries every listed relay within the time the first four took. New line at info, at most once per peer in 10 minutes: Relay reservation skipped: peer=<suffix> offers a relay but is not in this node's relay set. While a slot is free the node keeps looking for a relay, as it did before whenever it was short None for an Edge. An operator of listed relays restarts at most one of a network's listed relays at a time, so that no Edge is short of relays
A subscribe asks every peer it selected for the graph's shared memory (#3205) A catch-up job fetched from three peers and ended the rounds of the others without a request. It now waits for the metadata pool, at most ten seconds a transfer. A new subscription of a public graph that is not registered on chain reports synced within seconds where it stayed unsynced, and its first write is accepted. One job sends more metadata requests than before None
The peer that served a graph's seed is asked first for its definition (#3212) For a public graph that is not registered on chain. An empty answer from a peer that does not hold the graph is logged as Peer <suffix> holds no definition of "<graph>" at info, where it read Rejected curator metadata snapshot ... missing an unambiguous public definition at warn None. Tooling that counted the warning sees fewer of them
A sender's Sender Key epoch for a private graph is replaced once after the upgrade (#3203) One setup message per recipient, once per graph and sender agent, the first time the node shares into the graph. Shares that overlap get consecutive message indexes None
A catch-up job waits for the graph's policy to become active (#3217) Once per job and for at most 10 s, only for a graph whose public policy the node has accepted and not yet activated. New line at info for a job that waited: Catch-up for "<graph>": the selected SWM lane became active after <n> ms, or is still not active after <n> ms None
An Edge node can sign its EVM transactions with an external signer (#3177) Off by default: without a chain.signer block nothing changes. With chain.signer.type: "remote" the node signs through standard Ethereum JSON-RPC (eth dialect, Web3Signer-compatible, or Clef) over HTTP(S) or a unix socket, verifies every result and retries nothing; it still builds, records and broadcasts every transaction. Author and payer keys can both live in the signer. With a signer, the Core role, publisher.enabled and the mock chain are refused at start, and admin-key operations and dkg set-ask fail with an explicit error. POST /api/agent/register accepts address for an agent whose key the signer holds None, unless you want it: see docs/use-dkg/external-signer.md
The query-catalog read can be limited to one catalog and answers up to 2 MiB (#3235) POST /api/profile/query-catalog/read accepts catalogSlug (a nonempty string of at most 512 characters; anything else answers 400 QUERY_CATALOG_INVALID_FILTER) and then reads that catalog only. The serialized response limit is 2 MiB, up from 1 MiB, because the response carries canonical items and legacy bindings together None

Known issues

  • RFC-64 catalog replay loop: replays that come back incomplete repeat on
    every reconnect. Keep rfc64Catalog.rollout.killSwitch on wherever it is
    set.
  • A node updated from git can report its build as -dirty (#3169): a
    source build is reported as <commit>-dirty when its checkout holds
    untracked files, and the git updater can leave an untracked folder in
    a release slot by itself, when it sets stale package copies aside. No
    tracked file differs from the commit in that case, and the updater
    keeps working. Nodes installed from npm are not affected.
  • An author's catalog for a Context Graph holds 1,024 assets (#3134): the
    catalog is one signed bucket of at most 1,024 rows. A further new asset
    of that author in that graph is published, finalized and stored as
    before, but the catalog does not list it. Its placement is refused once
    and parked, and the node reports it: a catalog_full warning when a
    catalog first refuses and then at most one an hour, for the placement
    and for the share-time projection each, and
    rfc64Catalog.authorCatalogCapacity in GET /api/status. A newer
    version of an asset the catalog holds is still placed, and a removal
    frees a row. A member on the default configuration keeps receiving such
    assets, with a delay: in a test on a public testnet a member of a
    private graph with 1,050 assets from one author held all of them, and in
    later tests on that graph a further verified asset reached the member
    hours after its confirmation. By the receiver's rules, a member that
    takes a private graph through catalogs stops receiving an author's
    updates once that author's catalog cannot list one of the author's
    finalized assets; this is read from the code and was not reproduced.
    Capacity beyond one bucket is tracked in #3134.
  • Under a burst of publications the author's catalog can trail them by
    minutes
    (#3128): a publication no longer waits for its placement, and
    placements still run one at a time, in a fixed order. In a test on a
    public testnet with the catalog at 700 to 880 rows one placement took 6
    to 21 s at the median, and the catalog trailed a burst of 60
    publications by up to about five minutes in one run and by under 20 s in
    another. pending and oldestPendingAgeMs under
    rfc64Catalog.finalizedPrivatePlacementQueue in GET /api/status show
    the backlog.
  • A finalized publication into a private graph can stay out of the
    author's catalog
    (#3128): a publication is terminal once its placement
    is recorded, and in three cases, all older than this release, a
    confirmation ends with nothing recorded. The catalog lane cannot be
    resolved at the confirmation, which is logged as
    RFC-64 catalog authority was unavailable; the graph has no catalog
    lane at that moment, which writes no line; or the write of the placement
    marker fails, which is logged as
    finalized-private placement was not recorded, so nothing owes it. The
    asset then reaches the catalog only through the projection that its
    share requested, which the node retries when it fails. A fourth case
    concerns one version: when an assertion is re-opened for editing twice
    while the placement of its published version is still queued, and the
    node then restarts, that version cannot be placed. Every attempt writes
    catalog_private_repair_failed with kind: integrity and
    stage: seal, at most once a minute, and the placement stays counted
    in pending until the catalog holds a later version of the asset. Both
    are tracked in #3128.
  • A head replay to a peer that connects during a burst of publications
    can be rejected
    (#3129): a replay is rejected when a catalog change
    overlaps it, before its snapshot or while it is sent, and during a burst
    that is most replays. The requester asks again. Such a peer converges
    when the burst pauses, through a later replay or the next delivered
    head.
  • A queued publication can still wait several seconds after finality on a
    node with peers connected
    (#3128): the wait in the retirement of the
    asset's marker is gone (#3179). In the test of that change on a public
    testnet, what was left was the clear of the published asset's shared
    graph, 0.6 to 1.5 s at the median and about 10 s at the most, the
    lifecycle stamp, up to 2.9 s, and the admission of the recovery turn, up
    to 5.8 s. With no peer connected a publication waited 0.9 s at the median
    after finality. The fields tailPublishedGraphClearMs,
    tailLifecycleStampMs and recoveryAdmissionMs of the
    async_publish_post_finality line show them (#3136). There is no fix yet.
  • After a start a node may not serve its ontology graph for hours
    (#3157): a node whose ontology graph holds Context Graph bindings that
    it cannot classify from its own data keeps the graph from peers until it
    has classified them against the chain, at most 16 new bindings a pass. On
    test nodes that took from one minute to about three hours after a start,
    longest on a node whose store was under pressure. There is no time limit:
    a node whose chain reads keep failing does not serve the graph. A peer
    that asks is told to try again later, and other graphs are served as
    before.
  • Under load a subscriber of a public Context Graph may hold a newly
    verified asset later than before
    (#3179): a publication no longer waits
    for the retirement of its marker, so a publish job ends sooner. In tests
    under load on a public testnet a subscriber held newly verified assets
    later, counted from their chain confirmation, than without the change. The
    assets arrived in every run.
  • The write-ahead log of a managed Oxigraph store grows until the store is
    reopened
    (#2395): the maintenance reopen is off by default, so only a
    restart of the node or a recovery of the store truncates the log. In our
    tests the log grew by 0.25 to 0.6 GiB an hour on nodes with a light write
    load, and by 2.9 and 4.9 GiB an hour on two nodes under a mixed publish
    and sync load. A large log takes disk space and lengthens the next start,
    which replays it. The node reports the log from 4 GiB on. Restart the node
    at a time of your choosing, or set
    store.options.walRestartThresholdBytes and accept about a minute and a
    half of refused store work for each cycle.
  • A subscriber of a public Context Graph can receive a new asset minutes
    late
    (#3052): while a node applies a catalog update for a graph it
    refuses Shared Working Memory writes for that graph. A sync round that
    overlaps the update writes none of the snapshots it fetched, and the node
    does not pull that graph again at once: the assets arrive with a later
    round or through the catalog, in our tests 6 to 7 minutes after they were
    shared. An asset that was finalized in the meantime is then fetched on its
    own, and those fetches can fail and be retried minutes apart: on a graph
    of a few hundred assets that was being published to, single assets took up
    to 15 minutes. Nothing is lost, and this is not new in this release.
  • A member that restarted before its empty private graph was proven ready
    stays not ready
    (#3109): the readiness proof of an empty registered
    private graph now runs again when the member's copy of the metadata
    becomes authoritative (#3108). A node that restarted after that moment and
    before the proof succeeded keeps the graph subscribed and not writable
    until the member subscribes again or content arrives. Members that reached
    this state on an earlier version are in the same position after the
    upgrade. Subscribe again.
  • A read by a newly approved member of an unregistered private graph can
    answer a retryable 503
    (#3064): before it reads such a graph, the
    member's node has to accept a proof that the graph's name is not
    registered on chain. One slow or discarded run of that proof no longer
    fails a read (#3074), but in the seconds after an approval a read can
    still answer 503 with CONTEXT_GRAPH_READ_AUTHORITY_UNAVAILABLE. Send the
    read again.
  • A member of private graphs that are not registered on chain reads the
    chain much more often than other nodes
    (#3064, #3184): the node proves
    very often that the name of such a graph is not registered: one periodic
    refresh asks the chain about every such graph on each pass. These reads no
    longer keep a transaction waiting until its deadline, because such a read
    and an ordinary request now take turns when both are waiting (#3194), and
    a read that accepts bounded freshness no longer adds scans of its own
    (#3183). The reads themselves are as frequent as before. They still use a
    large part of the node's chain request budget, up to every other permit
    when ordinary requests are waiting, so other chain work on that node is
    slower.
  • Recovery through the batch stream depends on the Cores that serve it:
    only Cores with the switch set serve the stream. A Core that is busy
    answers BUSY and is asked again after a short pause, and a stream that
    breaks is opened once more with the same Core (#2979, #2987); a stream that
    stalls without closing is not detected. After a busy attempt, and after a
    broken one on a Core that has already served the graph, the Core keeps its
    turn, but between two stream attempts the node can probe peers without the
    stream. Each such probe is bounded to two minutes while a stream-capable
    Core is connected, three times out of four (#2978), and large assets still
    take minutes each there. Serve the stream from more than one Core that
    holds the graph.
  • The catalog of an unregistered private graph can stay unverified on a
    member
    (#2993): after the approval, data and reads reach a member Core
    within seconds and a member Edge after about a minute, but the member's
    catalog status reaches complete only after about eight minutes on a
    Core and may not reach it on an Edge, because the curator gives up its
    replay to the approved peer before the member is ready. Delivery and reads
    do not depend on it. These timings were measured before #3036, which
    retries an incomplete private catalog within a short budget; they were not
    measured again.
  • A node on an earlier version does not learn the protocols a peer
    registers after their connection opened
    (#2854): a node on this release
    announces such protocols to its connected peers, and it asks a connected
    peer directly before it passes the peer over (#3145). A node on 10.0.23 or
    earlier does neither. It learns a peer's protocols when the connection
    opens, and two such nodes that connected before either had registered its
    handlers recover when they reconnect, as before.
  • A member added to a private Context Graph only by wallet address does not
    receive its Shared Working Memory until it joins
    (#2862): the member needs
    the graph's metadata, which the curator sends on an approved join request.
    Have the member send a signed join request; a pre-authorized member is
    approved automatically.
  • A restarted member Edge may not reconnect to its private graph's curator
    Edge if the curator is unreachable
    (#2865): approval now carries a
    validated listener hint and recovery retries the curator connection
    (#3058). This does not make an unavailable curator reachable. If automatic
    discovery cannot establish the connection, connect to a reachable curator
    multiaddr with POST /api/connect.
  • Copies already stale before the upgrade may wait for another update
    (#2866): the update lane initially replays about 500 blocks. Older stale
    copies, and refresh targets given up after 24 hours, need the next update
    or an explicit asset fetch. The rate-limited backfill is not in this
    release.
  • The successor of an abandoned shared update draft is shared under the same
    number
    (#2964): peers' share gate, the RFC-64 catalog and curator
    confirmation (swmAwaitCuratorAck) do not replace a draft with another of
    the same number, so peers may keep showing the earlier draft until the
    update is published, and the replacement replaces the private payload
    sealed under that number.
  • A catalog authority refresh can wait for the next five-minute pass on a
    node with constant store writes
    (#2968): a refresh that is composed while
    the node's authority facts change is composed again, up to six more times
    within about four seconds. On a node whose unrelated writes never pause,
    every attempt can be discarded, and a newly approved member then receives
    the catalog only after the next periodic pass.
  • Some lifecycle pointer stamps are still only logged when they fail
    (#2962): the Verifiable Memory stamps after a confirmed publish, and the
    shared pointer stamp of publishAsync between its share and its publish.
  • A share from one member of a public graph to another member it has no
    connection with is not delivered
    (#1611, #3211): the owner receives it
    and nothing carries it on. Members that reach each other only through the
    owner's relays are in this layout.
  • A request over a relayed connection still sets up a second circuit that
    is closed at once
    (#3211): the request then goes out on the open
    connection (#3216), but each one costs a circuit on a relay.
  • A subscriber that does not get a graph's definition is not asked again
    until the next subscribe call
    (#3207): when no connected peer holds it,
    or the request fails for a reason other than the one #3216 removes.
  • A subscription to a public graph without content stays unsynced when one
    of the peers it asked failed in transport
    (#3207): such a graph is marked
    synced only when every asked peer answered, and the job is not repeated.
    It becomes synced when content arrives or with the next subscribe call;
    until then a write answers CONTEXT_GRAPH_NOT_WRITABLE.
  • Blazegraph can return the delete character U+007F as another character
    (#3209): a catalog row whose literal holds it is refused on such a node.
  • Overlapping first shares into a private graph set up one epoch each, and
    a recipient that misses more than 64 shares of an epoch refuses the later
    ones until the epoch is replaced
    (#3202).
  • A catalog filter given a slug as a read returns it finds nothing when the
    slug needed encoding
    (#3235, #3239): a read returns a catalog's slug
    percent-encoded, as its IRI stores it, and the filter encodes the value
    again. Slugs made from a name by the node ([a-z0-9-]) are not affected.
    Pass the slug as it was written.

Fixed

  • A queued publication into a private Context Graph is terminal once its
    catalog placement is recorded
    (#3081, #3127): a confirmed publication
    into a private graph stayed non-terminal until the first attempt at
    placing its asset in the author's catalog had ended. Placements run one
    at a time on a node, so a job waited behind the placements of the jobs
    before it, for work that does not decide whether the publication
    succeeded. The job is now terminal once finality is observed, the
    verified content is stored, the placement marker is stored and the
    placement is requested. The catalog supervisor owns the attempt, its
    retries and its drain at shutdown as before, and a marker that was not
    placed before a restart is placed after it. A synchronous publish answers
    at the same point. A recovery pass finalizes every proven job, where each
    waited behind the placement of the one before it, and the finalize of a
    held failed job no longer keeps the queue's node-wide claim lock while a
    placement runs. In a test on a public testnet, with one private graph,
    queued publications three at a time and the author catalog at 700 to 880
    rows, a burst of 60 publications took 1,313 s on the previous code, and a
    publication waited 816 s at the median after finality (1,122 s at the
    90th percentile). That run was the bad case, in which one placement held
    the catalog lock. With this change and #3133, #3136, #3141 and #3143 the
    same burst took 430 to 630 s, and a publication waited 2.7 to 9.8 s at
    the median (5.6 to 21 s at the 90th percentile). In a later run with
    #3130 as well, 30 publications into a full catalog all finalized,
    waiting 0.8 s at the median after finality (1.9 s at the 90th
    percentile, 3.8 s at the most).

  • A catalog placement no longer reads and verifies the whole catalog for
    each asset
    (#3072, #3081, #3133): placing one confirmed asset read and
    verified the whole applied catalog twice, for the coverage check and
    again under the catalog lock, and verified every row a third time while
    it produced the successor. The node now keeps the verified state of a
    catalog's applied head in memory. It uses that state only while the
    durable applied-head record is unchanged, and it reads and verifies the
    head and its delegation from the durable store on every use. A row whose
    exact bytes were verified earlier in the same process is not verified
    again. A confirmation that is observed a second time, or whose row is
    already in the catalog, is answered once the head's history and that
    row's bundle have been read back, without the bundles of all the other
    rows. The signed head, directory node and bucket are byte for byte what
    the previous code signs for the same inputs. In the change's benchmark on
    one machine, a placement into a catalog of 700 rows took 7.83 s with the
    memory switched off, which is the earlier behaviour, and 2.04 s with it;
    at 1,024 rows it took 11.8 s and 3.00 s. The first placement into a
    catalog after a start costs what it did. On the previous code, in a test
    on a public testnet, a burst of 60 publications took 287 s with the
    catalog at 30 to 90 rows and 969 to 1,129 s at 390 to 690 rows.

  • A new catalog head is delivered outside the catalog change, and only to
    peers the node's own policy admits
    (#3081, #3130): a catalog change
    committed its head and then waited, still holding the catalog's lock,
    until its peers had been told, one peer at a time and each with a budget
    of 10 s. A lane with no configured peers addressed every connected peer.
    For a private graph most of those are not members: the node refused each
    one locally, sent it nothing and logged the head as a failed announce at
    warn level. The change now hands the committed head to a delivery owner
    and returns. One owner per catalog asks the policy about every peer and
    then sends the newest head in waves of 16, under one budget for all the
    sends. A refused peer is left out and is not a failure. A peer the
    policy check could not answer for is sent nothing, counted on its own
    and logged at warn. A newer head replaces one that is still waiting, as
    long as it stays within 1,024 versions of the head sent before it: a
    receiver fetches the head it is told about and proves the lineage across
    the versions it skipped, which receivers do since 10.0.15. Receivers
    older than 10.0.15 accept only the direct successor of the head they
    hold and were not tested with this release. A head that was not
    delivered reaches a peer with the catalog's next head or through
    replay. With one peer that never answers, two changes of one catalog
    were placed after 10.5 s and 20.8 s before, and after 0.4 to 0.5 s and
    0.65 to 0.75 s now.

  • A head replay to a connecting peer no longer holds the catalog locks
    while it sends
    (#3081, #3129, #3141): a node replays its catalog heads
    to a peer on every new connection, on the peer's request and after a
    join approval. The replay took the lock of every catalog it replayed and
    sent head after head inside those locks, so a peer that did not answer
    held every change of those catalogs for up to 10 s a head. A replay now
    prepares its manifest under the locks and sends it after they are
    released. When the sends have ended, the inventory is compared once more
    and the replay is rejected as before if it moved. In a test on a public
    testnet a placement waited up to 392 s for the catalog lock or the state
    read under it before the change. With the change the longest such wait
    in two bursts of 60 publications was 28 s in one and 0 s in the other. A
    replay that a catalog change overlaps is still rejected; see Known
    issues.

  • A new asset for a full author catalog is refused once, parked and
    reported
    (#3134, #3143): an author's catalog for a Context Graph is one
    signed bucket of at most 1,024 rows. The placement of a further new
    asset failed inside the successor producer, was reported as
    catalog_private_repair_failed with kind unknown and neither graph nor
    author, and was retried for ever at intervals of up to 60 s. Every retry
    read and verified the whole catalog, and every publication after the
    catalog had filled added one more such placement. The asset is still
    published and finalized as before. The node now refuses its row before
    it builds anything, keeps the placement's marker and parks it: a pass
    over parked placements reads one row a catalog and attempts nothing.
    Parked placements are attempted again when the catalog shows a free row,
    and one of them every hour. A newer version of an asset the catalog
    holds is placed as before. The refusal is written as a catalog_full
    warning and counted in GET /api/status. This adds no capacity; see
    Known issues. In a test on a public testnet 170 queued publications took
    a catalog of about 880 rows across the cap. All 170 finalized, with a
    median wait of 3.2 s after finality, and the 26 placements past the cap
    stayed parked and were not retried.

  • A vector search over Working Memory covers the asking agent's own
    entries
    (#2652, #3150): embeddings of Working Memory now record which
    agent wrote them, and a vector search filters Working Memory by that
    identity before it ranks. Shared Working Memory and Verifiable Memory
    keep their graph-wide scope. The local vector store is migrated at
    start, and its existing rows are kept. Working Memory embeddings written
    before the upgrade have no recorded writer and no longer appear in an
    agent's vector search; a search by the node operator still reads them.
    The authorization checks of a Context Graph and of its Shared Working
    Memory, and the text search, behave as before.

  • A chain index start that could not complete is started again (#3115,
    #3116): at start a node builds its chain index from the deploy block of
    every indexed contract. Each read of that search has a 4 s deadline that
    also covers its wait in the node's own request queue. One read still
    queued at its deadline, or a head probe that no endpoint answered, ended
    the start with a line that begins
    [chain] one-log chain index disabled:, and the node ran without the
    index until its next restart. A start that fails for a reason another
    attempt can change (the node's own request capacity, a timeout, a
    throttled or failing endpoint, a network error) is now repeated after
    5 s and then at doubling waits of up to 60 s, until the index attaches.
    It logs one line when it is deferred, one every 30 minutes while it
    stays deferred, and one when the index comes up. A failure another
    attempt cannot change ends the start as before, with the same line.

  • Random-sampling proof and challenge transactions are sent with more gas
    headroom
    (#3117, #3122): a proof transaction was sent with its bare gas
    estimate. The proof settles the node's stake to the time of its block,
    and an estimate taken on the block that holds the node's challenge skips
    that work, so the same call needs more gas from the next block on. Such a
    transaction was mined and reverted, and the next tick sent the proof
    again: no proof was missed, and each case cost one failed transaction. On
    four nodes this happened in 3 of 12, 7 of 12, 5 of 13 and 8 of 9 proof
    periods of one morning. A proof is now signed with its estimate plus
    50 %. A challenge creation already had that headroom, but what a creation
    costs depends on which Context Graphs the draw of its block lands on, and
    the block it is mined in can need several times the estimate. It is now
    signed with its estimate plus the larger of 50 % and 2,500,000 gas. In
    23.6 days of one test network's public history 557 of 5,336 challenge
    creations failed, every one out of gas. Replayed on a local copy of that
    network's state, 38 of 300 creations failed with the old limit and one
    with the new, in a block where no limit succeeds. In 9.3 days of one
    main network's history none of 4,791 creations failed.

  • A graph whose reconcile pass was refused by the node's own request
    limiter is asked again within seconds
    (#3028, #3123): a Verifiable
    Memory reconcile pass that met a chain read the node's own request
    limiter had refused ended there, and its graph waited for its turn in
    the periodic sweep, with its chain-event nudges dropped meanwhile. The
    sweep goes round the bound graphs four at a tick, so on a node with 40
    graphs that turn came ten minutes later. It happened above all right
    after a start, when background reads are held back: a member that had
    missed one asset while it was down still lacked it nine minutes after
    its start. The graph is now asked again 2.5 s later, one graph at a
    time, for up to five minutes; after that the pass ends as it did and the
    sweep retries it. A read that was sent and then failed or timed out, and
    an answer from the chain, are handled as before.

  • A graph whose reconcile pass was overtaken by an update of its
    subscription is asked again
    (#3135): a reconcile pass works on the
    graph's subscription row as it was when the pass began and stops when
    the row has been stored again. Ordinary work does that on a running
    node: a subscribe request for a graph the node already holds, catch-up,
    or a sync round. The pass then ended as failed, with the warning
    VM reconcile for "<graph>" failed; retrying on the periodic sweep: VM reconcile queue is closed for node shutdown
    on a node that was not shutting down, and the graph got no further pass
    until the periodic sweep, minutes later. Such a pass now leaves the
    graph to be asked again 2.5 s later, on the row as it is then. In a test
    with a publisher and a subscriber node, 39 of 40 published assets were
    in the subscriber's Verifiable Memory at the first check with the fix,
    and 11 of 20 without it.

  • The checks of saved subscriptions after a start no longer take the
    node's whole chain request budget
    (#3118, #3119): after a start a node
    checks the saved subscriptions that its activation cap left dormant. It
    checked them back to back, and on a node with several hundred such rows
    that took the whole chain request budget for minutes after every start,
    while the node's other chain reads waited or were refused. A check that
    activates nothing is now followed by a pause of at least 5 s, at least
    four times the time the check took and at most 30 s. A row that
    something asked for is checked first, then rows with a saved chain id,
    then the rest. In a test process with 571 saved rows, 565 of them not
    confirmed by the chain, the checks made 10 chain reads a second for two
    minutes before the change and 1.3 a second after it, and the node's
    ordinary foreground reads waited up to 88 s for admission before and at
    most 1 ms after. The last row was checked 44 minutes after the start,
    where it was 2 min 39 s.

  • Rolling activation of saved subscriptions keeps its cap and its
    wakes
    (#3125, #3156): rolling activation could exceed its cap when
    another request took the last slot during an authority read, and it
    could miss a wake that was requested as a pass ended. The pass now
    checks the cap again after its reads, and a wake that arrives as a pass
    ends is kept. The pacing of the checks uses a monotonic clock, so a
    change of the wall clock cannot lengthen a pause.

  • Join metadata is fetched again when the fetch after a live approval
    fails
    (#3109, #3110): when a join is approved on a running node, the
    member fetches the curator's metadata in at most two attempts and then
    asks for a broadcast catch-up, which does not carry private metadata. If
    both fetches failed in transport, nothing fetched the metadata again
    until the node restarted, and every write of the member was answered
    CONTEXT_GRAPH_NOT_FOUND. The member now goes on with the bounded
    recovery that start-up already uses: one attempt at once, then after
    15 s, 30 s, 1 min and 2 min, and every 5 min from there, until the
    metadata is confirmed, the approval is withdrawn or the node stops.
    Metadata that the node refused because it names another registration
    than the one its own row holds is not fetched again. Fixes a 10.0.23
    known issue.

  • Gossip reconcile and catalog responsibility keep their state when an
    authority read gets no answer
    (#3042): two places that act on an
    authority check reduced it to yes or no, so a chain read that got no
    answer counted as a refusal. In the Shared Working Memory gossip
    reconcile a subscription the node holds now stays, and one it does not
    hold is not made. For the catalog of a private graph a responsibility
    the node holds as a member stays. The graph is asked again shortly, one
    graph every 2.5 s. A refusal and every other answered result take effect
    at once, as before.

  • The Shared Working Memory gate no longer drops a valid envelope because
    another graph's metadata changed
    (#2968, #3011): the gate took its
    snapshot of a graph's metadata again when a node-wide revision had
    moved, and gave up after two attempts. A metadata write to any other
    graph moves that revision, so on a busy node both attempts saw a change,
    the gate failed with
    metadata authority kept changing while resolving its SWM gate and the
    envelope was dropped. The gate now compares the graph's own
    authority-facts revision. A change to the graph itself still takes a
    fresh snapshot.

  • Concurrent callers share one chain adapter initialization (#3043):
    every caller that found the chain adapter uninitialized ran its own
    initialization of about 18 chain reads, in its own request class. After
    a reset of the contract bindings, every caller that arrived before the
    first one had finished repeated the reads, and the copies waited behind
    each other in the node's own request queue. One initialization now
    serves all the callers that wait for it, in the foreground class. On a
    local chain a node ran 8, 12 and 19 initializations at once in three
    starts before the change and one in each start after it, and the first
    registration after the reset took 1.8, 3.0 and 10.2 s before and 0.11 to
    0.13 s after.

  • A read that another endpoint serves no longer spends the first
    endpoint's re-probe
    (#3022, #3152): when the first chain endpoint
    refuses a kind of read, that read is served by another endpoint. Such a
    read used up the re-probe that was due for the first endpoint, although
    the first endpoint was not asked. The re-probe is now used only when the
    first endpoint is tried, so it stays available to the next ordinary
    read.

  • A dial by peer id alone has a deadline (#2864, #3149): the fallback
    that finds a peer by its id passed on only the caller's signal, which
    switched off the transport's default limits, and a Verifiable Memory
    reconcile could wait on it without end. The fallback now runs under a
    deadline, and the caller's wait ends there even when the transport
    ignores the cancellation. A dial that completes after its deadline does
    not count as a success.

  • A slow identity probe is retried once before the peer is suppressed
    (#2860, #3154): a peer whose identity responder was busy for longer
    than the 3 s deadline went into exponential suppression before it could
    prove its network. A probe that ends on a transport deadline is now
    retried once with a response deadline of 15 s, and later suppression is
    capped at 3 s while the peer has an open connection. The validation of
    the signed proof, quarantine and the back-off for disconnected peers
    are unchanged.

  • A subscribe or unsubscribe that fails on an authority outage returns
    an operation id
    (#2843, #3151): the response now carries an operation
    id, and exactly one server log line matches it. The subscription error
    code, the generic response body and the retry headers are unchanged.

  • A failed asynchronous share reports the transition that was stored
    (#2896, #3155): after a failed attempt the worker read the job's outcome
    in a separate step, which could fail or see a later attempt. The queue
    now returns the exact terminal or retrying transition with the write
    that stored it, and bookkeeping that could not be resolved is returned
    as an uncertain result, not as a fatal one.

  • A publication no longer waits for the retirement of its Shared Working
    Memory marker
    (#3081, #3179): after a confirmed publication the node
    clears the asset's copy in Shared Working Memory and retires the marker
    that the share left for it. The retirement took its turn on one node-wide
    chain, shared with the retirements and the receiver leases of every graph.
    A receiver lease is held while the node proves and commits a catalog
    change from a peer, which can take tens of seconds, and the publication
    waited for the retirement. So a publication into one graph waited behind
    work on other graphs, for up to 30 s. At that bound the retirement gave
    up: nothing was retired, and nothing came back for the marker. The
    retirement of one asset now waits only behind work of its own graph, and
    it answers at once when no marker of the asset is outstanding. Its caller
    waits one second at most. The retirement goes on without the caller, and
    one that did not get its turn or failed is repeated: after 30 s, then at
    doubling pauses of up to ten minutes, ten times at most. A node that stops
    ends the retirements that still wait and waits for one that is at work.
    What is still owed at a stop is dropped; a marker that is left goes with
    the next confirmed publication of its asset, or with a catalog projection
    that covers the asset. In a test on a public testnet, with bursts of 30
    queued publications into a private graph on a node with peers connected, a
    publication waited 25 to 32 s at the median after finality before the
    change and 2 to 4 s with it. What still waits is under Known issues.

  • Reads that prove a Context Graph's registration state take turns with
    other chain requests
    (#3064, #3194): the node's own chain request budget
    admitted a read that proves a graph's registration state ahead of every
    ordinary request, without a bound. On a node where such reads arrive as
    fast as the budget admits them, nothing else was admitted. A transaction
    then waited for a permit behind reads that had each arrived after it,
    until its 10 s deadline, and was not sent: in 10.0.23 the line is
    <step> waited 10000ms for local RPC admission and was not sent, which
    #3171 rewords. When both kinds are waiting they now take turns: neither
    waits for more than one admission of the other, and background work may
    take one turn between them, within its own budget. Whenever only one of
    the two kinds is waiting, admission is exactly as before. In a workload of
    such reads issued back to back beside one ordinary read a second, on the
    default budget, a pass of the reads took 165 s where it took 150 s, about
    10 % longer, and an ordinary read was admitted after about 0.2 s where it
    had waited out its deadline and was not sent. The change does not make
    these reads fewer; see Known issues.

  • A bounded read of an unregistered graph name is answered without a
    provider scan
    (#3064, #3183): some reads of a graph's name accept
    bounded freshness; the read-authority check behind queries and Shared
    Working Memory sync is one. When such a read found the name not
    registered, it paid for a scan of the chain endpoint almost every time:
    about nine requests a read whenever the chain had moved since the last
    one. Such a read is now answered from the chain log the node already
    follows. Where that log cannot answer, on a node without its chain index
    or with an index that has fallen behind, one scan a tick serves the
    bounded reads, where each read scanned. A read that asks for the live
    state is unchanged, and a registration is seen as soon as the node's index
    has committed it. This takes one source out of the reads that #3194 is
    about. It is not what ended the waiting of transactions there, and the
    live reads are as frequent as before.

  • A subscribe that changes nothing no longer stops a member's first
    store
    (#3182): a member that has just been approved for a private graph
    fetches the graph's published assets, authenticates each against the chain
    and stores it. The store runs under the graph's subscription record and
    stops when that record has been replaced. Every subscribe call installed a
    new record, also a call that changed nothing, and so did a catch-up result
    that only updated readiness. A client that repeated its subscribe while it
    waited stopped the store that way, with the error
    Context graph binding for <graph> changed during graph-scoped store. The
    verified asset was dropped, the catch-up reported the graph unreachable,
    and the member got the asset later by another route. A subscription write
    that changes nothing now keeps the stored record, and a store continues
    when the record was stored again with the same admission, lifetime, chain
    binding and metadata state. A record that was unsubscribed, that was
    deleted and subscribed again, or that differs in one of these four still
    stops the store. A Verifiable Memory reconcile pass is no longer overtaken
    by such a subscribe either (#3135).

  • A refused query that only aggregates answers the row it has over no
    data
    (#3181): a read the node refuses is answered with an empty result
    of the query's form. For a SELECT whose projection holds only aggregate
    calls, such as SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }, that was no
    row, although such a query has one row also when nothing matches: a count
    of 0. A client that asked for a count had nothing to read. The node now
    answers that row, as it does for a graph that holds nothing: COUNT,
    SUM and AVG are 0, GROUP_CONCAT is an empty string, and MIN, MAX
    and SAMPLE are unbound. The same holds for a view that resolves no
    graph, for example the Working Memory view of an agent with no draft. The
    row is made from the query text; the query is not run. It is made only
    when the WHERE group needs stored data to have a solution. A query that
    has solutions without data (inline VALUES, BIND, an OPTIONAL on its
    own) keeps the answer without rows, and so does a query with GROUP BY,
    HAVING or an expression around an aggregate. For those a refused read
    still differs from a read of an empty graph.

  • Daemon read routes authorize an agent token against the Context Graph it
    asks about
    (#3172): a read route of the daemon now checks a request made
    with an agent token against the Context Graph the request names. A request
    for a graph the agent may not read is refused with 403. When the node
    cannot establish the answer, the request is answered 503 with
    Retry-After and can be sent again. A listing of publisher jobs made with
    an agent token names its Context Graph: GET /api/publisher/jobs takes
    contextGraphId, and dkg publisher jobs takes --context-graph. The
    node operator's token and a node with authentication disabled read as
    before.

  • The ontology graph is served to peers only when every Context Graph
    binding in it is classified
    (#3157): a node keeps its ontology graph
    from sync, changelog sync and remote queries until a pass has classified
    or moved the Context Graph metadata that earlier builds left in it
    (#2787). That now also holds for a binding the node cannot classify from
    its own data. The pass at start makes no chain reads and leaves such a
    binding unclassified, and the graph stays closed until later passes have
    classified every binding against the chain, at most 16 new ones a pass.
    What a pass has proven is kept for the next one, so a large graph finishes
    over several passes. A later pass that fails, is cut short or leaves a
    binding unclassified closes the graph again. There is no time limit: a
    node whose chain reads keep failing does not serve the graph. See Known
    issues and docs/use-dkg/run-node.md.

  • A node tells its connected peers about protocols it registers after they
    connected
    (#2854, #3145): a node records a peer's protocols when the
    connection is identified, and a node registers its sync handlers after its
    transport has started. A peer that connected in between recorded the node
    without sync support and passed it over, for sync and for the fetch of a
    single asset, although the connection was live. This was seen on a main
    network after a Core restarted. A node now also runs the libp2p identify
    push protocol, so a change of its protocol list reaches the peers it is
    connected to. A peer on an earlier version does not take the push. So a
    requester whose record of a connected peer lacks the sync protocols now
    asks that peer directly before it passes it over, with a bounded probe
    that sends no application data. The readiness checks at the start of a
    catch-up run four at a time, where they ran one after another.

  • A batch stream request that cannot be served is answered with a
    refusal
    (#3137): a Core that serves the exact-asset batch stream reset
    the stream, before the first byte of its answer, when the requested asset
    had no confirmed export on that Core, when it refused the request, or when
    it could not read the graph's authority or its store at that moment. The
    requester saw a broken stream in each case. The Core now answers on the
    stream, with refusal codes the stream already has: ASSET_MISSING for an
    asset without a confirmed export, DENIED for a request it refuses, and
    BUSY when the authority cannot be read, the store is under pressure or
    the Core is not serving that graph for now. A refusal can follow assets
    that were already sent: those stay committed, and the asset in progress is
    not acknowledged. Malformed metadata, a root that does not match and a
    source that changes during the export still end the stream. The Core logs
    each refusal by stage and code, and a failed inbound stream by stage,
    without request data.

  • A deadline spent waiting for the node's own request budget is not
    counted against the endpoint
    (#2949, #3171): a step of several chain
    requests under one deadline, in a publish for example, could have one
    request answered and then spend the rest of its deadline waiting for a
    permit of the node's own request budget. That ended as a timeout of the
    endpoint: the node failed over and de-preferred an endpoint that had
    answered. A deadline that runs out while a request waits for a permit and
    no request is in flight now ends as local capacity, with the code
    RPC_REQUEST_GOVERNOR_QUEUE_FULL. No endpoint is blamed and none is tried
    next, because every endpoint shares that budget. A request that is in
    flight when the deadline runs out times out as before. The line reads
    <step> exhausted its <n>ms deadline waiting for local RPC admission with no endpoint request in flight.
    A step that had sent nothing at all was already taken for local capacity,
    and its line read
    <step> waited <n>ms for local RPC admission and was not sent.

  • Catalog status keeps a promised head pending when its provider stops
    answering
    (#2826, #3161): the per-graph catalog status in
    GET /api/status lost a head that a peer had promised in a replay once
    that peer stopped answering. When the promised head did not hold more rows
    than the applied one, because it replaced rows or removed the last one,
    the graph read complete, with the old catalog version and the digest of
    the head already applied as the expected one, although a newer signed head
    was owed. Promised heads now count as pending. Such a graph reads
    known-incomplete until the head is applied, and applying only while a
    live target is pending. The expected version, digest and row count come
    from the newest head of each author's catalog, and the expected digest is
    null when two heads compete at one version.

  • The Context Graph enumeration finishes one run of ids before it asks for
    the next
    (#3026, #3158): the enumeration reads Context Graph ids from
    the chain in runs of 32, with one aggregate request a run and single reads
    for a run whose aggregate request fails. When an early id could not be
    read, the requests for later runs had already been sent, and their answers
    were discarded. Each run, with its single reads, is now resolved before
    the next is requested. The ids returned, the pinned block and the bound of
    four readers are unchanged.

  • A refusal for a full catalog keeps the remembered catalog state
    (#3134, #3159): the node forgets the verified state of a catalog when a
    change to that catalog fails (#3133). The refusal of a new asset for a
    full catalog (#3143) counted as such a failure, so the next placement into
    that catalog read and verified every row again: once an hour, and once
    after a restart. The refusal is decided from the row count of the state
    the change has just read, before anything is built or written, so the
    state is now kept. Every other failure forgets it as before. The saving
    was not measured on a catalog at the cap.

  • The dashboard shows Node unavailable when the node does not answer
    (#1133, #3160): when the page's first status checks failed, the dashboard
    filled in demo data, while health, balances, graphs and integrations kept
    looking live. Every route now shows a Node unavailable view with a
    Retry connection button, which reloads the page. A status answer keeps
    the dashboard, and a 401 keeps the token prompt.

  • Shares into a private Context Graph that overlapped took the same message
    index, and recipients refused later shares
    (#3203). Recipients answered
    with a transient rejection and logged Sender Key message nonce does not match ratchet state; past 64 skipped messages they refused every later
    share of the epoch. The assets still arrived once finalized. Present since
    10.0.0.

  • An asset whose text holds a tab or another control character blocked its
    author's catalog for that graph
    (#3198, #3209). The author logged
    catalog_repair_failed with kind: unknown on every attempt, and neither
    that asset nor any later one shared into the graph's root reached other
    nodes.

  • A node on Blazegraph refused catalog rows with common typed literals
    (#3198, #3209): whole-second date-times, integral doubles and floats, and
    short durations, which that store returns in another lexical form of the
    same value.

  • Two Edge nodes that reach each other only through a relay did not deliver
    a share into a named sub-graph to each other
    (#3198, #3208). Share
    recipients of a public graph were the peers subscribed to its gossip topic,
    and gossip does not run over a relayed connection.

  • A request to a peer reached only through a relay failed whenever its dial
    ended in a closed connection
    (#3198, #3216), with Remote closed connection during opening. Sync requests, which are sent once, never used
    the relayed connection that was already open. A subscriber's request for a
    graph's definition to the one peer that holds it failed in most tries.

  • A subscribe to a public Context Graph asked only three peers for its
    shared memory
    (#3198, #3205), since 10.0.17. With four or more peers the
    subscription of a graph without content never reported synced and
    refused writes with CONTEXT_GRAPH_NOT_WRITABLE.

  • A subscriber could leave the peer that holds a graph's definition out of
    the peers it asked
    (#3198, #3212), on a node with more than eight
    connected peers.

  • An Edge could move a relay reservation to a relay outside its network's
    list and then not get it back
    (#3206), after a listed relay restarted;
    the node then announced an address that other nodes of its network refuse.

  • A new subscription whose graph definition arrived within a second or two
    never reported synced
    (#3198, #3217): its catch-up asked every peer
    before the graph's policy was active on the node, and nothing repeated it.
    A catch-up job now waits for that, once, for at most ten seconds.

  • A member of a private Context Graph stopped publishing its catalog after
    the roster changed again
    (#3220): for example when a second member was
    approved. The member's metadata then held two roster generations, its
    authority for the graph stayed incomplete with RFC-64 private roster metadata has conflicting generations, and the curator received none of
    that member's writes into the graph's root. The newest generation is now
    read.

  • A local-model chat failed when a request exceeded the model's context
    (#3235): when the model endpoint rejects a request for its context size
    before inference, the chat now retries without the earlier conversation
    history, and then from the current tool results without tool schemas; at
    most twice, and completed tool calls are not repeated. A domain profile can
    name requiredTools; the chat then refuses to start when one of them is not
    available.

Changed

  • A publish into a private graph resolves its recipients and its curated
    context once
    (#3069, #3086): each Verifiable Memory publish resolved
    the curated context twice, for payload and for chunk encryption, and
    concurrent resolutions of a private roster repeated every per-agent key
    lookup. One context is now resolved per publish attempt, with one policy
    probe and one sender-key setup pass where there were two of each. Key
    lookups are shared between concurrent callers and remembered for one
    second; the memo is bypassed while a key or route write is pending.
    After a sender-key setup the recipient authority is read once more, and
    the resolution starts over when the roster or a route changed meanwhile.
    An authority that still moves after three setups fails the attempt with
    a typed authority failure, which a queued Knowledge Asset publish
    records as authority_unavailable and retries.
  • A missing asset version view names the endpoints that failed (#3097,
    #3099, #3106): after a publish is confirmed the node reads the asset's
    version from the chain at one pinned block, from the first endpoint and
    then from the others. An endpoint that failed that read was passed over
    without a word, and when no endpoint served it, recovery deferred on
    every tick with the same message and no cause. The read now keeps which
    endpoint failed, by its position in the configured list and its host, at
    which step, and a closed class for why. No URL and no provider error
    text is kept. The recovery deferral, the exact-asset fetch and the
    Verifiable Memory refresh check append that cause to their messages. The
    remains pending warning is logged once per asset when a reason appears
    or changes and then as one summary per reason every 5 minutes, where it
    was logged on every tick. When no endpoint has served the read for 5
    minutes and at least 12 deferrals, one line that begins
    Operator action needed: says what is not finalizing, why, and what to
    do. What each caller decides is unchanged.
  • A node built from source keeps the identity of its build (#2852,
    #3153): a source build captures the commit it is built from before it
    compiles and stores it only when the build succeeds, and status keeps
    reporting that commit after the checkout has moved on. A build from a
    changed or unidentifiable tree is not certified: its commit in
    GET /api/status ends in -dirty, also when the tree only holds
    untracked files. A failed compilation keeps the earlier outputs and
    their metadata. When the auto-updater builds from source, it clears
    stale build metadata in its inactive slot first, and a failed clean-up
    or build leaves the active slot untouched. The devnet scripts rebuild
    stale executables.
  • The dashboard says when a size is a preview (#1015, #1026, #3147,
    #3148): a memory layer that was capped or unavailable looked like a
    complete graph size. Rows and the total now show a loaded preview, with
    + on an entity count that is a lower bound and ~ on a triple count
    that can still change, and tooltips explain the cap and the
    uncertainty. The triple counts of Working, Shared Working and
    Verifiable Memory are labelled as stored triples, and the Verifiable
    Memory count as data plus provenance.
  • The managed Oxigraph store reports its retained write-ahead log, and a
    maintenance reopen is available, off by default
    (#2395, #2571, #3195): a
    managed store (store.backend: oxigraph-server) keeps its write-ahead log
    for as long as the store process runs; only a reopen, such as a restart of
    the node, replays and truncates it. The node now measures the retained log
    once a minute and reports it in its log when it first reaches 4 GiB and
    again each time it has doubled; a confirmed reopen of the store starts
    these reports over. By default nothing else changes. A maintenance reopen
    is available and off: it runs only when
    store.options.walRestartThresholdBytes is set to a positive integer, a
    size in bytes. Any other value is ignored with a log line, and a node with
    the option set says at start which way it runs. With the reopen on, the
    node refuses all new store work once the log reaches that size, waits for
    the work in progress to end and for 30 s without any, reopens the store
    and then leaves it alone for an hour. In our tests one cycle refused store
    work for about 75 to 95 seconds, and store-backed API calls answered 503
    with outcome: "not_started" meanwhile. Also part of this change: the
    managed store is restarted when an update, and not only a query, runs past
    its client deadline. A catalog compare-and-swap whose confirming read
    fails is reported with an unknown outcome and invalidates cached
    revisions, a change that commits and then fails in its bookkeeping is
    flagged for reconciliation, and an abandoned read keeps its claim on a
    restart until a restart is confirmed. Described in
    docs/use-dkg/storage-sparql-http.md.
  • Chain adapter initialization accepts only a typed absence of an optional
    Hub contract
    (#2913, #3162): initialization looks up several contracts
    that a Hub may lack and goes on without the ones that are missing. It took
    a failed lookup for a missing contract unless the node itself had
    interrupted the lookup, so a lookup that failed on the endpoint left the
    node without that contract until the next initialization. Only the Hub's
    own answer that the name is not registered now counts as absence. Any
    other failure fails the initialization, and the next caller starts it
    again. The default endpoints of the three main networks answer a missing
    contract in the form the node accepts. A missing required contract fails
    initialization as before.
  • Catalog production yields the main thread, and a confirmed placement
    reads one inventory row
    (#3134, #3175): the producer of a catalog's
    successor verified the predecessor and built the replacement bucket in one
    turn of the main thread. It now takes its rows aside and gives the thread
    a turn between the two. A confirmed placement that has work to do read the
    author's whole local inventory to find one asset; it now reads that row by
    its key. The row is a hint: the seal and content checks still decide a
    placement. The signed objects are byte for byte what they were, and the
    capacity of a catalog is unchanged.
  • The refresh of a Verifiable Memory copy after an update is a fetch
    policy of its own
    (#2882, #3166): a refresh carried its need for a fresh
    responder session as a separate flag through Shared Working Memory hosting
    and durable sync. It is now one policy, vm-update-refresh, beside the
    ordinary and the challenge-pinned ones: an ordinary fetch resumes its
    session, and every peer attempt of a refresh starts a new snapshot. What a
    node does is unchanged. For callers of the agent API,
    forceFreshExactSession is still accepted and is deprecated, a request
    that selects assets no longer takes the whole-graph changelog lane, and a
    requester context of the older positional form refuses a refresh.
  • A Knowledge Asset update event keeps one identity from the chain to the
    refresh queue
    (#2885, #3168): the update event of an asset now carries
    one typed identity, made of its block, block hash, log position and
    transaction as far as the chain adapter reports them, through the adapter,
    the replay filter of the event lane, the lifecycle callback and the
    Verifiable Memory refresh queue. Before, each of these read the fields for
    itself. Two updates with the same root, an update whose block was
    replaced, and an event that comes again with a block position it lacked
    the first time are each delivered. An exact replay is still suppressed,
    and the other event lanes keep their replay handling.
  • Publisher wallet funding is assessed for each candidate wallet (#2869,
    #3167): the check that picks a funded publisher wallet now keeps, for each
    candidate, whether it is funded, not funded or could not be read. When no
    candidate is funded at the first look, the candidates are re-priced one
    after another, and a wallet that was topped up while an earlier candidate
    was re-priced is read again before its own decision. The order of
    selection, the codes NO_FUNDED_PUBLISHER_WALLET and
    PCA_FUNDING_UNKNOWN and their messages are unchanged.
  • Test and CI plumbing (#3131, #3132): the timing baseline from which
    CI plans the ten agent test shards is refreshed. 264 of the 644 agent
    test files had no measured time and were planned at 60 s each, so a
    branch that added a test file could move the heavy files into one shard,
    which was then cancelled at its 10-minute limit although no test had
    failed.

Added

  • External signer for EVM signing (#3177): an Edge node can delegate the
    signing of its transactions to an external signer, for example an HSM-backed
    one, and hold no operational private keys. Authors and payers are separate
    keys; payers covered by a Publishing Conviction Account need no TRAC, and a
    node whose author keys live in the signer can register a curated context
    graph backed by a PCA, into which every agent of the PCA can publish.
    Configuration and limits: docs/use-dkg/external-signer.md.
  • Timing of a queued publication's wait after finality (#3081, #3120,
    #3136): one async_publish_post_finality line per finalized queued job,
    at info when the wait is 10 s or more and at debug otherwise. It names
    the path the job took (inline, detached, held-failed or
    interrupted) and splits the wait into tailMs, recoveryAdmissionMs,
    recoveryRetryMs, proofMs, recoveryMs and terminalWriteMs, which
    sum to totalMs. tailMs, the executor's work after the confirmation,
    is split again by step: tailPublishMs, tailReceiptWriteMs,
    tailPublishedGraphClearMs, tailLegacySwmRetireMs,
    tailRemainingSwmClearMs, tailLifecycleStampMs, tailGraphIdReadMs,
    tailFinalizationGossipMs, tailShareMarkerClearMs,
    tailCatalogObserverMs and tailRestMs. A part whose step did not run
    prints -. One rfc64_catalog_placement_wait line is written for a
    catalog placement that takes 5 s or more, with requestMs, queueMs
    and attemptMs, and the attempt by phase (coverageMs, assetMs,
    stateMs, successorMs, casMs, otherMs). Its announceMs, peers,
    failedPeers and deniedPeers are always 0, because a head is no
    longer delivered inside an attempt (#3130). The lines of one job carry
    [from:<jobId>]. The marks are kept in memory, so a restart loses them.
  • Author catalog blocks in GET /api/status (#3120, #3127, #3130,
    #3143): three blocks under rfc64Catalog, with counts only and no
    graph, author, asset or peer. finalizedPrivatePlacementQueue counts
    the placements that are owed (depth, pending, oldestPendingAgeMs,
    waiters, oldestWaiterAgeMs, passRunning, lastPassDurationMs,
    cooldownSkips). authorCatalogCapacity counts the catalogs that
    refused a new row for lack of room and the placements parked for them
    (scopesAtCap, parkedPlacements). catalogHeadDelivery counts head
    deliveries since the node started (handoffsQueued, handoffsNobody,
    handoffsNotQueued, reportedHeads, deliveredPeers, failedPeers,
    refusedPeers, uncheckedPeers, unconfirmedPeers, supersededHeads,
    undeliverableHeads, checkpointCapacityExceeded). A parked placement
    is still owed, so a full catalog shows in the first block as a backlog
    that does not drain: pending minus parkedPlacements is what waits
    for an attempt.
  • chain.versionSnapshot in GET /api/status (#3097, #3099): the
    endpoints that failed the asset version read, by position and host. An
    endpoint that reads pass over stays on record while they succeed on
    another one.
  • DKG_RFC64_CATALOG_MUTATION_MEMORY (#3133): set to 0, it switches
    off the in-memory catalog state, and every placement reads and verifies
    the durable catalog as before. Described in docs/use-dkg/run-node.md.
  • An out-of-gas hint on a reverted write (#3117, #3122): a receipt
    carries neither the gas limit nor revert data, so the error of a write
    that was mined and reverted gave no hint of the cause. When at most 1/32
    of the gas limit was left, the message now ends
    it used <n> of its <m> gas limit, so it probably ran out of gas. V10
    publish and update transactions have their own receipt handling and do
    not get the hint.
  • chain.rpcRequestTiming and chain.rpcRequestGovernor in
    GET /api/status
    (#2949, #3171): rpcRequestTiming holds histograms
    since the node started, for the foreground and the background class:
    admissionWait for waits that got a permit of the node's own request
    budget, admissionDeadlineWait, admissionCancelledWait and
    admissionRejectedWait for waits that ended on a deadline, on the
    caller's cancellation or on a refusal, and endpointLatency with
    endpointFailures. rpcRequestGovernor is the budget's current log
    window: its rates, the tokens available, the queue depths and the counts
    of admitted, deferred, rejected and cancelled requests. Reading status
    sends no chain request. docs/use-dkg/rpc-request-budget.md is new: it
    says how to size chain.rpcRequestBudget and how to read these fields.
  • An AssertionWritten activity for every assertion write (#674,
    #3164): a successful assertionWrite records an activity in the Context
    Graph's _meta graph, in the same store insert as the payload: the
    writer, the time, the assertion, every Working Memory graph that received
    payload and the number of payload quads in the call. Repeated writes keep
    separate activities. Assertion history returns them as written events;
    the lifecycle state of the assertion does not change. With
    metadata.provenanceEvents: false they are not written.
  • dkg okf verify --sub-graph-name (#1419, #3165): with
    --sub-graph-name <name>, dkg okf verify reads that sub-graph's Shared
    Working Memory for every integrity count and for the missing-concept
    query, as dkg okf export does. Without the option it reads what it did.
    Described in docs/use-dkg/okf.md.

Full Changelog: v10.0.23...v10.0.24