Skip to content

federation

Latest

Choose a tag to compare

@avijeetsingh1 avijeetsingh1 released this 09 Sep 07:42

The release that made federation something a stranger can join, and stopped four surfaces describing themselves incorrectly. Nothing here changes the wire format, the receipt preimage or the address space.

Two threads run through it. The first is that a name which does not match the thing is a fault with a delay fuse: a health endpoint reporting bedrock while every call went local, /splats answering 200 while serving an empty directory, a disclaimer crediting a model that does not exist, and an agent card claiming an A2A version that was never released. Each looked correct from one angle and was false from the one that mattered. The second is that a check which cannot run is not a check that passed, which is why two gates in here grew a third outcome.

Added
scripts/verify_node.py: checks an origin against the join bar in federation.md §8e, anonymously, from anywhere. Two roles, because they need different things published: a WITNESS proves it exists, has a stable key and a DNS record binding one to the other; a RESOLVER additionally publishes the registries a reader needs to check the facts it serves. Exit 1 names what is missing.
docker-compose.yml: a node in one command. §9d called out that "anyone can host" was priced by twenty systemd units; none of them are needed to join as a witness. Carries the three steps beyond up -d that actually constitute joining, and says plainly that read federation does not exist yet, so joining buys mutual witnessing and nothing else.
GET /.well-known/did.json: the node's did:web document, responder key and declared witness key (EMEM_WITNESS_PUBKEY_B32) as Multikey. No new cryptography; the keys every DID verifier can now resolve.
GET /.well-known/emem-agents.json: the organisation vouching document, served from config/emem-agents.json. The enlistment ladder fetched this from other domains and this responder never published its own.
federation block in /.well-known/emem.json: node DID, declared peers (EMEM_PEERS), the _emem-node TXT a peer should find, and how witnessed the head is, computed from the same rows as /v1/log/witnesses.
cid_v1 beside fact_cid on recall facts: the same blake3 digest as a CIDv1 (raw, multihash 0x1e), so IPFS, Filecoin and ATProto tooling address a fact without rehashing. Truncated entity and bundle anchors get none.
tree_size on /v1/log/inclusion: prove against a historical head; root_b32 and root_is say what the path reaches.
scripts/witness_peers.py + deploy/systemd/emem-witness.timer: the witness job. Verifies each peer's STH, proves growth from the pinned head, co-signs, spot-checks four sampled leaves for custody, and identifies the signer through the peer's did.json.
verify_node_dns in scripts/witness_peers.py: resolves _emem-node. TXT over DoH and compares v=emem1; k= to the responder key of the STH it just signature-checked. A mismatch is a finding and the head is not co-signed; a missing record is unchecked and does not block. did.json is served by the peer's own web server, so on its own it established only that the box agreed with itself; moving a key now takes the zone as well.
Independence on /v1/log/witnesses and the federation block: head_is_independently_witnessed, freshest_independent_witness_entries_behind, independent_witness_count, independent_cosignature_count, self_witness_pubkey_b32. Computed over every stored row, not the returned page.
count, source and source_path on /.well-known/emem-agents.json, and a startup warning when the roster was not loaded.
did, did_document and federation under responder on the A2A agent card. did.json already listed the card as its #a2a service; the reverse pointer was missing.
scripts/external_witness_lag.py and scripts/perception_liveness.py: probes for two faults that returned a well-formed answer. scripts/hooks/commit-msg moves into the repo, where the documented install line expects it.
Search and a day picker on /channel. 2,494 messages sat in one 4 MB document with no anywhere, so the only way to find one was the browser's own find. Filtering marks .nomatch rather than the .hide the rail's per-agent toggle owns, so the two controls compose; day headings fold away when nothing under them survives. The 42 day separators gain id="d-" and a picker that jumps to them.

on /channel for /v1/inbox and the memory SSE stream: an agent landing on the page had no discoverable route to the signed JSON behind it. --accept-count N on build_channel.py: the degradation guard catches a failed read, and refuses forever after a legitimate drop. Accepting one requires stating the exact count, so a stale invocation fails closed. Changed The fact index and fact bodies move from sled 0.34 to redb (var/emem/facts.redb, beside the sled directory, which keeps the memory trees and side indexes). New facts are written to redb only, durable when the commit returns; reads fall through to the old sled trees until a rate-limited background backfill (resumable cursor, free-disk guard, EMEM_HOT_BACKFILL_*) has copied them, after which sled is never consulted for facts. Every wedge snapshot on 2026-09-03 paired sled fact reads with a stuck flush in the same file. EMEM_HOT_BACKEND=sled is the rollback. EMEM_EXPLICIT_FLUSH=0 disables the per-write fsync in request paths and leaves durability to sled's flusher. Default on; an operator decision, off nowhere by default. A writer waits at most EMEM_FLUSH_WAIT_SECS (default 20) for a flush cycle, then returns an error naming the state (applied in memory, not yet durable) instead of hanging to the request budget. The watchdog's storage probe now also makes one signed write (storage_liveness.py --write): a wedge that stops writes while reads answer was invisible to the read probe for ten minutes on 2026-09-03. Flushes coalesce. flush_off_runtime used to run one whole-database fsync per caller, serialised on a mutex; under the example gate's load six memory writes sat 32 seconds in that queue and a live snapshot showed 33 blocking threads behind one flush. Now a caller asks for a flush that starts after its write, one flush serves every ask queued before it started, and a failed flush still reports to every caller it covered. The watchdog gains a second signal: scripts/storage_liveness.py every minute, three consecutive wedged verdicts plus a five-minute cold-start grace before a snapshot and restart (EMEM_WATCHDOG_DRY_RUN=1 to log only). The store can wedge while /live still answers; on 2026-09-02 that state lasted an hour, every storage-touching request hung, and the watchdog reported healthy throughout. /v1/log/witnesses returns the newest limit rows (default 20, max 200) with count (all rows) and returned, and rows no longer repeat a 250-character sentence each; unbounded, the list crossed the MCP tool-result cap within a day of a witness running every fifteen minutes, and the truncation nulled fields the tool's own schema requires. /v1/log/inclusion refuses unknown query arguments with 400. It used to ignore them, and a witness that passed tree_size got proofs against a head it had not pinned. a2a_async_tasks_spec and the metadata on every A2A message reply reported protocolVersion: "1.2.0", which A2A never released. Both now use A2A_PROTOCOL_VERSION, as the agent card and the channel extension already did, so a client stops reading 1.0 from the card and 1.2.0 from the reply to its own message. /v1/inbox returns truncated and limit, and says to re-request with ?limit=. limit worked and was uncapped, and nothing in the response mentioned it: a caller saw count: 50 against total_matched: 184 with no way to learn the other 134 were one parameter away. build_channel.py says plainly when the JS syntax gate did not run. It printed as a parenthetical, which reads like a gate that ran and passed. The llm-shim spend ledger fails CLOSED. A write failure was swallowed, so the next cap check re-read a stale file, spend stopped accumulating and the cap could never trip: an unwritable ledger meant unbounded billing. A failed write now marks the ledger unhealthy, which counts as spent, and the shim degrades to the local model. /health reports ledger_ok and ledger_file, because a spent cap and a broken counter both serve the local model and an operator has to tell them apart. The echo_verify example_args in the MCP catalogue, the "a recomputed derivation" button on /verify, and the ARM benchmark's self-check token all cited a fact this responder no longer holds. All three dereference it live, so all three taught a chain that stops. The compute-quota error no longer promises larger quotas to high-score attesters; no code granted them. /.well-known/emem-agents.json falls back to $EMEM_DATA/emem-agents.json after the relative config/emem-agents.json. The relative path resolves inside a container whose WORKDIR is / and whose image does not ship config/, so a missing file was served as {"agents": []} and read by every peer's ladder as a deliberate empty list. Not baked into the image: the document attests that the operator of a domain vouches for these keys. live_perception.cameras_with_retained_clips reads upstream's clips_retained instead of counting clips inside the returned cameras page, which is cameras_returned of camera_count: 5 of 267 at Trafalgar Square. The mcp-protocol-version header on an initialize response mirrors the body's negotiation rule. A client offering a version newer than ours was told 2025-11-25 in the result and 2025-03-26 in the header of the same response. ops/systemd/emem-watchdog.sh takes EMEM_SYSTEMCTL instead of a hardcoded systemctl --user, which finds nothing and exits clean against a system unit, leaving the watchdog reporting that it had handled a stall it never restarted. The hot store opens sled with an 8 GiB pagecache and a 200 ms flush interval (EMEM_SLED_CACHE_BYTES, EMEM_SLED_FLUSH_MS) instead of sled's defaults. With a 1 GiB cache in front of 58 GB, reads pulled pages from the log and waited on the fsync of the current buffer; the wedge snapshots of 2026-09-02 show 147 threads in that wait and the watchdog restarting the server eleven times in a day.