Bulk changes: batched push frames and batch commits (0357) - #563
Merged
crs48 merged 8 commits intoJul 18, 2026
Merged
Conversation
…nature-over-many (0357) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Adds verifyFast/verifyMany in @xnetjs/crypto backed by WebCrypto Ed25519, which is available in both Node 18.4+ and browsers (Baseline) — one seam, no node:crypto import, no bundler special-casing. Measured ~13x faster than the pure-JS @noble verifier (101us vs 1374us). Routes the three bulk verification paths through it: hub relay, hub NDJSON restore, and .xnetpack bundle verify. Interactive single writes keep the synchronous verifyChange. Deliberately NOT Ed25519 batch verification — that checks the cofactored equation and can disagree with single verification on adversarial input (ZIP-215), which would let replicas split convergence. Signed-off-by: xNet Test <test@xnet.dev>
Push was strictly one change per WebSocket frame at a 40 msg/s client throttle, so a 10k-change bulk import took 4+ minutes just to transmit. Adds a node-change-batch frame carrying up to 1000 changes, negotiated via a new batch-push handshake feature (older hubs keep the one-frame-per-change path). A batch is a TRANSPORT batch, never a trust batch: the hub verifies, authorizes, and stores every change individually through the same pipeline, re-broadcasts accepted ones as ordinary node-change messages so batch-unaware subscribers are unaffected, and reports rejections per change (each tagged with its hash) rather than dropping the whole frame. Rate limiting gains a second, independent budget: frames count against perConnectionRate, changes against perConnectionChangeRate (default 5000/s). Charging a 1000-change batch as 1000 messages would have made batching pointless; not counting changes at all would have made a batch frame an unlimited bypass. Two budgets bound the two real costs. Signed-off-by: xNet Test <test@xnet.dev>
applyRemoteChanges and .xnetpack import verified each change with a cold pure-JS Ed25519 call (~1.4ms each, the cost 0344 measured). Both now verify a whole slice up front via verifyRemoteChanges, which shares the native-support probe and one key import per distinct author. Adds an internal preVerified option to applyRemoteChange that skips ONLY the crypto — ledger enforcement, authorization, dedup, and LWW still run per change, covered by a test asserting a denied change is still refused when preVerified is set. Signed-off-by: xNet Test <test@xnet.dev>
A bulk operation stays N changes, one per node: that is what makes per-node parent chains, per-property LWW, and selective sync work. But signing/verifying each one individually is the dominant cost, and it is avoidable — every mature system (Hypercore, Certificate Transparency, AT Protocol) hashes every unit and signs only at a batch boundary. Adds BatchCommit: an ordered list of change hashes, a BLAKE3 root over them, signed ONCE. Verifying a batch is one signature check plus N hash recomputations a verifier already owes. Membership rules keep it from being weaker than per-change signatures: a change must hash to its claimed hash, appear in the commit's list, and share the commit's author, and the commit itself must verify (editing the list and recomputing the root still fails, since the signature covers the commit hash). Additive: the Change record, its hash recipe, and LWW ordering are untouched, so the conformance kernels gain vectors rather than a re-derivation. Spec lands as L1 6.1; golden vectors cover the root, canonical bytes, signature, and three tamper cases; the Python reference kernel reproduces all of it byte-for-byte (29/29). Signed-off-by: xNet Test <test@xnet.dev>
Export now emits commits.ndjson covering the owner's own changes (one commit per 1000), and import uses them: a change covered by a valid commit from its own author is authenticated by that commit's single signature and skips its own signature check, keeping only the hash recomputation every change already owes. Commits cover only the owner's changes, because a commit must not be able to vouch for a change another author signed. A bundle with several authors commits the owner's and leaves the rest on the per-change path. The commits entry is folded into the manifest content digest, so a swapped or truncated commits list is caught like any other entry. Bundles without commits.ndjson (every bundle written before this) import exactly as before. Tests assert the path is genuinely live: with commits the signature verifier receives zero changes; without them it receives all of them. Signed-off-by: xNet Test <test@xnet.dev>
…0357) Adds the throughput floor (10k changes ingested end-to-end through batch frames in 570ms / 17,544/s, against a 30s budget), client negotiation tests proving an older hub never sees a batch frame and that a reconnect re-negotiates, and batch-commit support in the Swift reference kernel so all four conformance kernels cover the new record. Also records two implementation deviations in the exploration: commit-covered changes keep their signature at rest (dropping it would leave an imported change unable to re-enter the live relay lane), and the hub uses two independent rate budgets rather than one weighted one. Signed-off-by: xNet Test <test@xnet.dev>
Signed-off-by: xNet Test <test@xnet.dev>
Contributor
|
Preview removed for PR #563. |
crs48
deleted the
claude/0357-bulk-changes-one-signature-over-many-and-batch-e
branch
July 18, 2026 21:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements exploration 0357.
The question
If you import, delete, or transactionally touch a large data set — is that one change or many? And can the signature/hash/envelope work be batched?
Answer: many changes, deliberately. One
Changecarries one node, and that is what makes per-node history, per-property LWW, and selective sync work. What was not batched was everything around it.What was actually slow
Not the crypto — the wire. Push was one change per WebSocket frame at a 40 msg/s client throttle, so a 10,000-change import took 4+ minutes just to transmit, no matter how fast the hub was.
What landed
Tier 1 — no protocol change
node-change-batchframes (up to 1000 changes), negotiated via abatch-pushhandshake feature; older hubs keep the one-frame-per-change path.verifyFast/verifyManyin@xnetjs/crypto, backed by WebCrypto Ed25519 — available in both Node 18.4+ and browsers, so one seam covers hub and client. 101µs vs 1374µs measured..xnetpackimport andapplyRemoteChangesbatch-verify a whole slice instead of paying a cold verify per change.Tier 2 — additive protocol record
BatchCommit: one Ed25519 signature over a BLAKE3 root of up to 1000 ordered change hashes. Verifying a batch is one signature check plus the hash recomputations a verifier already owes..xnetpackbundles carrycommits.ndjson; importing a self-export does zero per-change signature verifications.Measured
.xnetpackself-export importSafety
Batching is transport and authentication only. Every change is still verified, authorized, share-grant checked, quota counted, and LWW applied individually. Specifically:
node-changemessages — batch-unaware clients see exactly the stream they expect.perConnectionRate, changes againstperConnectionChangeRate. Charging a 1000-change batch as 1000 messages would make batching pointless; counting only frames would make it an unlimited bypass.Two documented deviations from the plan
Also found
A pre-existing sync bug, unrelated to this change and tracked separately: the pull path caps at 1000 changes but the client fast-forwards its cursor to the room-wide high-water mark, so a cold catch-up of >1000 changes in one room can skip rows.
Testing
🤖 Generated with Claude Code