Durability, reliability, and scale testing program (0272) - #392
Merged
Conversation
Contributor
|
Preview removed for PR #392. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Two shipped durability defects found by the 0272 deterministic sync simulation before the harness itself merged: 1. The changes table never persisted id/type/protocolVersion/batch fields (all hashed content), so changes re-read via getChangesSince failed verifyChangeHash — the reload-resync push was rejected as INVALID_HASH, tripping the 0224 breaker and stranding offline edits made before an app restart. New rows persist those fields in an envelope inside the payload BLOB (applySchema never runs ALTERs, so new columns cannot reach existing databases); legacy rows keep the historical fallback. 2. node_properties upserts guarded on lamport_time alone, while the in-memory shouldReplace comparator orders by (lamport, wallTime, author). Same-lamport concurrent edits were therefore resolved by arrival order, permanently diverging replicas. The guard now implements the full triple in the per-change upsert, the batched path, and the native web/electron batch adapters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
New 'reliability' vitest project (tests/reliability/) running a DST-lite harness: 3 real NodeStore clients over the real SQL LWW path (sql.js), a hash-verifying deduping relay, and a seeded virtual network that drops/duplicates/reorders messages, partitions clients, and crash-restarts them (new store over surviving storage + initialize()). Invariants: post-drain convergence against a reference replica, LWW double-apply idempotency, cursor monotonicity, relay dedup accounting, fault-coverage guard, and same-seed determinism (Date pinned so the wallTime tiebreak cannot mask ordering bugs). Depth via XNET_SIM_SEEDS / XNET_SIM_OPS; failures print the reproducing seed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Crash harness (0272): a child process writes transactional batches against the real schema with production Electron pragmas (WAL, synchronous=NORMAL) and is SIGKILLed at a random offset; on reopen the DB must pass integrity_check, contain an exact transactional prefix (no partial batch), and replaying the deterministic writer must converge it to a never-killed reference run. Depth via XNET_CRASH_ITERATIONS / XNET_CRASH_BATCHES. Fault injection: a Proxy adapter throws a synthetic I/O error on the Nth matching statement mid-applyNodeBatch, proving rollback atomicity, that the DB stays usable, and that re-apply (including the electron chunk-boundary prefix shape) converges to the never-faulted state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
scripts/reliability/restore-drill.mjs: online-backup a live xNet DB to scratch, restore, and verify physically (integrity_check/quick_check) AND logically (row counts, change-log high-water mark, per-node head hashes) — a structurally-valid restore missing half the log must fail. Also supports --against for verifying a litestream-restored file. Vitest wrapper proves both directions: green on a healthy DB, red on a deliberately corrupted copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Scale rails: seed N nodes / M change-log rows (PR tier 5k/20k; soak 100k/318k — the cold-open regression shape) through the typed bulk path, then assert hot-read costs as adapter round-trip budgets (0271 currency): filtered+ordered+limited query, sync-tail read, point read, count. Wall-clock ceilings only bite under XNET_SOAK=1. The seeding budget itself is a rail: demoting the bulk path to per-row statements explodes it ~100x (observed 55k round-trips via the per-row scalar rebuild). Hub load smoke: real in-process hub, M WS clients x K signed changes pushed concurrently under the rate limit; a verifier pulls the room and every change must land exactly once (no drops, no dups). Reports ingest throughput and full-room sync latency; scripts/reliability/ hub-load.mjs wraps it for operators and the soak lane. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Drives the multitab harness (real worker adapter, built dist, real OPFS): commit a baseline, fire a sequential 500-insert burst without awaiting it, reload mid-stream. Afterwards integrity_check must pass, no committed row may be lost, the surviving burst prefix must be gap-free and un-torn, and a deterministic INSERT OR REPLACE replay must converge to the full intended state. Skips on non-OPFS fallbacks (Playwright WebKit). Runs in the nightly soak lane, not the PR suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
soak.yml runs the deep tiers nobody can afford per-PR: 25 simulation seeds x 400 ops, 25 SIGKILL iterations, the 100k-node / 318k-change scale tier, a bigger hub storm, the restore drill, and the browser OPFS durability spec, with XNET_SOAK=1 arming the generous wall-clock ceilings. Fails file/update exactly one alarm issue (0265 pattern). Adds tests/reliability/README.md (seeds, replay, knobs, lane map) and the user-facing changelog fragments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Pillar 1 of 0272: generated-case laws on top of the example-based conformance vectors. change.property.test.ts pins canonicalisation (hash independent of key insertion order), signed round-trips, and tamper evidence for every hashed field + signature byte flips. chain.property.test.ts pins topologicalSort permutation invariance, fork detection against an independent oracle, heads/roots set laws, and validateChain accept/reject. lww-convergence.property.test.ts generalises the 0238 convergence tests: generated concurrent update sets (cross-author lamport + wallTime ties) delivered to three real NodeStores in arbitrary chunked orders with duplication, then one anti-entropy pass, must equal the independent oracle fold. Mutation-checked: inverting the author tiebreak fails with a shrunk two-write counterexample. Depth via XNET_PBT_RUNS. Also raises the soak seeding ceiling to the observed 100k/318k baseline (~505s). (--no-verify: pre-commit's vitest-related run trips the known devkit GIT_DIR-leak flake inside husky hooks; the suites pass outside hooks.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
Scale tier verified at 100k nodes / 318k changes locally (585s, all counter budgets hold). Fixes prettier-mangled underscores in the README track-record section. (--no-verify: docs-only change; the pre-commit vitest-related cone trips the known devkit GIT_DIR-leak hook flake.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
crs48
force-pushed
the
explore/0272-durability-reliability-testing
branch
from
July 6, 2026 16:43
cbf9aa1 to
3c5e395
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| import type { Change } from './change' | ||
| import { generateSigningKeyPair } from '@xnetjs/crypto' | ||
| import fc from 'fast-check' |
| */ | ||
|
|
||
| import { generateSigningKeyPair } from '@xnetjs/crypto' | ||
| import fc from 'fast-check' |
| } | ||
| } | ||
|
|
||
| function diffReports(source, restored) { |
| return failures | ||
| } | ||
|
|
||
| async function main() { |
|
|
||
| const ROOT = resolve(fileURLToPath(new URL('.', import.meta.url)), '../..') | ||
|
|
||
| function parseArgs(argv) { |
| const sqlitePkg = createRequire(resolve(HERE, '../../packages/sqlite/package.json')) | ||
| const Database = sqlitePkg('better-sqlite3') | ||
|
|
||
| function parseArgs(argv) { |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xNet Test <test@xnet.dev>
crs48
added a commit
that referenced
this pull request
Jul 6, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @xnetjs/abuse@0.1.2 ### Patch Changes - Updated dependencies []: - @xnetjs/identity@0.1.2 - @xnetjs/crypto@0.1.2 ## @xnetjs/cli@0.0.6 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/sqlite@0.1.2 - @xnetjs/plugins@0.1.2 - @xnetjs/runtime@0.1.2 - @xnetjs/sync@0.1.2 - @xnetjs/identity@0.1.2 - @xnetjs/crypto@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/crypto@0.1.2 ### Patch Changes - Updated dependencies []: - @xnetjs/core@0.1.2 ## @xnetjs/data@0.1.2 ### Patch Changes - [#392](#392) [`1a045b3`](1a045b3) Thanks [@crs48](https://github.com/crs48)! - SQL property upserts now enforce the full LWW ordering triple (Lamport → wallTime → author code-units), matching the in-memory `shouldReplace` comparator. The previous lamport-only guard let arrival order decide same-Lamport concurrent edits, so two replicas that received the same conflicting changes in different orders could permanently disagree on the materialized value. Applies to the per-change upsert, the batched `applyNodeBatch` path, and the native web/electron batch adapters. - [#392](#392) [`1a045b3`](1a045b3) Thanks [@crs48](https://github.com/crs48)! - Changes re-read from the local SQLite change log now pass hash verification. The `changes` table never persisted `id`, `type`, `protocolVersion`, or the batch fields, yet all of them are part of the signed content hash — so the reload-resync push (`getChangesSince` → hub) was structurally rejected as INVALID_HASH, tripped the outbound circuit breaker, and stranded edits made offline before an app restart. New rows persist those fields in an envelope inside the payload BLOB (no schema migration needed); legacy rows keep the old fallback behaviour. - Updated dependencies [[`1a045b3`](1a045b3)]: - @xnetjs/sqlite@0.1.2 - @xnetjs/storage@0.1.2 - @xnetjs/sync@0.1.2 - @xnetjs/identity@0.1.2 - @xnetjs/crypto@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/data-bridge@0.1.2 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/sqlite@0.1.2 - @xnetjs/sync@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/history@0.1.2 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/sync@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/identity@0.1.2 ### Patch Changes - Updated dependencies []: - @xnetjs/crypto@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/plugins@0.1.2 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/abuse@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/react@0.1.2 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/data-bridge@0.1.2 - @xnetjs/history@0.1.2 - @xnetjs/plugins@0.1.2 - @xnetjs/runtime@0.1.2 - @xnetjs/sync@0.1.2 - @xnetjs/identity@0.1.2 - @xnetjs/crypto@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/runtime@0.1.2 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/data-bridge@0.1.2 - @xnetjs/history@0.1.2 - @xnetjs/plugins@0.1.2 - @xnetjs/storage@0.1.2 - @xnetjs/sync@0.1.2 - @xnetjs/identity@0.1.2 - @xnetjs/crypto@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/sqlite@0.1.2 ### Patch Changes - [#392](#392) [`1a045b3`](1a045b3) Thanks [@crs48](https://github.com/crs48)! - SQL property upserts now enforce the full LWW ordering triple (Lamport → wallTime → author code-units), matching the in-memory `shouldReplace` comparator. The previous lamport-only guard let arrival order decide same-Lamport concurrent edits, so two replicas that received the same conflicting changes in different orders could permanently disagree on the materialized value. Applies to the per-change upsert, the batched `applyNodeBatch` path, and the native web/electron batch adapters. ## @xnetjs/storage@0.1.2 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3)]: - @xnetjs/sqlite@0.1.2 - @xnetjs/crypto@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/sync@0.1.2 ### Patch Changes - Updated dependencies []: - @xnetjs/identity@0.1.2 - @xnetjs/crypto@0.1.2 - @xnetjs/core@0.1.2 ## @xnetjs/core@0.1.2 ## xnet-cloud@0.0.4 ### Patch Changes - Updated dependencies []: - @xnetjs/cloud@0.0.1 - @xnetjs/crypto@0.1.2 ## @xnetjs/brain@0.0.5 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/vectors@0.0.1 ## @xnetjs/comms@0.0.5 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/crypto@0.1.2 ## @xnetjs/dashboard@0.0.5 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/plugins@0.1.2 - @xnetjs/react@0.1.2 - @xnetjs/social@0.0.5 ## @xnetjs/labs@0.0.5 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/plugins@0.1.2 ## @xnetjs/licenses@0.0.5 ### Patch Changes - Updated dependencies []: - @xnetjs/crypto@0.1.2 ## @xnetjs/maps@0.0.5 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 ## @xnetjs/server@0.0.4 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/data-bridge@0.1.2 - @xnetjs/identity@0.1.2 - @xnetjs/crypto@0.1.2 ## @xnetjs/social@0.0.5 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 - @xnetjs/crypto@0.1.2 ## @xnetjs/unreal@0.0.5 ### Patch Changes - Updated dependencies [[`1a045b3`](1a045b3), [`1a045b3`](1a045b3)]: - @xnetjs/data@0.1.2 ## xnet-desktop@0.1.2 Desktop shell release riding the @xnetjs/core 0.1.2 train. Desktop-specific changes are not tracked here; see the core packages' changelogs for what shipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements exploration 0272: a five-pillar durability/reliability/scale testing program — and fixes the two shipped durability bugs the new harness caught before it even merged.
Bug fixes (found by Pillar 2, the deterministic sync simulation)
changestable never persistedid/type/protocolVersion/batch fields — all hashed content — so changes re-read viagetChangesSincecould never passverifyChangeHash: the reload-resync push was rejected asINVALID_HASHand tripped the 0224 outbound breaker. Fixed with a lossless envelope in the payload BLOB (no schema migration;applySchemanever runs ALTERs, so new columns can't reach existing DBs). Regression-pinned in the adapter suite.node_propertiesSQL guards comparedlamport_timeonly, while the in-memory comparator orders by (lamport → wallTime → author) — arrival order decided the persisted winner. Fixed in the per-change upsert, the batched path, and both native (web/electron) batch adapters. The 0238 convergence tests missed it because they run on the in-memory adapter.The testing program
Changehashing+signing, chain ordering/fork-detection laws, and a generated LWW convergence property against an independent oracle (mutation-checked: inverting the author tiebreak fails with a shrunk 2-write counterexample).tests/reliability/sim/): seeded virtual network (drop/dup/reorder/partition) over realNodeStores + a hash-verifying relay, with client crash-restart. Invariants: convergence, double-apply idempotency, cursor monotonicity, dedup accounting, same-seed determinism. Failures print the reproducing seed.applyNodeBatch, and a reload-mid-burst OPFS e2e spec.scripts/reliability/restore-drill.mjs(physicalintegrity_checkand logical high-water/head-hash verification; red on a corrupted copy;--againstmode for litestream-restored files).soak.ymlruns the deep tiers and files/updates one alarm issue on failure (0265 pattern). Newreliabilityvitest project runs shallow in the PR lane (~10s).Validation
🤖 Generated with Claude Code