v0.2.0: delta sync — additive wire protocol over CRDT-merged state#52
Merged
v0.2.0: delta sync — additive wire protocol over CRDT-merged state#52
Conversation
Adds Op enum + oplog to Resolver, bumps version via bump_with_ops at every mutation site, exposes apply_weight_updates and apply_ops on NamespaceHandle, and updates /api/sync to serve ops or full-export based on supports_delta flag. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ipped delta sync) The background thread was receiving ops from the server, logging them, and bumping the local version counter — but never applying them to the live resolver. State diverged silently on every delta tick. Changes: - `run_background` gains a second `apply_delta` callback parameter with signature `Fn(&str, &[Op]) -> Result<(), Error>`. The delta branch calls this instead of throwing the ops away. - Version counter is only advanced AFTER successful apply; on failure it stays put so the server ships a fresh full export on the next tick. - `batch_sync` now sends `"supports_delta": true` so the server actually sends ops instead of always falling back to full export. - Call site in `engine.rs` passes a closure that acquires the namespace write lock and delegates to `connect::apply_ops`. - New integration test `tests/delta_sync_e2e_connected.rs` (191 lines): boots a real server, boots a connected-mode engine, verifies "howdy partner" routes to "greet" after a PhraseAdded delta tick, and verifies a WeightUpdates (decay) delta tick also applies cleanly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
apply_ops mutates state via set_weight et al. but never bumps the version counter — that's the server's job. Without an explicit set, client-side Resolver::version() lags by one after every WeightUpdates op even though the data is correct. Add pub(crate) Resolver::set_version(v) and have the connected-mode delta callback set it to the target version after apply_ops succeeds. The callback signature gains a u64 (target_version) param; the existing ConnectState::versions map already had the right value, this just propagates it into the resolver itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…l + /api/snapshot endpoint - `POST /api/sync` no longer includes `export` field; instead returns `cold_start_required: true, version: N` when the client is too far behind (oplog gap) or doesn't send `supports_delta`. - New `POST /api/snapshot` engine-level endpoint: accepts optional `namespace_ids` list, returns full exported state for each namespace in a single round-trip. Unknown IDs silently omitted. - `ConnectState::fetch_snapshot()` replaces per-namespace `pull()` for initial bootstrap; a single HTTP call covers all subscribed namespaces. - `run_background` loop collects cold-start namespaces then calls a single `fetch_snapshot` per tick instead of embedding export in sync. - `engine.rs` initial connect sequence uses `fetch_snapshot` (one call) then enters the normal sync loop. - `tests/snapshot_endpoint.rs`: 3 new tests covering all-namespaces response, explicit-ids filtering, and the cold_start_required → snapshot round-trip; all passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bandwidth-optimized sync transport on top of existing CRDT semantics. Wire protocol is additive: v0.2.0 clients without
supports_delta: trueget full exports unchanged. New clients opt in and get ops only.Implementation
Spec:
_internal/V0_2_0_DELTA_SYNC_SPEC.md— implementation followed it.VecDeque<(u64, Op)>on Resolver, persisted to_oplog.json, ring-buffer max 1000 entriesbump_with_opshelper: everyself.version += 1site migrated (8 sites, 5 files)IntentIndex::get_weight/set_weight— primitives for weight snapshot/diff and idempotent replaysupports_delta+oplog_min_versionrequest fields. Falls back to full export if client too far behind or doesn't opt inapply_opsinconnect/mod.rs+NamespaceHandle::apply_opsVerified
cargo test --lib --release— 60/60 passcargo test --tests --release— 81/81 pass (includes new idempotency + e2e tests)cargo fmt --check+cargo clippy -D warnings— cleanFiles
16 files changed, +1263 / -20 LOC. Bandwidth bench at
benchmarks/delta_sync_bandwidth.py(runs against live server).Decisions noted
update_namespace/set_domain_descriptionnow bump version (was previously a hole — oplog requires it; additive + correct)IntentEdit/NamespaceEditderive Serialize/Deserialize (needed for metadata-updated ops; no existing callers broken)connect/mod.rsdefers inline op application;NamespaceHandle::apply_opsis the canonical delta path. Loop redesign for full delta-in-background can come later — it's a callback-shape thing, not a correctness thing.Closes #108.