resync on broadcast lag and map mqdb conflicts to typed error - #1
Merged
Conversation
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Follow-up to the chorale AWS scale test (10×200 tasks) which surfaced
liveness: FAILagainst the embedded mqdb backend. Root cause split across two crates:Database::update/deleteCAS conflicts could stall single-process embedded callers and silently drop writes on the fjall backend. mqdb added bounded internal retry-on-conflict (32 attempts, field-level last-writer-wins) and an atomic fjall commit. No stitch change is required for the stall itself — this PR is the stitch-side hardening + the remaining liveness gap.Changes
memory_store.rs): map mqdbError::Conflict→ stitch's existingError::Conflict { entity, id }on the two CAS paths (update,delete), so the residual backstop conflicts (mqdb cap-exhaustion / vault path) are classifiable viais_conflict()instead of falling into an opaqueMqdberror.store.rs,config.rs): the internalmutation_looppreviously dropped inbound remote deliveries onRecvError::Laggedandcontinued, causing local divergence + missed scheduler wakeups under high throughput (Finding 2). It now runs an authoritativesync_root_entity_listre-fetch when connected. Defaultevent_channel_capacityraised 1024 → 4096 for high-write scopes.store.rs):handle_remote_mutationreplaced.unwrap_or_default()and thelet _ =cache-mirror writes withwarn-level logging on genuine failures; intentional LWW/scope drops still returnOk(false)quietly.tests/memory_store.rs):concurrent_updates_to_same_record_converge_without_conflict— 16 concurrent same-key updates on a multi-thread runtime must all returnOk, converge to a last-writer value, and preserve the untouched field (field-level merge).Cargo.lock: mqdb-agent 0.8.8 / mqdb-core 0.7.3.Notes
mutation_loop(the background loops hold strongArc<StoreInner>refs, so spawning would add an Arc cycle) — natural backpressure, nothing new to track.sync_root_entity_listrecovers root + child entities for the connected user, matching existing reconnect coverage. Top-level wildcard entities aren't re-fetched here — a pre-existing reconnect gap, not introduced by this PR.Verification
cargo clippy -p stitch --all-targetscleancargo test -p stitchgreen (91 tests, incl. the new concurrency test)liveness: PASS) needs the AWS Fargate harness and was not run locally.