Jepsen Scheduled Stress (Redis): duplicate-elements + future-read + G-single/G2-item-realtime on main

[bootjp]

Jepsen Scheduled Stress (Redis workload) — duplicate-elements + future-read + G-single/G2-item-realtime on main

Summary

The Jepsen Scheduled Stress Test run #27033231956 (Redis list-append workload, scheduled cron on main) failed validation with a hard anomaly set. The prior scheduled stress run on main at 12:57 UTC the same day (#27016167818) passed.

The single commit landed on main between the two scheduled runs is PR #928 — strictly inside internal/backup/ (S3 backup encoder); it does not touch the Redis adapter, the Raft engine, the coordinator, or any path the Redis workload exercises. So this is not introduced by PR #928 — it is a pre-existing edge case that the stress harness occasionally exposes.

Per-PR Jepsen runs (jepsen-test.yml) use --time-limit 5 --rate 5 --concurrency 5. The scheduled stress uses --time-limit 150 --rate 10 --concurrency 8 --key-count 16 --max-writes-per-key 250 --max-txn-length 4 — ~30× wall-clock time, 2× throughput per client, longer transactions. The duplicate / future-read patterns surface only at that scale.

Verdict (from the failed run)

{:valid? false,
 :anomaly-types (:G-single-item-realtime
                 :G2-item-realtime
                 :cycle-search-timeout
                 :duplicate-elements
                 :future-read)}

Notable anomaly entries:

Type	Evidence
:duplicate-elements	{:duplicates {187 2}} on op index 525, process 1, :f :txn reading key 9 = [1 2 3], appending [13 50], reading key 15. Element 187 appears twice in some list.
:future-read	op index 1073, process 18, transactional read returned a list whose tail contained values not yet appended in real-time at the read-start instant.
:G-single-item-realtime	first witness at op index 1667 (~84.2s into the run); second around 1669. SCC traversed; cycle confirmed.
:G2-item-realtime	op index 2797 (~137.7s into the run). Multi-key anti-dependency cycle.
:cycle-search-timeout	{:scc-size 495 :anomaly-spec-type :G-single-item-realtime} — Elle exhausted its budget exploring one SCC. Indicates the history has many candidate cycles to check, consistent with the realtime violations above.

The two anomalies that are most diagnostic are :duplicate-elements and :future-read — these are not "noise" from real-time analysis alone:

• :duplicate-elements in a list-append workload means the same value appended twice to the same list. The standard cause is client retry of a transaction that actually committed at the server but whose ACK was lost (timeout / connection close / leader change mid-RPC). Without an idempotency token or matching commit-deduplication path, the retry produces a second commit.
• :future-read = a successful read returned a value before the value's write started in wall-clock real-time. Combined with :duplicate-elements, the most likely explanation is the same retry-committed-twice value being observable on a path that is not tied to the first write's commit timestamp.

Both are consistent with an ambiguous-commit window: the Redis adapter, the Lua bridge, or the txn dispatch returns an uncertain error (io.unknown) on the path back to the client, the client retries, the original commit had already been applied → duplicate append + tail of the list reordered relative to other clients' reads.

Reproduction profile

• Branch: main, HEAD 00a8d0bc ("backup: Phase 0b M4-2b impl - S3 collision-rename reversal (backup: Phase 0b M4-2b impl - S3 collision-rename reversal #928)").
• Workflow: .github/workflows/jepsen-test-scheduled.yml (display name Jepsen Scheduled Stress Test).
• Workload: elastickv.redis-workload (list-append, Elle checker).
• Args: --time-limit 150 --rate 10 --concurrency 8 --key-count 16 --max-writes-per-key 250 --max-txn-length 4 --ports 63791,63792,63793 --host 127.0.0.1.
• Reproducibility unknown — only two scheduled stress runs exist in the visible history; 12:57 passed, 18:37 failed. Recommend manually re-dispatching once or twice to confirm whether this is reliably reproducible or sporadic.

Suggested investigation order

1. Audit the Redis adapter's RPC-failure → client-error → retry boundary for any commit-applied-but-not-acked window. The relevant entry points are adapter/redis.go and the Lua dispatch (adapter/redis_lua_pool.go); the Raft propose ACK path is in kv/sharded_coordinator.go / kv/shard_store.go.
2. Check the EXEC reuse / dedup path (PR Consider whether WAL can be substituted with Raft Log. #46 — runTransactionWithDedup, prepareDispatch, onePhaseTxnDedup). The dedup probe was meant to make exactly this class of retry idempotent for one-phase txns; verify the same protection covers the path that produced the duplicate here (:r 9 → :append 13 50 → :r 15).
3. Reproduce locally with ./scripts/run-jepsen-local.sh using the same stress parameters once it's clear which adapter code path is suspect. The stress params are at the top of jepsen-test-scheduled.yml — paste them into the local script.
4. If reproduced, add an Elle regression test under jepsen/ (or a Go-side targeted test under adapter/) that exercises the ambiguous-commit retry directly so the next regression is caught at PR time, not at scheduled-stress time.

Out of scope

• Not introduced by PR feat(snapshot-skip B3): cold-start restoreSnapshotState skip gate + CRC verifier + metrics #934 (impl/snapshot-skip-b3-skip-gate, cold-start FSM snapshot-restore skip metrics plumbing) — different branch, per-PR Jepsen passed, metrics-only change.
• Not introduced by PR backup: Phase 0b M4-2b impl - S3 collision-rename reversal #928 (S3 backup collision-rename encoder) — only touches internal/backup/.

Links

• Failing run: https://github.com/bootjp/elastickv/actions/runs/27033231956/job/79790587770
• Prior passing scheduled stress: https://github.com/bootjp/elastickv/actions/runs/27016167818
• PR backup: Phase 0b M4-2b impl - S3 collision-rename reversal #928 (in-window main commit, S3-only): backup: Phase 0b M4-2b impl - S3 collision-rename reversal #928

[bootjp]

Root cause — confirmed: all five anomalies cascade from ONE root (gate-off Redis retry double-apply)

Analyzed the failing run's elastickv-redis-append history.edn / results.edn.

The duplicates are server-side double-applies, not generator artifacts. The three duplicated values each appear exactly once per key in the invoke stream (163, 187, 87 — each appended once to its key; elle assigns unique per-key values). So a read showing the value twice means the server stored one logical append twice.

:future-read is the same root, not an independent fault. The future-read finding is:

op 1073, process 18:  [[:r 16 <list…87…>] [:append 16 87]]   :element 87

The txn reads key 16 and the list already contains 87, then appends 87 — i.e. the read saw the value this very txn is about to write. And 87 is exactly one of the duplicated values (:duplicates {87 2}). This is the signature of a committed-but-ambiguous retry: attempt 1 committed append 16 87, the ACK was lost (timeout / leader change), the client retried; the retry's pre-read observed its own first-attempt commit (→ future-read) and appended 87 again (→ duplicate-elements).

:G-single-item-realtime, :G2-item-realtime, and :cycle-search-timeout are the Elle checker's downstream report of the corrupted per-key version order the duplicate creates (the same cascade documented for the DynamoDB case in docs/design/2026_05_21_proposed_txn_secondary_idempotency.md) — not independent bugs.

This is the known gate-off legacy path; the fix already exists and is green

This run is the general scheduled workflow, which runs the Redis adapter gate-off (ELASTICKV_REDIS_ONEPHASE_DEDUP unset) precisely to keep covering the legacy path. The retry-double-apply under leadership churn is exactly the bug the option-2 one-phase dedup (PR #796/#884/#887, runTransactionWithDedup / onePhaseTxnDedup) was built to fix.

The dedup-mode workflow (jepsen-test-scheduled-dedup.yml, gate on) has run the Redis list-append workload green for 7 consecutive days (2026-06-01 → 06-07) with no :duplicate-elements / :G-single-item-realtime. That is the design doc's M4 default-on criterion for Redis — now met.

Resolution

Nothing in the dedup code path is wrong; the path that produced this duplicate ([:r k] … [:append k v] multi-mop EXEC) IS covered by runTransactionWithDedup, and the gate-on workflow proves it. The remaining step is the rollout completion: flip RedisServer.onePhaseTxnDedup to default-on (R5 ship-reader-before-writer — the probe has shipped everywhere since M2b, and the 7-day dedup-mode green criterion is satisfied). Once default-on, the general workflow no longer exercises the buggy legacy path.

(DynamoDB's equivalent default-on is not yet due — its dedup workload only joined the dedup-mode workflow on 06-04, so its 7-day window closes ~06-11.)

Recommend: track the Redis default-on as the closing action for this issue; the failure itself is the expected, now-fixed legacy-path behavior.