Conversation
Comprehensive design for the Delta + Claim pattern to resolve write conflicts on Redis collection metadata (List, Hash, Set, ZSet). Includes key layouts, write/read paths, background compaction, POP claim mechanism, migration strategy, and implementation status.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request outlines a design for implementing a Delta pattern across Redis collection types to eliminate write conflicts on metadata keys. It introduces wide-column decomposition for Hash and Set types, a CAS-based Claim mechanism for List POP operations, and a background compaction process. The review feedback highlights an inconsistency in the List metadata key layout description, identifies a broken internal section reference, and suggests consolidating redundant terminology for delta limits to improve clarity for implementers.
- Fix base meta key format: !lst|meta|<userKey> (no length prefix), matching existing codebase (store/list_helpers.go) - Fix broken cross-reference to nonexistent Section 11.1 - Remove maxDeltaHardLimit alias, use maxDeltaScanLimit consistently
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request proposes a design for 'Collection Metadata Delta' to resolve write conflicts in Redis collection types (List, Hash, Set, and ZSet) by moving from a Read-Modify-Write pattern to a Delta-based approach. The design includes wide-column decomposition for Hash and Set, a background compaction process, and a CAS-based Claim mechanism for List POP operations. Feedback was provided regarding an inaccurate future date in the implementation status section.
|
|
||
| ## Implementation Status | ||
|
|
||
| > Last updated: 2026-04-13 |
There was a problem hiding this comment.
The date '2026-04-13' is in the future. Please ensure this date is accurate or remove it to avoid confusion.
… tests) Data Loss: - Add DEL command batched-loop cleanup design for Delta/Claim keys - Document that DEL must exhaust all delta/claim keys, not just maxDeltaScanLimit Consistency: - Document dual-path txnStartTS problem and required Delta prefix scan fix - Add HLC overflow behavior (error on logical counter exhaustion) - Add Raft single-leader guarantee as explicit dependency for commitTS uniqueness Performance: - Add expected latency targets for resolveListMeta (< 100us at 64 deltas) - Document LRANGE compound scan cost (meta + items + claims) Migration: - Add batched migration design for large Hash/Set collections - Document partial migration marker and merged read path Tests: - Add Jepsen mixed standalone/MULTI workload requirement to Phase L5 - Add Hash/Set migration edge case tests (crash, OCC, batch, partial read)
Data Loss (6 fixes): - DEL: tombstone marker + batched cleanup for large collections - Claim GC: must respect MVCC retention window for item/claim deletion - RPOP: re-resolve metadata guard against compacted sequences - Partial migration: wide-column keys take precedence (dedup rule) - Lazy reaping: TOCTOU prevention via atomic Delta prefix scan - txnStartTS: added implementation status tracking line Concurrency/Fault (6 fixes): - HLC overflow: updated to match actual implementation (wall advance) - Compactor cursor: reset on leader transition (term change detection) - ABA analysis: explicit Claim OCC analysis under compaction recycling - Compaction vs writer: explicit safety analysis added - Sync compaction: per-key mutex/singleflight for hard limit - Dual-path interaction matrix table Consistency (4 fixes): - Negative Len: defensive clamp in all resolve*Meta functions - readTS invariant: explicit same-timestamp constraint - Stale Claim OCC: PUSH must register Claim key in readSet - MULTI/EXEC isolation: corrected from 'matches Redis' to 'weaker than Redis serializable' Performance (4 fixes): - txnStartTS reverse scan: cost analysis and caching suggestion - maxKeysPerTick: default 512 specified - Migration batch: byte-bounded (512 KiB) not just count-bounded - Eager migration: background worker option to limit merge window Tests (10 additions): - Jepsen workloads for Hash, Set, ZSet (not just List) - Crash recovery tests for List delta/claim - DEL large-delta test (>256 deltas) - maxDeltaScanLimit overflow integration test - txnStartTS Delta-awareness test - Concurrent compaction test - Empty collection lifecycle test - SRANDMEMBER wide-column test - Benchmark pass/fail criteria (100us/500us targets)
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive design for 'Collection Metadata Deltas' to eliminate write conflicts on Redis collection metadata (List, Hash, Set, ZSet) under high concurrency. The design transitions from a Read-Modify-Write pattern on a single metadata key to a conflict-free Delta pattern, where writers append unique delta entries. Key components include wide-column decomposition for Hash and Set types, a CAS-based 'Claim' mechanism for List POP operations to ensure mutual exclusion without metadata locks, and a background compaction worker to collapse deltas and perform garbage collection of claimed items. The design also addresses critical consistency requirements for MULTI/EXEC transactions and provides a detailed multi-phase implementation and migration plan. I have no feedback to provide as the design is thorough and addresses the identified performance bottlenecks effectively.
1 participant
Summary
Test plan