bd-resilient-kv: add versioned store #307

snowp · 2025-11-05T01:48:56Z

This adds a versioned key-value store that supports points in time recovery with disk persistence. This will be used to power a new primitive within the SDK known as "state" that will power feature flags, "global" fields as well as other state tracked by the system.

Refer to VERSIONED_FORMAT.md for details about the implementation.

Fixes BIT-6799

Adds a new variant of the resilient-kv that supports automatically rotating versioned entries

Add SnapshotCleanup utility to manage disk space by removing old archived journal snapshots based on version thresholds or retention counts. Supports two cleanup strategies: version-based (keep snapshots >= min version) and count-based (keep N most recent). Includes comprehensive test coverage with 15 tests covering various scenarios including empty directories, edge cases, and multi-journal isolation.

Add test that documents and verifies the impossibility of buffer overflow during rotation. The test demonstrates that rotation always succeeds because compacted state fits in the same-sized buffer used during normal operation.

…unique keys" This reverts commit 96394a2.

Add explicit documentation about failure modes applications must handle versus those that are architecturally impossible. Clarifies that rotation cannot fail due to buffer overflow since compacted state always fits in same-sized buffer.

mattklein123 · 2025-11-05T19:31:12Z

Cargo.toml

 arc-swap          = "1.7.1"
 assert_matches    = "1.5.0"
 assert_no_alloc   = "1.1.2"
+async-compression = { version = "0.4.20", features = ["tokio", "zlib"] }


We should make sure this does the right thing with flate2 in terms of using system zlib otherwise maybe better to do it manually.

mattklein123 · 2025-11-05T19:32:16Z

bd-resilient-kv/AGENTS.md

+- Automatic rotation when journal size exceeds high water mark (triggered during async write operations)
+- Current state is compacted into a new journal as versioned entries
+- Old journal is archived with `.v{version}.zz` suffix
+- Archived journals are automatically compressed using zlib (RFC 1950, level 3) asynchronously


FYI we use level 5 in the rest of the mobile code.

Make sense to keep it consistent, I'll update

mattklein123 · 2025-11-05T19:43:40Z

bd-resilient-kv/src/kv_journal/versioned.rs

+    // Write all current state as versioned entries at the snapshot version
+    // Use the original timestamp from each entry to preserve historical accuracy


I don't think this is correct. I think we need to the entries to have the original versions, and for each snapshot to have increasing versions. Reset would only happen on full clear I think. This allows us to replay the the full sequence against the ring buffer and get accurate state for every log. As long as the version field is u64 I don't think rollover can happen in practice.

Or maybe I'm misunderstanding how this would be used? Basically my thinking was we would just keep making files and then clean them up as it rolls over. Compression of the previous file is orthogonal.

Per offline convo it would be nice to have the encoder use varints for int fields. I don't think that should be all that difficult.

mattklein123 · 2025-11-05T19:49:32Z

bd-resilient-kv/src/versioned_kv_store.rs

+/// # Rotation Strategy
+/// When the journal reaches its high water mark, the store automatically:
+/// 1. Creates a new journal file with a rotated name (e.g., `store.jrn.v12345`)
+/// 2. Writes the current state as versioned entries at the rotation version


This is related to my other comment, but I'm not sure this is the way it should work. I think in the new world we basically never compact and just delete old journals, with optional compression. If we pass through the hashmap cache, it stays consistent. Then on startup the way you reconstruct state is you read all the files and process them all sequentially. I'm not sure the benefit of compacting the state into the new journal.

mattklein123 · 2025-11-05T19:50:56Z

bd-resilient-kv/src/versioned_kv_journal/recovery.rs

+  max_version: u64,
+}
+
+impl VersionedRecovery {


Seems like this code was generated to do basically what I said about restart of reading all files, so wondering what the compaction is needed for.

mattklein123

LGTM at a high level. We should add a fuzzer per discussion but we would do this in a different PR if you want.

mattklein123 · 2025-11-09T19:30:00Z

bd-resilient-kv/VERSIONED_FORMAT.md

+│                    VERSIONED JOURNAL ENTRY                              │
+│  (Protobuf-encoded StateKeyValuePair)                                   │
+├─────────────────────────────────────────────────────────────────────────┤
+│  Frame Length (u32)         │  4 bytes                                  │


I should have done this with the ring buffer but since you are already in here I would do varint encoding for this as well since it will generally be pretty small.

Updated this to varint

mattklein123 · 2025-11-09T19:32:32Z

bd-resilient-kv/src/tests/mod.rs

+pub fn make_string_value(s: &str) -> state::payload::StateValue {
+  state::payload::StateValue {
+    value_type: Some(state::payload::state_value::Value_type::StringValue(
+      s.to_string(),
+    )),
+    ..Default::default()
+  }
+}


We probably don't want to go down the path of custom encoder for all the things but this could be trivially zero copy also. Maybe next week we could try hacking on the proc macro. (Definitely a TODO)

mattklein123 · 2025-11-09T19:33:14Z

bd-resilient-kv/src/versioned_kv_journal/file_manager.rs

+  let pattern = format!("{name}.jrn.");
+
+  let mut max_gen = 0u64;
+  let Ok(entries) = std::fs::read_dir(dir) else {


mattklein123 · 2025-11-09T19:35:57Z

bd-resilient-kv/src/versioned_kv_journal/framing.rs

+#[cfg(test)]
+#[path = "./framing_test.rs"]
+mod tests;


nit: move to top of file / update agents.md about this.

mattklein123 · 2025-11-09T19:43:05Z

bd-resilient-kv/src/versioned_kv_journal/store.rs

+  /// Insert a value for a key, returning the timestamp assigned to this write.
+  ///
+  /// Note: Inserting `Value::Null` is equivalent to removing the key.
+  ///
+  /// # Errors
+  /// Returns an error if the value cannot be written to the journal.
+  pub async fn insert(&mut self, key: String, value: StateValue) -> anyhow::Result<u64> {


In practice I wonder if we are going to need to pass in the timstamp, so for example it can line up with logs. We can always change this later.

Yeah we can see how it goes when we integrate it with the rest of the system

Directly create the mmap instead of initializing temp buffer

snowp added 12 commits November 4, 2025 17:45

bd-resilient-kv: add versioned store

55fb6b7

Adds a new variant of the resilient-kv that supports automatically rotating versioned entries

add versioned recovery, zlib compression

773c679

make async

1c64982

better compression, allows, +

edee87c

revert

5e83fa7

add u64 helper

a2a7dcc

only read from latest journal

a22768f

docs(bd-resilient-kv): update documentation to reflect async API changes

ebc6c06

Revert "test(bd-resilient-kv): add test verifying rotation with many …

6eae295

…unique keys" This reverts commit 96394a2.

mattklein123 reviewed Nov 5, 2025

View reviewed changes

snowp added 10 commits November 5, 2025 16:33

use timestamp as version

f14c523

use flate2 to avoid pulling in minz_oxide

70a112b

update some docs

aa19899

some more docs

f0910bd

simplify tests

bcef819

use 5

fb1b250

remove base_timestamp, not needed

4fd4947

remove cleanup for now

127f8ad

remove rotation callback

f00945c

update docs

8045327

snowp assigned Copilot Nov 6, 2025

snowp added 6 commits November 6, 2025 11:05

revert cargo

fb10cb8

update docs

163c53c

simplify docs

2bbc81e

simplify recovery

790a533

only consider snapshots in recovery

cf732ee

remove tests we don't need

747ea7f

snowp added 12 commits November 7, 2025 13:20

docs fixes, naming

c96c20e

use protobuf for varint encoding

ac9d000

simplify

fbbd0c8

clean up framing

651fe5f

renames, move stuff around

e413633

fixes

4ee6510

remove

af5c3a1

revert

631ddb7

revert

45fdec6

comments

7feb85f

add more tests

93e5f46

use time provider

5498458

mattklein123 reviewed Nov 9, 2025

View reviewed changes

snowp added 6 commits November 9, 2025 19:21

simplify rotation code, use tokio fs where possible

35abc73

mode mod

1047e02

clippy

afb6d3e

update api

8b266bf

use varint for length

89f9a56

add license headers

2aca5d0

snowp marked this pull request as ready for review November 15, 2025 01:36

snowp added 6 commits November 14, 2025 20:10

propagate partial failure case, remove read() in favor of ctor arg

2be28cd

clippy

1d26a05

impprove rotation

3004014

Directly create the mmap instead of initializing temp buffer

clippy

23bd853

simplify

518fb23

refactor

6f19078

mattklein123 approved these changes Nov 17, 2025

View reviewed changes

snowp merged commit b6620e3 into main Nov 17, 2025
6 checks passed

snowp deleted the versioned-kv branch November 17, 2025 18:53

github-actions bot locked and limited conversation to collaborators Nov 17, 2025

		// Write all current state as versioned entries at the snapshot version
		// Use the original timestamp from each entry to preserve historical accuracy

bd-resilient-kv: add versioned store #307

bd-resilient-kv: add versioned store #307

Uh oh!

Conversation

snowp commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattklein123 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

snowp commented Nov 5, 2025 •

edited

Loading