Skip to content
Merged
120 changes: 78 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,16 +40,16 @@ Stable structures are able to work directly in stable memory because each data s
its own memory.
When initializing a stable structure, a memory is provided that the data structure can use to store its data.

Here are some basic examples:
### Basic Usage

### Example: BTreeMap
Here's a basic example:

```rust
use ic_stable_structures::{BTreeMap, DefaultMemoryImpl};
let mut map: BTreeMap<u64, u64, _> = BTreeMap::init(DefaultMemoryImpl::default());
let mut map: BTreeMap<u64, String, _> = BTreeMap::init(DefaultMemoryImpl::default());

map.insert(1, 2);
assert_eq!(map.get(&1), Some(2));
map.insert(1, "hello".to_string());
assert_eq!(map.get(&1), Some("hello".to_string()));
```

Memories are abstracted with the [Memory] trait, and stable structures can work with any storage
Expand All @@ -58,54 +58,90 @@ This includes stable memory, a vector ([VectorMemory]), or even a flat file ([Fi

The example above initializes a [BTreeMap] with a [DefaultMemoryImpl], which maps to stable memory when used in a canister and to a [VectorMemory] otherwise.

### Example: BTreeSet
### Memory Isolation Requirement

The `BTreeSet` is a stable set implementation based on a B-Tree. It allows efficient insertion, deletion, and lookup of unique elements.
> **⚠️ CRITICAL:** Stable structures **MUST NOT** share memories!
> Each memory must belong to only one stable structure.

```rust
use ic_stable_structures::{BTreeSet, DefaultMemoryImpl};
let mut set: BTreeSet<u64, _> = BTreeSet::new(DefaultMemoryImpl::default());

set.insert(42);
assert!(set.contains(&42));
assert_eq!(set.pop_first(), Some(42));
assert!(set.is_empty());
```


Note that **stable structures cannot share memories.**
Each memory must belong to only one stable structure.
For example, this fails when run in a canister:

```no_run
```rust,ignore
use ic_stable_structures::{BTreeMap, DefaultMemoryImpl};
let mut map_1: BTreeMap<u64, u64, _> = BTreeMap::init(DefaultMemoryImpl::default());
let mut map_2: BTreeMap<u64, u64, _> = BTreeMap::init(DefaultMemoryImpl::default());
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(DefaultMemoryImpl::default());
let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(DefaultMemoryImpl::default());

map_1.insert(1, 2);
map_2.insert(1, 3);
assert_eq!(map_1.get(&1), Some(2)); // This assertion fails.
map_a.insert(1, b'A');
map_b.insert(1, b'B');
assert_eq!(map_a.get(&1), Some(b'A')); // ❌ FAILS: Returns b'B' due to shared memory!
assert_eq!(map_b.get(&1), Some(b'B')); // ✅ Succeeds, but corrupted map_a
```

It fails because both `map_1` and `map_2` are using the same stable memory under the hood, and so changes in `map_1` end up changing or corrupting `map_2`.
It fails because both `map_a` and `map_b` are using the same stable memory under the hood, and so changes in `map_b` end up changing or corrupting `map_a`.

To address this issue, we make use of the [MemoryManager](memory_manager::MemoryManager), which takes a single memory and creates up to 255 virtual memories for our disposal.
Here's the above failing example, but fixed by using the [MemoryManager](memory_manager::MemoryManager):
### Using MemoryManager

To address this issue, we use the [MemoryManager](memory_manager::MemoryManager), which takes a single memory and creates up to 255 virtual memories for our use.
Here's the above failing example, but fixed:

```rust
use ic_stable_structures::{
memory_manager::{MemoryId, MemoryManager},
BTreeMap, DefaultMemoryImpl,
};
let mem_mgr = MemoryManager::init(DefaultMemoryImpl::default());
let mut map_1: BTreeMap<u64, u64, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(0)));
let mut map_2: BTreeMap<u64, u64, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(1)));
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(0)));
let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(1)));

map_1.insert(1, 2);
map_2.insert(1, 3);
assert_eq!(map_1.get(&1), Some(2)); // Succeeds, as expected.
map_a.insert(1, b'A');
map_b.insert(1, b'B');
assert_eq!(map_a.get(&1), Some(b'A')); // ✅ Succeeds: Each map has its own memory
assert_eq!(map_b.get(&1), Some(b'B')); // ✅ Succeeds: No data corruption
```

### Memory Reclamation

During data migration scenarios, you often need to create a new data structure (B) and populate it with data from an existing structure (A). Without memory reclamation, this process doubles memory usage even after A is no longer needed.

Consider this migration scenario:

```rust
use ic_stable_structures::{
memory_manager::{MemoryId, MemoryManager},
BTreeMap, DefaultMemoryImpl,
};

let mem_mgr = MemoryManager::init(DefaultMemoryImpl::default());
let (mem_id_a, mem_id_b) = (MemoryId::new(0), MemoryId::new(1));

// ========================================
// Scenario 1: WITHOUT reclamation
// ========================================
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_a));
map_a.insert(1, b'A'); // Populate map A with data
let data = map_a.get(&1); // Extract data for migration
map_a.clear_new(); // A is now empty
drop(map_a); // Memory stays allocated to mem_id_a

let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_b));
map_b.insert(1, data.unwrap()); // B allocates NEW memory
// Result: 2x memory usage

// ========================================
// Scenario 2: WITH reclamation
// ========================================
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_a));
map_a.insert(1, b'A'); // Populate map A with data
let data = map_a.get(&1); // Extract data for migration
drop(map_a); // Drop A completely
mem_mgr.reclaim_memory(mem_id_a); // Free A's memory buckets for reuse

let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_b));
map_b.insert(1, data.unwrap()); // B reuses A's reclaimed memory buckets
// Result: 1x memory usage
```

**Important**: Always drop the original structure before calling `reclaim_memory`.

## Example Canister

Here's a fully working canister example that ties everything together.
Expand All @@ -116,7 +152,7 @@ Dependencies:
[dependencies]
ic-cdk = "0.18.3"
ic-cdk-macros = "0.18.3"
ic-stable-structures = "0.5.6"
ic-stable-structures = "0.7.0"
```

Code:
Expand All @@ -135,7 +171,7 @@ thread_local! {
RefCell::new(MemoryManager::init(DefaultMemoryImpl::default()));

// Initialize a `StableBTreeMap` with `MemoryId(0)`.
static MAP: RefCell<StableBTreeMap<u128, u128, Memory>> = RefCell::new(
static MAP: RefCell<StableBTreeMap<u64, String, Memory>> = RefCell::new(
StableBTreeMap::init(
MEMORY_MANAGER.with(|m| m.borrow().get(MemoryId::new(0))),
)
Expand All @@ -144,32 +180,32 @@ thread_local! {

// Retrieves the value associated with the given key if it exists.
#[ic_cdk_macros::query]
fn get(key: u128) -> Option<u128> {
fn get(key: u64) -> Option<String> {
MAP.with(|p| p.borrow().get(&key))
}

// Inserts an entry into the map and returns the previous value of the key if it exists.
#[ic_cdk_macros::update]
fn insert(key: u128, value: u128) -> Option<u128> {
fn insert(key: u64, value: String) -> Option<String> {
MAP.with(|p| p.borrow_mut().insert(key, value))
}
```

### More Examples

- [Basic Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/basic_example) (the one above)
- [Basic Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/basic_example): Simple usage patterns
- [Quickstart Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/quick_start): Ideal as a template when developing a new canister
- [Custom Types Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/custom_types_example): Showcases storing your own custom types

## Combined Persistence

If your project exclusively relies on stable structures, the memory can expand in size without the requirement of `pre_upgrade`/`post_upgrade` hooks.
If your project uses only stable structures, memory can expand in size without requiring `pre_upgrade`/`post_upgrade` hooks.

However, it's important to note that if you also intend to perform serialization/deserialization of the heap data, utilizing the memory manager becomes necessary. To effectively combine both approaches, refer to the [Quickstart Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/quick_start) for guidance.
However, if you also need to serialize/deserialize heap data, you must use the memory manager to avoid conflicts. To combine both approaches effectively, refer to the [Quickstart Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/quick_start) for guidance.

## Fuzzing

Stable structures requires strong guarantees to work reliably and scale over millions of operations. To that extent, we use fuzzing to emulate such operations on the available data structures.
Stable structures require strong guarantees to work reliably and scale over millions of operations. To that extent, we use fuzzing to emulate such operations on the available data structures.

To run a fuzzer locally,
```sh
Expand Down
62 changes: 55 additions & 7 deletions docs/src/concepts/memory-manager.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,64 @@ use ic_stable_structures::{
let mem_mgr = MemoryManager::init(DefaultMemoryImpl::default());

// Create two separate BTreeMaps, each with its own virtual memory
let mut map_1: BTreeMap<u64, u64, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(0)));
let mut map_2: BTreeMap<u64, u64, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(1)));
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(0)));
let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(MemoryId::new(1)));

// Demonstrate independent operation of the two maps
map_1.insert(1, 2);
map_2.insert(1, 3);
assert_eq!(map_1.get(&1), Some(2)); // Succeeds as expected
map_a.insert(1, b'A');
map_b.insert(1, b'B');
assert_eq!(map_a.get(&1), Some(b'A')); // ✅ Succeeds: Each map has its own memory
assert_eq!(map_b.get(&1), Some(b'B')); // ✅ Succeeds: No data corruption
```

```admonish warning ""
Virtual memories from the `MemoryManager` cannot be shared between stable structures.
Each memory instance should be assigned to exactly one stable structure.
**⚠️ CRITICAL:** Stable structures **MUST NOT** share memories!
Each memory instance must be assigned to exactly one stable structure.
```

## Memory Reclamation

During data migration scenarios, you often need to create a new data structure and populate it with data from an existing structure. Without memory reclamation, this process doubles memory usage even after the original structure is no longer needed.

The `MemoryManager` provides a `reclaim_memory` method to efficiently handle these scenarios:

```rust
use ic_stable_structures::{
memory_manager::{MemoryId, MemoryManager},
BTreeMap, DefaultMemoryImpl,
};

let mem_mgr = MemoryManager::init(DefaultMemoryImpl::default());
let (mem_id_a, mem_id_b) = (MemoryId::new(0), MemoryId::new(1));

// ========================================
// Scenario 1: WITHOUT reclamation
// ========================================
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_a));
map_a.insert(1, b'A'); // Populate map A with data
let data = map_a.get(&1); // Extract data for migration
map_a.clear_new(); // A is now empty
drop(map_a); // Memory stays allocated to mem_id_a

let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_b));
map_b.insert(1, data.unwrap()); // B allocates NEW memory
// Result: 2x memory usage

// ========================================
// Scenario 2: WITH reclamation
// ========================================
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_a));
map_a.insert(1, b'A'); // Populate map A with data
let data = map_a.get(&1); // Extract data for migration
drop(map_a); // Drop A completely
mem_mgr.reclaim_memory(mem_id_a); // Free A's memory buckets for reuse

let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(mem_mgr.get(mem_id_b));
map_b.insert(1, data.unwrap()); // B reuses A's reclaimed memory buckets
// Result: 1x memory usage
```

```admonish info ""
**Important**: Always drop the original structure before calling `reclaim_memory`.
The method returns the number of pages that were reclaimed and made available for reuse.
```
10 changes: 5 additions & 5 deletions src/btreemap.rs
Original file line number Diff line number Diff line change
Expand Up @@ -710,14 +710,14 @@ where
// TODO: In next major release (v1.0), rename this method to `clear` to follow
// standard Rust collection naming conventions.
///
/// # Safety Note for Bucket Release
/// If using manual bucket release via `MemoryManager::release_virtual_memory_buckets()`:
/// # Safety Note for Memory Reclamation
/// If using manual memory reclamation via `MemoryManager::reclaim_memory()`:
/// 1. **MANDATORY**: Drop this BTreeMap object first (let it go out of scope)
/// 2. Call `release_virtual_memory_buckets()` on the memory manager
/// 2. Call `reclaim_memory()` on the memory manager
/// 3. Create new structures as needed
///
/// Using this BTreeMap after bucket release causes data corruption.
/// Note: You can still call `clear_new()` if you need to clear data without bucket release.
/// Using this BTreeMap after memory reclamation causes data corruption.
/// Note: You can still call `clear_new()` if you need to clear data without memory reclamation.
pub fn clear_new(&mut self) {
self.root_addr = NULL;
self.length = 0;
Expand Down
10 changes: 5 additions & 5 deletions src/btreeset.rs
Original file line number Diff line number Diff line change
Expand Up @@ -353,14 +353,14 @@ where
/// assert!(set.is_empty());
/// ```
///
/// # Safety Note for Bucket Release
/// If using manual bucket release via `MemoryManager::release_virtual_memory_buckets()`:
/// # Safety Note for Memory Reclamation
/// If using manual memory reclamation via `MemoryManager::reclaim_memory()`:
/// 1. **MANDATORY**: Drop this BTreeSet object first (let it go out of scope)
/// 2. Call `release_virtual_memory_buckets()` on the memory manager
/// 2. Call `reclaim_memory()` on the memory manager
/// 3. Create new structures as needed
///
/// Using this BTreeSet after bucket release causes data corruption.
/// Note: You can still call `clear()` if you need to clear data without bucket release.
/// Using this BTreeSet after memory reclamation causes data corruption.
/// Note: You can still call `clear()` if you need to clear data without memory reclamation.
pub fn clear(&mut self) {
self.map.clear_new();
}
Expand Down
Loading