v0.2.0 — Multi-chunk elastic growth

What's new

SharedStore now grows automatically. Instead of allocating a single fixed-size shared memory block, the store splits into a control block (header + index) and separate data chunks that are created on demand when the current chunk fills up.

API changes

# Before
store = SharedStore.create("pipeline", size_mb=256, max_entries=1024)
store = SharedStore.connect("pipeline", locks, max_entries=1024)

# After
store = SharedStore.create("pipeline", chunk_size_mb=64, max_entries=1024)
store = SharedStore.connect("pipeline", locks)  # reads config from header

Details

• Elastic growth: put() tries all existing chunks. If none have space, it creates a new shared memory segment ({name}_{i}) and retries.
• Lazy chunk discovery: Connected processes open chunks on demand when get() encounters a virtual offset pointing to a chunk not yet opened locally.
• Virtual offsets: Index entries store chunk_index * chunk_data_size + local_offset. Encoding/decoding is transparent to the index layer.
• Per-chunk allocators: Each chunk has its own ChunkHeader (free_list_head, data_size) and BlockAllocator. No free-list links span chunks.
• Format version 2: New StoreHeader layout (chunk_data_size, chunk_count fields) and ChunkHeader (32 bytes).

Performance

No regression on the core hot path (single-chunk put/get). Auto-growth adds ~1ms per new chunk creation (OS syscall overhead). Zero-copy get() latency is unaffected regardless of which chunk the data lives in (~4 us).

Tests

64 tests passing, including:

• Auto-growth (fill one chunk, verify second chunk created)
• Multi-chunk get/delete across chunks
• connect() reading config from header
• Concurrent chunk growth from multiple processes

v0.1.1

Fix Python version requirement to >=3.12 (was incorrectly cached as >=3.13 in v0.1.0 wheel)

v0.1.0

shmem v0.1.0

Zero-copy shared memory key-value store for sharing numpy arrays and raw bytes between Python processes.

Highlights

• put() copies data into shared memory (one memcpy)
• get() returns a read-only zero-copy np.ndarray view — ~3.5μs regardless of data size
• get_mut() returns a writable zero-copy view for in-place mutation
• put_bytes() / get_bytes() for raw bytes with zero-copy memoryview
• Readers-writers locking for safe multi-process access
• Free-list allocator with forward/backward coalescing

Performance (1080p, 6.2 MB)

Operation	Latency
put()	~300 μs
get() (zero-copy)	~3.5 μs
Cross-process throughput	~10 GB/s

Install

uv add shmem@https://github.com/zeeshanhaque21/shmem/releases/download/v0.1.0/shmem-0.1.0-py3-none-any.whl

Requires Python ≥ 3.12 and NumPy ≥ 2.0.