Handling memory with a memory pool

Using a single-source memory manager provides significant advantages for both static and dynamic memory allocation. In Loom, a memory pool safely allows efficient dynamic allocation of memory chunks for handling sensor data and data transmissions.

Traditionally, these data types are managed using a large statically allocated buffer, combined with bounds checking to prevent overflow into adjacent memory. While effective, this approach requires careful sizing upfront and can either waste memory or risk overflow if underestimated.

With a memory pool (mempool), this constraint is largely eliminated. Memory can be allocated dynamically as needed, reducing the need to predict buffer sizes in advance and minimizing the risk of overflow-related issues. This leads to more flexible and safer memory usage, especially in systems with variable data loads.

The mempool manager, MemPool.hpp, is responsible for all memory pool operations. It is designed to be both easy to use and highly transparent for debugging, providing insight into allocation behavior and memory usage.

Although static allocation outside the mempool is still possible, it is discouraged for large buffers (greater than 500 bytes). The mempool reserves a substantial portion of SRAM at compile time (~10KB [8KB arena + 2KB Metadata]), so excessive static allocation can quickly exhaust available memory.

For more detailed information: (https://en.wikipedia.org/wiki/Memory_pool)

Demonstration of RAII/RAID Leases

The following video demonstrates a LoomJsonDocument and automated resource destruction for a lease that has gone out-of-scope. Skip to ~30s for outputs.

Read more about RAII here, https://en.cppreference.com/cpp/language/raii

Reliability Tests

The above test has been ran for 72 hours with mock sample data and a broken RTC. We use an ArduinoJsonDoc allocated with the mempool to perform the R/W ops.

API Examples

I chose to not import any additional Arduino or embedded dependencies to keep MemPool.hpp compact and easily transferable. This means developers have the freedom to choose how they implement the API based on their own project requirements. All examples shown below demonstrate ways the API has been integrated into Loom and serve as reference use-cases.

Configuration

The pool is configured at compile time before including MemPool.hpp.

Macro	Default	Meaning
MEMPOOL_BLOCK_SIZE	64	Bytes per allocation block
MEMPOOL_BLOCK_COUNT	128	Total blocks in the arena
MEMPOOL_MAX_LEASES	64	Maximum active lease records
MEMPOOL_ZERO_ON_ALLOC	1	Clear memory when allocated
MEMPOOL_ZERO_ON_FREE	1	Clear memory when released
MEMPOOL_TAG_MAX_LEN	16	Debug tag storage per lease
MEMPOOL_DUMP_LINE_BYTES	128	Temporary line buffer for dumps
SYSTEM_RAM_TOTAL_BYTES	32768	Total RAM value reported in stats

With the defaults, usable arena capacity is 64 * 128 = 8192 bytes. Each allocation is rounded up to whole blocks.

Core Types

MemPool

Owns the fixed arena, lease metadata, counters, and diagnostics.

MemPool pool;

MemPool::Handle

A lightweight identifier for an allocation.

MemPool::Handle h = pool.alloc(100, "sample");

A handle contains a lease slot and generation number. Generation checking helps detect stale handles after release/reuse.

Use MemPool::Handle::invalid() to produce an invalid handle.

MemPool::Lease

RAII wrapper around a handle. It releases the allocation automatically when the lease goes out of scope.

{
    MemPool::Lease lease = pool.allocLease(100, "sample");
    if (lease) {
        snprintf(lease.chars(), lease.capacity(), "hello");
    }
} // memory is released here

Lease is movable but not copyable, so ownership cannot accidentally be shared.

MemPool::Stats

Snapshot of pool usage, counters, failure counts, and optional system RAM data.

MemPool::Stats s = pool.stats();

Lifecycle

bool init()

Resets the arena, lease table, and runtime counters.

pool.init();

init() preserves the registered free-RAM provider callback.

void setFreeRamProvider(FreeRamProviderFn provider)

Registers a function used by stats() and dumpStats() to report free system RAM.

pool.setFreeRamProvider(freeMemory);

The callback signature is:

typedef int (*FreeRamProviderFn)(void);

If no provider is registered, reported system free RAM is 0.

Allocation

Handle alloc(size_t bytes)

Allocates at least bytes bytes and returns a handle.

MemPool::Handle h = pool.alloc(128);
if (pool.valid(h)) {
    // use h
}

Requesting 0 bytes is normalized to 1 byte. Failed allocations return Handle::invalid().

Handle alloc(size_t bytes, const char *tag)

Allocates bytes and records a short debug tag.

MemPool::Handle h = pool.alloc(256, "packet");

Tags are useful in dumpActiveLeases().

Lease allocLease(size_t bytes, const char *tag = nullptr)

Allocates and returns an RAII lease.

MemPool::Lease packet = pool.allocLease(256, "packet");
if (!packet) {
    return;
}

This is the preferred API for scoped temporary buffers.

Lease allocBlock(const char *tag = nullptr)

Allocates one block.

MemPool::Lease block = pool.allocBlock("scratch");

Release

bool release(Handle h)

Releases a valid handle.

bool ok = pool.release(h);

Returns false if the handle is invalid, stale, already released, or if internal ownership metadata is inconsistent.

bool Lease::release()

Manually releases a lease before it goes out of scope.

lease.release();

After release, the lease becomes invalid.

bool releasePointer(void *ptr)

Releases the active allocation that starts at ptr.

pool.releasePointer(buffer);

This is mainly for allocator-style integrations. The pointer must be the exact start of a mempool allocation.

Size And Capacity

size_t size(Handle h) const

Returns the logical requested size.

size_t requested = pool.size(h);

size_t capacity(Handle h) const

Returns the physical capacity reserved in whole blocks.

size_t usable = pool.capacity(h);

Example: with 64-byte blocks, a 100-byte request has 100 bytes of logical size and 128 bytes of physical capacity.

bool resize(Handle h, size_t newSize)

Changes the logical size if newSize <= capacity(h).

pool.resize(h, 120);

This does not move memory or allocate new blocks.

Data Access

bool write(Handle h, size_t offset, const void *src, size_t len)

Bounds-checked copy into the lease's logical size.

const char msg[] = "hello";
pool.write(h, 0, msg, sizeof(msg));

bool read(Handle h, size_t offset, void *dst, size_t len)

Bounds-checked copy out of the lease's logical size.

char msg[16];
pool.read(h, 0, msg, sizeof(msg));

bool clear(Handle h, uint8_t value = 0)

Fills the full physical capacity of the lease with value.

pool.clear(h);
pool.clear(h, 0xFF);

uint8_t *bytes(Handle h) / const uint8_t *bytes(Handle h) const

Returns a raw byte pointer to the lease start, or nullptr if invalid.

uint8_t *buf = pool.bytes(h);

char *chars(Handle h) / const char *chars(Handle h) const

Returns the same pointer cast as a character buffer.

char *text = pool.chars(h);

Raw pointer access is deliberately low-level. The caller must stay within the lease's lifetime and capacity.

Validation And Lookup

bool valid(Handle h) const

Returns true only when the slot is active and the generation matches.

if (!pool.valid(h)) {
    return;
}

Handle handleFromPointer(const void *ptr) const

Finds the active handle whose allocation starts at ptr.

MemPool::Handle h = pool.handleFromPointer(ptr);

Returns an invalid handle if the pointer is null, outside the arena, not block aligned, not at an allocation start, or no longer active.

const char *tag(Handle h) const

Returns the debug tag for a valid handle, or nullptr for an invalid handle.

Serial.println(pool.tag(h));

Diagnostics

Stats stats() const

Returns a snapshot of current usage and counters.

Important fields include:

Field	Meaning
freeBlocks, usedBlocks, totalBlocks	Block usage
bytesFree, bytesUsed, bytesTotal	Byte usage
activeLeases	Current active allocations
highWaterBlocks	Peak used block count
failedAllocs	Total allocation failures
allocCalls, releaseCalls, readCalls, writeCalls, clearCalls	Operation counters
bytesRequested, bytesGranted, bytesReleased	Allocation/release byte counters
failedTooLarge, failedNoContiguousRun, failedNoLeaseSlot	Allocation failure causes
failedReleaseInvalid, failedReleaseCorrupt	Release failure causes
maxContiguousFreeBlocks	Largest currently free contiguous block run
lastAllocFailureReason	Last allocation failure enum value

size_t maxAllocBytes() const

Returns the largest currently allocatable contiguous byte span.

size_t maxNow = pool.maxAllocBytes();

size_t dumpActiveLeases(LeaseDumpCallback cb, void *userCtx) const

Iterates active leases and optionally emits one formatted line per lease.

void printLine(const char *line, void *ctx) {
    (void)ctx;
    Serial.println(line);
}

size_t count = pool.dumpActiveLeases(printLine, nullptr);

The callback signature is:

typedef void (*LeaseDumpCallback)(const char *line, void *userCtx);

Each line includes slot, generation, starting block, block count, logical size, and tag.

void dumpStats(LeaseDumpCallback cb, void *userCtx) const

Emits a self-contained stats report through a callback.

pool.dumpStats(printLine, nullptr);

If cb is nullptr, nothing is emitted.

Allocation Failure Reasons

lastAllocFailureReason uses MemPool::AllocFailureReason.

Value	Meaning
ALLOC_FAIL_NONE	No failure
ALLOC_FAIL_TOO_LARGE	Request needs more blocks than the pool contains
ALLOC_FAIL_NO_CONTIGUOUS_RUN	Enough total space may exist, but not as one contiguous run
ALLOC_FAIL_NO_LEASE_SLOT	No lease metadata slot is available

ArduinoJson Integration

Include MemPoolJson.hpp to use the mempool-backed ArduinoJson allocator.

#include <MemPool.hpp>
#include <MemPoolJson.hpp>

MemPool pool;

void example() {
    LoomJsonDocument doc(512, MemPoolJsonAllocator(&pool));
    doc["type"] = "mempool-json";
}

MemPoolJsonAllocator

Allocator used by BasicJsonDocument.

MemPoolJsonAllocator allocator(&pool);

Methods:

Method	Behavior
allocate(size_t size)	Allocates a mempool lease tagged json_doc and returns its bytes
deallocate(void *ptr)	Releases the allocation starting at ptr
reallocate(void *ptr, size_t new_size)	Resizes in place when capacity allows; otherwise allocates, copies, and releases old storage

LoomJsonDocument

Alias for an ArduinoJson document using MemPoolJsonAllocator.

using LoomJsonDocument = BasicJsonDocument<MemPoolJsonAllocator>;

When the document goes out of scope, ArduinoJson calls the allocator's deallocate(), returning the backing storage to the pool.

Recommended Usage Pattern

Prefer scoped leases for temporary buffers:

MemPool::Lease lease = pool.allocLease(256, "message");
if (!lease) {
    return;
}

snprintf(lease.chars(), lease.capacity(), "value=%d", 42);

Use raw handles when ownership must cross a function boundary:

MemPool::Handle makeBuffer(MemPool &pool) {
    return pool.alloc(128, "owned");
}

For JSON documents, construct LoomJsonDocument with a MemPoolJsonAllocator bound to the active pool:

LoomJsonDocument doc(1024, MemPoolJsonAllocator(&pool));

Important Constraints

• The allocator requires contiguous free blocks.
• write() and read() are checked against logical size(), not physical capacity().
• clear() fills physical capacity, not just logical size.
• bytes() and chars() return borrowed raw pointers; they become invalid after release.
• releasePointer() only works for pointers that exactly match the start of a live allocation.
• Lease ownership is automatic; do not store raw pointers beyond the lease's scope.