MemTier

MemTier is a user-level GPU-aware data supply runtime.

Supported paths

• POSIX buffered path
• DRAM cache path (userspace cache)
• Optional GDS path (real backend if available, otherwise stub/fallback)
• CUDA GPU target if available

Build

CPU-only

mkdir -p build && cd build
cmake ..
make -j
ctest

One-click rebuild

./scripts/rebuild.sh
# optional custom build dir and extra cmake args
./scripts/rebuild.sh ./build -DMEMTIER_ENABLE_CUDA=ON

CUDA

cmake -DMEMTIER_ENABLE_CUDA=ON ..

GDS

cmake -DMEMTIER_ENABLE_GDS=ON ..

Run examples

./build/basic_read <file>
./build/cache_reuse
./build/path_policy_demo
./build/readv_small_io
./build/async_pipeline
./build/prefetch_demo

Benchmarks

./build/bench_large_read --file=test.bin --size=1073741824 --mode=memtier
./build/bench_reuse test.bin
./build/bench_small_readv test.bin

API quick usage

memtier_ctx_t* ctx = NULL;
memtier_init(NULL, &ctx);
memtier_read(ctx, path, offset, size, dst, MEMTIER_TARGET_CPU);
memtier_prefetch(ctx, path, offset, size, NULL);
memtier_stats_t st;
memtier_stats(ctx, &st);
memtier_finalize(ctx);

Limitations

• Does not modify Linux Page Cache or kernel behavior.
• DRAM cache is userspace cache only.
• UVM/HBM cache is not fully implemented.
• GDS is optional; missing GDS uses stub/fallback.
• Prefetch TO_HBM and async prefetch are currently unsupported.

Troubleshooting

• No CUDA: GPU target returns MEMTIER_ERR_UNSUPPORTED.
• No GDS: build still works; runtime uses stub/fallback.
• O_DIRECT alignment error: ensure offset/size meet alignment requirements.
• Permission error: check file permissions and mount options.

Prepare experiment files on NVMe

Use the helper module/executable to create experiment files under /mnt/gds2/cwd_test:

./build/prepare_experiment_files memtier_exp_1g.bin 1073741824

This creates /mnt/gds2/cwd_test/memtier_exp_1g.bin with deterministic content.

Object-level API (v1)

• memtier_load_tensor: for model weights/safetensors/checkpoint tensor payload reads.
• memtier_load_chunks: for RAG chunks/KV blocks/index blocks.
• MemTierDataset: simple DataLoader-style indexed sample reader.
• These are lightweight wrappers that only translate object semantics to file-range reads; final path selection is still decided by MemTier runtime.

Python wrapper status

• Current Python API is a lightweight wrapper (python/memtier) intended for quick integration.
• CPU-first behavior works without CUDA/GDS/PyTorch.
• If PyTorch is unavailable, APIs return bytes/NumPy-compatible data.

PyTorch-compatible Object API

• memtier.read_into_tensor(path, offset, tensor)
• memtier.load_tensor(path, offset, shape, dtype, device)
• memtier.MemTierDataset(index_file, data_file, target)

Compatibility approach:

• No PyTorch source changes.
• Integration uses Python wrapper + C API + tensor.data_ptr().
• If PyTorch is absent, CPU bytes/NumPy fallback still works.

Example use:

• Use load_tensor for model-weight payload slices.
• Use MemTierDataset directly or with torch.utils.data.DataLoader.

Current limits:

• Not a full replacement of torch.load.
• Does not fully parse safetensors metadata.
• CUDA/GDS are optional runtime paths.
• Non-contiguous tensors are not supported in read_into_tensor.
• Full async stream semantics are deferred.

Python scripts:

python tests/test_python_tensor.py
python tests/test_python_dataset.py