[MMAP] Add in-memory (mmap) stream index mapping. by rfsaliev · Pull Request #310 · intel/ScalableVectorSearch

rfsaliev · 2026-04-03T13:25:29Z

This pull request adds to CPP Runtime API support for loading vector search indices (FlatIndex and VamanaIndex) from memory-mapped files and memory buffers, in addition to existing stream-based loading. It introduces new API methods, updates internal implementations to manage mapped resources, and adds comprehensive tests to ensure correct behavior for all supported storage types.

Major new features:

Added map_to_file and map_to_memory static methods to both FlatIndex and VamanaIndex classes, allowing indices to be loaded directly from memory-mapped files or memory buffers, in addition to streams. [1] [2]
Implemented the corresponding logic in the source files, including resource management to keep memory-mapped streams alive as long as the index exists. [1] [2]

Internal implementation changes:

Added and updated constructors and helper methods in FlatIndexImpl and VamanaIndexImpl to support mapping from streams, and to manage the lifetime of mapped streams via unique_ptr members. [1] [2] [3] [4] [5]
Included necessary headers for memory stream support in relevant files. [1] [2] [3] [4]
Updated dataset loading logic to enforce that view allocators are only used with memory streams, improving safety and correctness.

Testing improvements:

Added a generic test helper (write_and_map_index) and comprehensive test cases to verify correct loading and querying of indices from memory-mapped files for all supported storage kinds and index types. [1] [2] [3] [4] [5] [6] [7]

rfsaliev · 2026-04-14T14:32:50Z

+
+    // Load from a memory-mapped file.
+    // The file is expected to be in the format produced by save().
+    static Status map_to_file(
+        VamanaIndex** index, const char* path, MetricType metric, StorageKind storage_kind
+    ) noexcept;
+
+    // Load from a memory buffer.
+    // The buffer is expected to be in the format produced by save().
+    static Status map_to_memory(
+        VamanaIndex** index,
+        const void* data,
+        size_t size,
+        MetricType metric,
+        StorageKind storage_kind
+    ) noexcept;


@alexanderguzhva, is this API suitable?

…ests - Introduced `memstream.h` and `memstream.cpp` for memory-mapped stream functionality. - Updated `io.h` and `simple.h` to include memory stream support. - Enhanced `simple.cpp` and `flat.cpp` tests to validate loading from memory streams.

…emory and memory-mapped stream support

…View

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds mmap/in-memory stream support for loading indices and datasets (zero-copy “view” loading) across the core library and C++ runtime API, plus tests to validate behavior.

Changes:

Introduces mmstream/mmstreambuf, is_memory_stream, and pointer helpers (current_ptr/begin_ptr/end_ptr) plus a MemoryStreamAllocator to enable view-based, zero-copy loading.
Updates loading paths (e.g., SimpleData, SimpleGraph, Vamana graph selection) to support view allocators and to enforce memory-stream requirements.
Extends C++ runtime API with map_to_file / map_to_memory for Flat and Vamana indices and adds new coverage.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
include/svs/core/io/memstream.h	Adds mmapped streambuf/istream + memory-stream utilities and allocator for zero-copy loads
tests/svs/core/io/memstream.cpp	Adds unit tests for mmstream, pointer helpers, and memory-stream detection
include/svs/core/data/simple.h	Adds view-only load path from `ContextFreeLoadTable` + memory-stream enforcement
include/svs/core/data/io.h	Skips `populate()` for view allocators; enforces memory-stream constraint
include/svs/core/graph/graph.h	Allows forwarding allocator args during stream load to support view allocators
include/svs/orchestrators/vamana.h	Adjusts graph type selection for view-allocated data (mmapped/index views)
include/svs/lib/array.h	Generalizes “view allocator” DenseArray specialization to support MemoryStreamAllocator
include/svs/quantization/scalar/scalar.h	Fixes min/max init and adjusts SQDataset load APIs + adds mutable `get_datum`
bindings/cpp/include/svs/runtime/flat_index.h	Adds `map_to_file` / `map_to_memory` for FlatIndex
bindings/cpp/src/flat_index.cpp	Implements FlatIndex mapping APIs
bindings/cpp/src/flat_index_impl.h	Holds mapped stream lifetime and adds `map_to_stream` implementation
bindings/cpp/include/svs/runtime/vamana_index.h	Adds `map_to_file` / `map_to_memory` for VamanaIndex
bindings/cpp/src/vamana_index.cpp	Implements VamanaIndex mapping APIs
bindings/cpp/src/vamana_index_impl.h	Holds mapped stream lifetime and adds `map_to_stream` implementation
bindings/cpp/tests/runtime_test.cpp	Adds runtime tests for mapping APIs across storage kinds
tests/svs/core/data/simple.cpp	Adds tests for loading SimpleDataView from stringstream + error-path test
tests/svs/quantization/scalar/scalar.cpp	Adds SQDataset view-load test from stringstream
tests/svs/index/flat/flat.cpp	Adds FlatIndex view-load tests from stringstream and mmapped file
tests/svs/index/vamana/index.cpp	Adds Vamana view-load tests from stringstream/mmapped file + SQ view-load test
tests/CMakeLists.txt	Registers new memstream test source

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 8 comments.

…stream&) to accept custom accessors

…views Co-authored-by: Copilot <copilot@github.com>

ibhati

Looks great to me!

ethanglaser · 2026-04-23T16:10:09Z

Please address CI failure before merging

ethanglaser · 2026-04-23T16:13:25Z

And if it is expected to require updates from private source to work correctly then we'll need to update https://github.com/intel/ScalableVectorSearch/blob/main/bindings/cpp/CMakeLists.txt#L126 with LTO shared lib build from the private PR (can publish as nightly here)

Co-authored-by: Copilot <copilot@github.com>

…ed allocator handling Co-authored-by: Copilot <copilot@github.com>

…get_allocator to 'view' DenseArray Co-authored-by: Copilot <copilot@github.com>

Co-authored-by: Copilot <copilot@github.com>

rfsaliev requested a review from razdoburdin April 3, 2026 13:25

rfsaliev force-pushed the rfsaliev/mmap-loading branch 2 times, most recently from 1b35ca7 to 9e5588f Compare April 14, 2026 13:32

rfsaliev commented Apr 14, 2026

View reviewed changes

Base automatically changed from dev/razdoburdin_streaming to main April 16, 2026 09:55

rfsaliev added 5 commits April 16, 2026 15:40

Enhance mmap functionality for SimpleGraph and Vamana index with in-m…

cec2c73

…emory and memory-mapped stream support

Add scalar quantization mem-mapping tests

ee59fc8

Refactor SimpleData mmapping using MemoryStreamAllocator rather than …

6963335

…View

Add mem-map CPP Runtime API for Flat and (static)Vamana

a5ed391

rfsaliev force-pushed the rfsaliev/mmap-loading branch from 9e5588f to a5ed391 Compare April 16, 2026 14:25

Refactor mmstream/mmstreambuf using MMapPtr

f52cf19

rfsaliev marked this pull request as ready for review April 20, 2026 09:54

rfsaliev requested review from ahuber21, ethanglaser, ibhati, mihaic and yuejiaointel as code owners April 20, 2026 09:54

rfsaliev requested a review from Copilot April 20, 2026 10:00

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Copilot started reviewing on behalf of rfsaliev April 20, 2026 10:26 View session

rfsaliev added 3 commits April 21, 2026 15:05

Enable rebinding for blocked allocator

82e026f

Address some review comments

d2c6cf2

Fix map_to_memory() by using ispanstream

f04b1c2

rfsaliev force-pushed the rfsaliev/mmap-loading branch from 748365f to f04b1c2 Compare April 22, 2026 12:43

rfsaliev requested a review from Copilot April 22, 2026 12:43

Copilot AI reviewed Apr 22, 2026

View reviewed changes

Copilot started reviewing on behalf of rfsaliev April 22, 2026 14:35 View session

rfsaliev and others added 2 commits April 22, 2026 17:24

Enhance DefaultWriteAccessor with vector_reader and update populate(i…

9d50d0e

…stream&) to accept custom accessors

Refactor SimpleData load methods to handle istream compatibility for …

1ffec19

…views Co-authored-by: Copilot <copilot@github.com>

ibhati approved these changes Apr 23, 2026

View reviewed changes

rfsaliev and others added 2 commits April 24, 2026 10:17

Add copy constructor and stream accessor to MemoryStreamAllocator

bf1f6a3

Co-authored-by: Copilot <copilot@github.com>

Fix CPP Runtime map_to_stream() and SimpleData load method for improv…

ab51fef

…ed allocator handling Co-authored-by: Copilot <copilot@github.com>

razdoburdin mentioned this pull request Apr 27, 2026

Adopt load_dataset with non-default accessors. #320

Closed

rfsaliev and others added 2 commits April 28, 2026 11:19

Refactor MemoryStreamAllocator by adding default constructor and add …

344f341

…get_allocator to 'view' DenseArray Co-authored-by: Copilot <copilot@github.com>

Address code review comments

b7a6c1f

rfsaliev requested a review from Copilot April 28, 2026 16:32

ethanglaser and others added 2 commits April 28, 2026 16:02

Use nightly tarball from private branch

165435b

Enhance StreamArchiver::write_table to align end of written table.

3d2b310

Co-authored-by: Copilot <copilot@github.com>

Conversation

rfsaliev commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rfsaliev Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ibhati left a comment

Choose a reason for hiding this comment

Uh oh!

ethanglaser commented Apr 23, 2026

Uh oh!

ethanglaser commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rfsaliev commented Apr 3, 2026 •

edited

Loading

ethanglaser commented Apr 23, 2026 •

edited

Loading