sparse float vector index to support mmap when loading from index file #530
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: zhengbuqian The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
buqian-zilliz
force-pushed
the
sparse-index-mmap
branch
from
e14ca08
to
c7202ec
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #530 +/- ##
=========================================
+ Coverage 0 71.69% +71.69%
=========================================
Files 0 67 +67
Lines 0 4431 +4431
=========================================
+ Hits 0 3177 +3177
- Misses 0 1254 +1254 |
| #endif | ||
| } | ||
| map_ = static_cast<char*>(mmap(nullptr, map_size_, PROT_READ, map_flags, reader.descriptor(), 0)); | ||
| madvise(map_, map_size_, MADV_RANDOM); |
There was a problem hiding this comment.
please check the results of mmap and madvise whether any of these two fails
| if (map_ != nullptr) { | ||
| auto res = munmap(map_, map_size_); | ||
| if (res != 0) { | ||
| throw std::runtime_error("failed to munmap when trying to delete index: " + |
There was a problem hiding this comment.
this may cause an exception to be thrown inside ~SparseInvertedIndexNode()
There was a problem hiding this comment.
in the current impl seems the only thing we can do is to swallow the error? we provide no method for the users to explicitly destroy all the resources used so we can only clean in the destructor
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
buqian-zilliz
force-pushed
the
sparse-index-mmap
branch
from
c7202ec
to
8f6c703
Compare
|
/lgtm |
issue: milvus-io/milvus#29419
/kind: improvement