[Feature]: Support Lazy load collection #30361

congqixia · 2024-01-30T06:13:39Z

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

Milvus 2.x need to load collection before search/query could be executed. In some cases, not all segment is needed for a read operation.
We want Milvus could support a collection, which could search/query certain group of segments without pre-loading them. The segments could be loaded lazily and dynamically.

Describe the solution you'd like.

Add attribute to colllection -- lazy loading
When lazy loading is set to true, search/query could be executed without whole group of segments are loaded.
Delegator could dispatch task with loading & reading merged to query executors.

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

- remove the unused LRU cache - add new LRU cache impl which wraps github.com/karlseguin/ccache related #30361 --------- Signed-off-by: yah01 <yang.cen@zilliz.com>

#30361 Signed-off-by: sunby <sunbingyi1992@gmail.com>

milvus-io#30361 Signed-off-by: sunby <sunbingyi1992@gmail.com>

issue: #30361 --------- Signed-off-by: sunby <sunbingyi1992@gmail.com>

…31977) issue: #30361 - Add configurable disk capacity limit - fix bitset reset logic - make insert record reinsert after clear Signed-off-by: chyezh <chyezh@outlook.com>

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com>

…t when ReleaseData (#32012) issue: #30361 - Delete may be lost when segment is not data-loaded status in lru cache. skip filtering to fix it. - `stats_` and `variable_fields_avg_size_` should be reset when `ReleaseData` - Remove repeat load delta log operation in lru. --------- Signed-off-by: chyezh <chyezh@outlook.com>

…t itself (#32350) issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

Signed-off-by: chyezh <chyezh@outlook.com> Add metric for lru and fix lost delete data when enable lazy load (milvus-io#31868) Signed-off-by: chyezh <chyezh@outlook.com> feat: Support stream reduce v1 (milvus-io#31873) related: milvus-io#31410 --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Change do wait lru dev (milvus-io#31878) Signed-off-by: sunby <sunbingyi1992@gmail.com> enhance: add config for disk cache (milvus-io#31881) fix config not initialized (milvus-io#31890) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix error handle in search (milvus-io#31895) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: thread safe vector (milvus-io#31898) fix: insert record cannot reinsert (milvus-io#31900) enhance: cancel concurrency restrict for stream reduce and add metrics (milvus-io#31892) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: bit set (milvus-io#31905) fix bitset clear to reset (milvus-io#31908) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Fix 0404 lru dev (milvus-io#31914) fix: 1. sealed_segment num_rows reset to std::null opt 2. sealed_segment lazy_load reset to true after evicting to avoid shortcut --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix possible block due to unpin fifo activating principle (milvus-io#31924) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Add lru reloader lru dev (milvus-io#31952) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix query limit (milvus-io#32060) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: lru cache lost delete and wrong mem size (milvus-io#32072) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add more metrics for cache and search (milvus-io#31777) (milvus-io#32097) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com> fix:panic due to empty search result when stream reducing(milvus-io#32009) (milvus-io#32083) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: sealed segment may not exist when throw (milvus-io#32098) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> Major compaction 1st edition (milvus-io#31804) (milvus-io#32116) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Signed-off-by: chasingegg <chao.gao@zilliz.com> Co-authored-by: chasingegg <chao.gao@zilliz.com> fix: inconsistent between state lock and load state (milvus-io#32171) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: Throw error instead of crash when index cannot be built (milvus-io#31844) issue: milvus-io#27589 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> (cherry picked from commit 1b76766) Signed-off-by: jaime <yun.zhang@zilliz.com> update knowhere to support clustering (milvus-io#32188) Signed-off-by: chasingegg <chao.gao@zilliz.com> fix: segment release is not sync with cache (milvus-io#32212) issue: milvus-io#32206 Signed-off-by: chyezh <chyezh@outlook.com> fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136) (milvus-io#32238) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: possible panic when stream reducing (milvus-io#32247) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enhance: [lru-dev] add the related data size for the read apis (milvus-io#32274) cherry-pick: milvus-io#31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com> add debug log (milvus-io#32303) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Refine code for analyze task scheduler (milvus-io#32122) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: memory leak on stream reduce (milvus-io#32345) related: milvus-io#32304 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> feat: adding cache stats support (milvus-io#32344) See milvus-io#32067 Signed-off-by: Ted Xu <ted.xu@zilliz.com> Fix bug for version (milvus-io#32363) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: remove sub entity in load delta log, update entity num in segment itself (milvus-io#32350) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: clear data when loading failure (milvus-io#32370) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: stream reduce memory leak for failing to release stream reducer(milvus-io#32345) (milvus-io#32381) related: milvus-io#32345 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Keep InProgress state when getting task state is init (milvus-io#32394) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> add log for search failed (milvus-io#32367) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enable asan by default (milvus-io#32423) Signed-off-by: sunby <sunbingyi1992@gmail.com> Major compaction refactoring (milvus-io#32149) Signed-off-by: wayblink <anyang.wang@zilliz.com> Lru dev debug (milvus-io#32414) Co-authored-by: wayblink <anyang.wang@zilliz.com> fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock (milvus-io#32436) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: use Get but not GetBy of SegmentManager (milvus-io#32438) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: return growing segment when sealed (milvus-io#32460) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add request resource for lru loading process(milvus-io#32205) (milvus-io#32452) related: milvus-io#32205 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: unexpected deleted index files when lazy loading(milvus-io#32136) (milvus-io#32469) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: reference count leak cause release blocked (milvus-io#32465) issue: milvus-io#32379 Signed-off-by: chyezh <chyezh@outlook.com> Fix compaction fail (milvus-io#32473) Signed-off-by: wayblink <anyang.wang@zilliz.com>

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com>

congqixia added the kind/feature Issues related to feature request from users label Jan 30, 2024

congqixia assigned sunby, yah01, congqixia and MrPresent-Han Jan 30, 2024

yah01 mentioned this issue Jan 30, 2024

enhance: add new LRU cache impl #30360

Merged

xiaofan-luan pushed a commit that referenced this issue Feb 27, 2024

enhance: add new LRU cache impl (#30360)

57397b1

- remove the unused LRU cache - add new LRU cache impl which wraps github.com/karlseguin/ccache related #30361 --------- Signed-off-by: yah01 <yang.cen@zilliz.com>

sunby mentioned this issue Mar 1, 2024

feat: support lazy load on querycoord #30372

Merged

sre-ci-robot pushed a commit that referenced this issue Mar 1, 2024

feat: support lazy load on querycoord (#30372)

7783098

#30361 Signed-off-by: sunby <sunbingyi1992@gmail.com>

MrPresent-Han pushed a commit to MrPresent-Han/milvus that referenced this issue Mar 3, 2024

feat: support lazy load on querycoord (milvus-io#30372)

1f6d805

milvus-io#30361 Signed-off-by: sunby <sunbingyi1992@gmail.com>

sunby mentioned this issue Mar 4, 2024

fix: check collection lazy load prop using schema #30992

Merged

sre-ci-robot pushed a commit that referenced this issue Mar 6, 2024

fix: check collection lazy load prop using schema (#30992)

fd17a5f

issue: #30361 --------- Signed-off-by: sunby <sunbingyi1992@gmail.com>

chyezh mentioned this issue Apr 8, 2024

fix: add configuration disk capacity config for lru and fix some bug #31977

Merged

This was referenced Apr 8, 2024

fix: delete may lost when enable lru cache, some field should be reset when ReleaseData #32012

Merged

fix: lru cache lost delete and wrong mem size #32072

Merged

jaime0815 pushed a commit that referenced this issue Apr 9, 2024

fix: lru cache lost delete and wrong mem size (#32072)

fb0b93d

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 10, 2024

fix: sealed segment may not exist when throw #32098

Merged

tedxu pushed a commit that referenced this issue Apr 10, 2024

fix: sealed segment may not exist when throw (#32098)

adfed78

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 11, 2024

fix: inconsistent between state lock and load state #32171

Merged

jaime0815 pushed a commit that referenced this issue Apr 11, 2024

fix: inconsistent between state lock and load state (#32171)

ab4a583

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh added a commit to chyezh/milvus that referenced this issue Apr 11, 2024

fix: inconsistent between state lock and load state (milvus-io#32171)

59082ee

issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 17, 2024

fix: remove sub entity in load delta log, update entity num in segment itself #32350

Merged

jaime0815 pushed a commit that referenced this issue Apr 17, 2024

fix: remove sub entity in load delta log, update entity num in segmen…

de90fff

…t itself (#32350) issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 17, 2024

fix: clear data when loading failure #32370

Merged

jaime0815 pushed a commit that referenced this issue Apr 17, 2024

fix: clear data when loading failure (#32370)

003dc9f

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

This was referenced Apr 17, 2024

fix: disk cache related fixup patch #32375

Closed

fix: change load lock only apply to sealed segment #32499

Merged

jaime0815 pushed a commit that referenced this issue Apr 22, 2024

fix: change load lock only apply to sealed segment (#32499)

483bb93

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 22, 2024

fix: load index will fail if disable lazyload #32510

Merged

jaime0815 pushed a commit that referenced this issue Apr 23, 2024

fix: load index will fail if disable lazyload (#32510)

d837762

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 24, 2024

fix: unittest failure of lru-dev #32575

Merged

jaime0815 pushed a commit that referenced this issue Apr 24, 2024

fix: unittest failure of lru-dev (#32575)

510399b

issue: #30361 Signed-off-by: chyezh <chyezh@outlook.com>

sunby pushed a commit to sunby/milvus that referenced this issue Apr 24, 2024

fix: unittest failure of lru-dev (milvus-io#32575)

1b173bb

issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com>

sunby pushed a commit to sunby/milvus that referenced this issue Apr 25, 2024

fix: unittest failure of lru-dev (milvus-io#32575)

f090654

issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com>

sunby pushed a commit to sunby/milvus that referenced this issue Apr 26, 2024

fix: unittest failure of lru-dev (milvus-io#32575)

e29cb35

issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com>

sunby pushed a commit to sunby/milvus that referenced this issue Apr 26, 2024

fix: unittest failure of lru-dev (milvus-io#32575)

3af76bc

issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature]: Support Lazy load collection #30361

[Feature]: Support Lazy load collection #30361

congqixia commented Jan 30, 2024

[Feature]: Support Lazy load collection #30361

[Feature]: Support Lazy load collection #30361

Comments

congqixia commented Jan 30, 2024

Is there an existing issue for this?

Is your feature request related to a problem? Please describe.

Describe the solution you'd like.

Describe an alternate solution.

Anything else? (Additional Context)