[Feature]: Support Streaming Reduce for Milvus Serverless Mode #31410
Assignees
Labels
kind/feature
Issues related to feature request from users
Comments
MrPresent-Han
added
the
kind/feature
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 19, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 19, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 20, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 21, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 21, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 21, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 21, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 21, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 22, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 22, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 25, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 25, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 25, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 25, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 25, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 26, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 26, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 26, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 27, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 27, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 27, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 27, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 27, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 27, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 28, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 28, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 28, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 28, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 28, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 29, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Mar 29, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
chyezh
pushed a commit
to chyezh/milvus
that referenced
this issue
Apr 2, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
chyezh
pushed a commit
to chyezh/milvus
that referenced
this issue
Apr 2, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
chyezh
pushed a commit
to chyezh/milvus
that referenced
this issue
Apr 2, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Apr 3, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Apr 3, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Apr 3, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
jaime0815
pushed a commit
that referenced
this issue
Apr 3, 2024
related: #31410 --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Apr 16, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Apr 22, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
sunby
pushed a commit
to sunby/milvus
that referenced
this issue
Apr 22, 2024
Signed-off-by: chyezh <chyezh@outlook.com> Add metric for lru and fix lost delete data when enable lazy load (milvus-io#31868) Signed-off-by: chyezh <chyezh@outlook.com> feat: Support stream reduce v1 (milvus-io#31873) related: milvus-io#31410 --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Change do wait lru dev (milvus-io#31878) Signed-off-by: sunby <sunbingyi1992@gmail.com> enhance: add config for disk cache (milvus-io#31881) fix config not initialized (milvus-io#31890) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix error handle in search (milvus-io#31895) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: thread safe vector (milvus-io#31898) fix: insert record cannot reinsert (milvus-io#31900) enhance: cancel concurrency restrict for stream reduce and add metrics (milvus-io#31892) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: bit set (milvus-io#31905) fix bitset clear to reset (milvus-io#31908) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Fix 0404 lru dev (milvus-io#31914) fix: 1. sealed_segment num_rows reset to std::null opt 2. sealed_segment lazy_load reset to true after evicting to avoid shortcut --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix possible block due to unpin fifo activating principle (milvus-io#31924) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Add lru reloader lru dev (milvus-io#31952) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix query limit (milvus-io#32060) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: lru cache lost delete and wrong mem size (milvus-io#32072) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add more metrics for cache and search (milvus-io#31777) (milvus-io#32097) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com> fix:panic due to empty search result when stream reducing(milvus-io#32009) (milvus-io#32083) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: sealed segment may not exist when throw (milvus-io#32098) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> Major compaction 1st edition (milvus-io#31804) (milvus-io#32116) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Signed-off-by: chasingegg <chao.gao@zilliz.com> Co-authored-by: chasingegg <chao.gao@zilliz.com> fix: inconsistent between state lock and load state (milvus-io#32171) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: Throw error instead of crash when index cannot be built (milvus-io#31844) issue: milvus-io#27589 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> (cherry picked from commit 1b76766) Signed-off-by: jaime <yun.zhang@zilliz.com> update knowhere to support clustering (milvus-io#32188) Signed-off-by: chasingegg <chao.gao@zilliz.com> fix: segment release is not sync with cache (milvus-io#32212) issue: milvus-io#32206 Signed-off-by: chyezh <chyezh@outlook.com> fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136) (milvus-io#32238) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: possible panic when stream reducing (milvus-io#32247) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enhance: [lru-dev] add the related data size for the read apis (milvus-io#32274) cherry-pick: milvus-io#31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com> add debug log (milvus-io#32303) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Refine code for analyze task scheduler (milvus-io#32122) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: memory leak on stream reduce (milvus-io#32345) related: milvus-io#32304 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> feat: adding cache stats support (milvus-io#32344) See milvus-io#32067 Signed-off-by: Ted Xu <ted.xu@zilliz.com> Fix bug for version (milvus-io#32363) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: remove sub entity in load delta log, update entity num in segment itself (milvus-io#32350) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: clear data when loading failure (milvus-io#32370) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: stream reduce memory leak for failing to release stream reducer(milvus-io#32345) (milvus-io#32381) related: milvus-io#32345 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Keep InProgress state when getting task state is init (milvus-io#32394) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> add log for search failed (milvus-io#32367) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enable asan by default (milvus-io#32423) Signed-off-by: sunby <sunbingyi1992@gmail.com> Major compaction refactoring (milvus-io#32149) Signed-off-by: wayblink <anyang.wang@zilliz.com> Lru dev debug (milvus-io#32414) Co-authored-by: wayblink <anyang.wang@zilliz.com> fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock (milvus-io#32436) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: use Get but not GetBy of SegmentManager (milvus-io#32438) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: return growing segment when sealed (milvus-io#32460) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add request resource for lru loading process(milvus-io#32205) (milvus-io#32452) related: milvus-io#32205 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: unexpected deleted index files when lazy loading(milvus-io#32136) (milvus-io#32469) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: reference count leak cause release blocked (milvus-io#32465) issue: milvus-io#32379 Signed-off-by: chyezh <chyezh@outlook.com> Fix compaction fail (milvus-io#32473) Signed-off-by: wayblink <anyang.wang@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Apr 22, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han
added a commit
to MrPresent-Han/milvus
that referenced
this issue
Apr 23, 2024
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Is your feature request related to a problem? Please describe.
Currently, the reduce operation on the querynode is batch-mode, meaning that reducing required all search_results are already perfectly prepared without really retrieving output data from the segment. This existing mechanism blocks the Serverless development of milvus because all the segments needed to retrieve output data had to be locked on the querynode and cannot be released for memory space. This feature is targeted to solve this problem
Describe the solution you'd like.
Change Reduce Behaviour on the querynode to stream style
Describe an alternate solution.
No response
Anything else? (Additional Context)
No response
The text was updated successfully, but these errors were encountered: