[Enhancement]: More metrics at query node for elastic scale of query nodes. #30931

chyezh · 2024-02-29T07:36:51Z

Is there an existing issue for this?

I have searched the existing issues

What would you like to be added?

More metrics on query node.

Why is this needed?

Elastic scale of query node is hard to do with current metric of query node.
LRU cache need more metric to monitor. enhance: add new LRU cache impl #30360

Anything else?

No response

#31167) issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

#31177) issue: #30931 pr: #31167 Signed-off-by: chyezh <chyezh@outlook.com>

issue: #30931 - move resource estimate function outside from segment loader. - add load info and collection to base segment. - add resource usage method for sealed segment. Signed-off-by: chyezh <chyezh@outlook.com>

issue: #30931 pr: #31050 - move resource estimate function outside from segment loader. - add load info and collection to base segment. - add resource usage method for sealed segment. Signed-off-by: chyezh <chyezh@outlook.com>

…30936) issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

…ilvus-io#30936) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com>

…31322) issue: #30931 pr: #30936 Signed-off-by: chyezh <chyezh@outlook.com>

stale · 2024-03-31T00:00:17Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

…31715) issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

…ilvus-io#31715) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com> (cherry picked from commit 1ad5ccc) Signed-off-by: jaime <yun.zhang@zilliz.com>

issue: #30931 --------- Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#30931 --------- Signed-off-by: chyezh <chyezh@outlook.com>

issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io/milvus#30931 - Add param `config` for `create_resource_group` api. - Add `update_resource_groups` api to implement declarative resource group api. - Add new example for declarative resource group api. --------- Signed-off-by: chyezh <chyezh@outlook.com>

Signed-off-by: chyezh <chyezh@outlook.com> Add metric for lru and fix lost delete data when enable lazy load (milvus-io#31868) Signed-off-by: chyezh <chyezh@outlook.com> feat: Support stream reduce v1 (milvus-io#31873) related: milvus-io#31410 --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Change do wait lru dev (milvus-io#31878) Signed-off-by: sunby <sunbingyi1992@gmail.com> enhance: add config for disk cache (milvus-io#31881) fix config not initialized (milvus-io#31890) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix error handle in search (milvus-io#31895) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: thread safe vector (milvus-io#31898) fix: insert record cannot reinsert (milvus-io#31900) enhance: cancel concurrency restrict for stream reduce and add metrics (milvus-io#31892) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: bit set (milvus-io#31905) fix bitset clear to reset (milvus-io#31908) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Fix 0404 lru dev (milvus-io#31914) fix: 1. sealed_segment num_rows reset to std::null opt 2. sealed_segment lazy_load reset to true after evicting to avoid shortcut --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix possible block due to unpin fifo activating principle (milvus-io#31924) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Add lru reloader lru dev (milvus-io#31952) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix query limit (milvus-io#32060) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: lru cache lost delete and wrong mem size (milvus-io#32072) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add more metrics for cache and search (milvus-io#31777) (milvus-io#32097) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com> fix:panic due to empty search result when stream reducing(milvus-io#32009) (milvus-io#32083) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: sealed segment may not exist when throw (milvus-io#32098) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> Major compaction 1st edition (milvus-io#31804) (milvus-io#32116) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Signed-off-by: chasingegg <chao.gao@zilliz.com> Co-authored-by: chasingegg <chao.gao@zilliz.com> fix: inconsistent between state lock and load state (milvus-io#32171) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: Throw error instead of crash when index cannot be built (milvus-io#31844) issue: milvus-io#27589 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> (cherry picked from commit 1b76766) Signed-off-by: jaime <yun.zhang@zilliz.com> update knowhere to support clustering (milvus-io#32188) Signed-off-by: chasingegg <chao.gao@zilliz.com> fix: segment release is not sync with cache (milvus-io#32212) issue: milvus-io#32206 Signed-off-by: chyezh <chyezh@outlook.com> fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136) (milvus-io#32238) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: possible panic when stream reducing (milvus-io#32247) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enhance: [lru-dev] add the related data size for the read apis (milvus-io#32274) cherry-pick: milvus-io#31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com> add debug log (milvus-io#32303) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Refine code for analyze task scheduler (milvus-io#32122) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: memory leak on stream reduce (milvus-io#32345) related: milvus-io#32304 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> feat: adding cache stats support (milvus-io#32344) See milvus-io#32067 Signed-off-by: Ted Xu <ted.xu@zilliz.com> Fix bug for version (milvus-io#32363) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: remove sub entity in load delta log, update entity num in segment itself (milvus-io#32350) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: clear data when loading failure (milvus-io#32370) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: stream reduce memory leak for failing to release stream reducer(milvus-io#32345) (milvus-io#32381) related: milvus-io#32345 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Keep InProgress state when getting task state is init (milvus-io#32394) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> add log for search failed (milvus-io#32367) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enable asan by default (milvus-io#32423) Signed-off-by: sunby <sunbingyi1992@gmail.com> Major compaction refactoring (milvus-io#32149) Signed-off-by: wayblink <anyang.wang@zilliz.com> Lru dev debug (milvus-io#32414) Co-authored-by: wayblink <anyang.wang@zilliz.com> fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock (milvus-io#32436) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: use Get but not GetBy of SegmentManager (milvus-io#32438) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: return growing segment when sealed (milvus-io#32460) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add request resource for lru loading process(milvus-io#32205) (milvus-io#32452) related: milvus-io#32205 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: unexpected deleted index files when lazy loading(milvus-io#32136) (milvus-io#32469) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: reference count leak cause release blocked (milvus-io#32465) issue: milvus-io#32379 Signed-off-by: chyezh <chyezh@outlook.com> Fix compaction fail (milvus-io#32473) Signed-off-by: wayblink <anyang.wang@zilliz.com>

issue: milvus-io/milvus#30931 - Add param `config` for `create_resource_group` api. - Add `update_resource_groups` api to implement declarative resource group api. - Add new example for declarative resource group api. --------- Signed-off-by: chyezh <chyezh@outlook.com>

chyezh added the kind/enhancement Issues or changes related to enhancement label Feb 29, 2024

chyezh self-assigned this Feb 29, 2024

This was referenced Feb 29, 2024

enhance: transfer resource group and dbname to querynode when load #30936

Merged

enhance: add resource usage estimate for segment interface #31050

Merged

This was referenced Mar 8, 2024

enhance: add metric with database and resource group label at querynode. #31146

Closed

fix: lost dbname when only passing collection id to describeCollection #31167

Merged

fix: lost dbname when only passing collection id to describeCollection #31177

Merged

sre-ci-robot pushed a commit that referenced this issue Mar 11, 2024

fix: lost dbname when only passing collection id to describeCollection (

6a9418d

#31167) issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

sre-ci-robot pushed a commit that referenced this issue Mar 11, 2024

fix: lost dbname when only passing collection id to describeCollection (

7105e0b

#31177) issue: #30931 pr: #31167 Signed-off-by: chyezh <chyezh@outlook.com>

This was referenced Mar 15, 2024

enhance: add resource usage estimate for segment interface #31309

Merged

enhance: transfer resource group and dbname to querynode when load #31322

Merged

sre-ci-robot pushed a commit that referenced this issue Mar 21, 2024

enhance: transfer resource group and dbname to querynode when load (#…

9f9ef8a

…30936) issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh added a commit to chyezh/milvus that referenced this issue Mar 22, 2024

enhance: transfer resource group and dbname to querynode when load (m…

d2c107f

…ilvus-io#30936) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com>

This was referenced Mar 25, 2024

enhance: add metric with database and resource group label at querynode. #31562

Closed

enhance: add resource group declarative api milvus-io/pymilvus#2002

Merged

sre-ci-robot pushed a commit that referenced this issue Mar 27, 2024

enhance: transfer resource group and dbname to querynode when load (#…

96cec78

…31322) issue: #30931 pr: #30936 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Mar 29, 2024

enhance: add rg and db interface for segment and db/rg metric label #31715

Merged

stale bot added the stale indicates no udpates for 30 days label Mar 31, 2024

sre-ci-robot pushed a commit that referenced this issue Apr 1, 2024

enhance: add rg and db interface for segment and db/rg metric label (#…

1ad5ccc

…31715) issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 1, 2024

enhance: add more metrics for cache and search #31777

Merged

sre-ci-robot pushed a commit that referenced this issue Apr 10, 2024

enhance: add more metrics for cache and search (#31777)

c9faa6d

issue: #30931 --------- Signed-off-by: chyezh <chyezh@outlook.com>

chyezh added a commit to chyezh/milvus that referenced this issue Apr 10, 2024

enhance: add more metrics for cache and search (milvus-io#31777)

1e863e2

issue: milvus-io#30931 --------- Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this issue Apr 10, 2024

enhance: add more metrics for cache and search (#31777) #32097

Merged

jaime0815 pushed a commit that referenced this issue Apr 10, 2024

enhance: add more metrics for cache and search (#31777) (#32097)

84da1bf

issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

stale bot closed this as completed Apr 14, 2024

chyezh mentioned this issue Apr 26, 2024

[Cherry-Pick] enhance: add resource group declarative api (#2002) milvus-io/pymilvus#2054

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement]: More metrics at query node for elastic scale of query nodes. #30931

[Enhancement]: More metrics at query node for elastic scale of query nodes. #30931

chyezh commented Feb 29, 2024

stale bot commented Mar 31, 2024

[Enhancement]: More metrics at query node for elastic scale of query nodes. #30931

[Enhancement]: More metrics at query node for elastic scale of query nodes. #30931

Comments

chyezh commented Feb 29, 2024

Is there an existing issue for this?

What would you like to be added?

Why is this needed?

Anything else?

stale bot commented Mar 31, 2024