enhance: add more metrics for cache and search #31777

chyezh · 2024-04-01T04:01:51Z

issue: #30931

mergify · 2024-04-01T04:37:55Z

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

codecov · 2024-04-01T06:15:28Z

Codecov Report

Attention: Patch coverage is 89.10506% with 28 lines in your changes are missing coverage. Please review.

Project coverage is 81.56%. Comparing base (1b76766) to head (ee01639).
Report is 11 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #31777      +/-   ##
==========================================
- Coverage   81.63%   81.56%   -0.08%     
==========================================
  Files         978      990      +12     
  Lines      121665   121680      +15     
==========================================
- Hits        99326    99251      -75     
- Misses      18525    18611      +86     
- Partials     3814     3818       +4

Files	Coverage Δ
internal/querynodev2/segments/utils.go	`11.81% <100.00%> (+1.63%)`	⬆️
internal/querynodev2/tasks/query_task.go	`92.85% <100.00%> (+0.26%)`	⬆️
internal/querynodev2/tasks/search_task.go	`90.78% <100.00%> (+6.34%)`	⬆️
pkg/metrics/querynode_metrics.go	`100.00% <100.00%> (ø)`
pkg/util/cache/cache.go	`93.92% <86.66%> (ø)`
...ernal/querynodev2/segments/metricsutil/observer.go	`97.63% <97.63%> (ø)`
internal/querynodev2/segments/retrieve.go	`83.48% <50.00%> (-0.83%)`	⬇️
internal/querynodev2/segments/search.go	`89.85% <50.00%> (-2.09%)`	⬇️
...nternal/querynodev2/segments/metricsutil/record.go	`91.52% <91.52%> (ø)`
internal/querynodev2/segments/manager.go	`79.00% <0.00%> (-1.70%)`	⬇️

... and 220 files with indirect coverage changes

chyezh · 2024-04-01T15:26:41Z

rerun ut

jaime0815 · 2024-04-02T03:28:19Z

internal/querynodev2/segments/metricsutil/observer.go

+		globalObserver = newSegmentsObserver()
+		go func() {
+			d := 15 * time.Minute
+			ticker := time.NewTicker(d)


We'd better make d configurable

it's just a metric expire time. we do not modify it frequently.

configurable metric expire time will be given at main PR: #31562

longjiquan

LGTM

jaime0815 · 2024-04-02T06:12:01Z

/lgtm

chyezh · 2024-04-03T07:57:09Z

rebase and solve the conflict

czs007 · 2024-04-08T02:56:54Z

internal/querynodev2/segments/metricsutil/observer.go

+	case SearchSegmentAccessRecord:
+		o.prom.ObserveSearchAccess(mm)
+	default:
+		panic("unknown segment access metric")


just log.Warn?

it's just an assertion.
it's an unreachable code branch.
we can find bug when develop and CI by panic message easily.

czs007 · 2024-04-08T07:41:51Z

internal/querynodev2/segments/metricsutil/record.go

+
+// CacheLoadRecord records the metrics of a cache load.
+type CacheLoadRecord struct {
+	bytes uint64


numBytes will be better?

Signed-off-by: chyezh <chyezh@outlook.com>

chyezh · 2024-04-09T06:10:11Z

rebase and remove the conflict

mergify · 2024-04-09T07:18:19Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-04-09T07:26:49Z

/run-cpu-e2e

chyezh · 2024-04-09T09:32:11Z

rerun ut

czs007 · 2024-04-10T02:53:56Z

/approve
/lgtm

sre-ci-robot · 2024-04-10T02:54:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chyezh, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

issue: milvus-io#30931 --------- Signed-off-by: chyezh <chyezh@outlook.com>

issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh · 2024-04-15T08:16:54Z

related dev PR: #31868, #32097

Signed-off-by: chyezh <chyezh@outlook.com> Add metric for lru and fix lost delete data when enable lazy load (milvus-io#31868) Signed-off-by: chyezh <chyezh@outlook.com> feat: Support stream reduce v1 (milvus-io#31873) related: milvus-io#31410 --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Change do wait lru dev (milvus-io#31878) Signed-off-by: sunby <sunbingyi1992@gmail.com> enhance: add config for disk cache (milvus-io#31881) fix config not initialized (milvus-io#31890) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix error handle in search (milvus-io#31895) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: thread safe vector (milvus-io#31898) fix: insert record cannot reinsert (milvus-io#31900) enhance: cancel concurrency restrict for stream reduce and add metrics (milvus-io#31892) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: bit set (milvus-io#31905) fix bitset clear to reset (milvus-io#31908) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Fix 0404 lru dev (milvus-io#31914) fix: 1. sealed_segment num_rows reset to std::null opt 2. sealed_segment lazy_load reset to true after evicting to avoid shortcut --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix possible block due to unpin fifo activating principle (milvus-io#31924) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Add lru reloader lru dev (milvus-io#31952) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix query limit (milvus-io#32060) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: lru cache lost delete and wrong mem size (milvus-io#32072) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add more metrics for cache and search (milvus-io#31777) (milvus-io#32097) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com> fix:panic due to empty search result when stream reducing(milvus-io#32009) (milvus-io#32083) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: sealed segment may not exist when throw (milvus-io#32098) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> Major compaction 1st edition (milvus-io#31804) (milvus-io#32116) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Signed-off-by: chasingegg <chao.gao@zilliz.com> Co-authored-by: chasingegg <chao.gao@zilliz.com> fix: inconsistent between state lock and load state (milvus-io#32171) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: Throw error instead of crash when index cannot be built (milvus-io#31844) issue: milvus-io#27589 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> (cherry picked from commit 1b76766) Signed-off-by: jaime <yun.zhang@zilliz.com> update knowhere to support clustering (milvus-io#32188) Signed-off-by: chasingegg <chao.gao@zilliz.com> fix: segment release is not sync with cache (milvus-io#32212) issue: milvus-io#32206 Signed-off-by: chyezh <chyezh@outlook.com> fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136) (milvus-io#32238) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: possible panic when stream reducing (milvus-io#32247) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enhance: [lru-dev] add the related data size for the read apis (milvus-io#32274) cherry-pick: milvus-io#31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com> add debug log (milvus-io#32303) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Refine code for analyze task scheduler (milvus-io#32122) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: memory leak on stream reduce (milvus-io#32345) related: milvus-io#32304 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> feat: adding cache stats support (milvus-io#32344) See milvus-io#32067 Signed-off-by: Ted Xu <ted.xu@zilliz.com> Fix bug for version (milvus-io#32363) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: remove sub entity in load delta log, update entity num in segment itself (milvus-io#32350) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: clear data when loading failure (milvus-io#32370) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: stream reduce memory leak for failing to release stream reducer(milvus-io#32345) (milvus-io#32381) related: milvus-io#32345 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Keep InProgress state when getting task state is init (milvus-io#32394) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> add log for search failed (milvus-io#32367) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enable asan by default (milvus-io#32423) Signed-off-by: sunby <sunbingyi1992@gmail.com> Major compaction refactoring (milvus-io#32149) Signed-off-by: wayblink <anyang.wang@zilliz.com> Lru dev debug (milvus-io#32414) Co-authored-by: wayblink <anyang.wang@zilliz.com> fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock (milvus-io#32436) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: use Get but not GetBy of SegmentManager (milvus-io#32438) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: return growing segment when sealed (milvus-io#32460) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add request resource for lru loading process(milvus-io#32205) (milvus-io#32452) related: milvus-io#32205 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: unexpected deleted index files when lazy loading(milvus-io#32136) (milvus-io#32469) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: reference count leak cause release blocked (milvus-io#32465) issue: milvus-io#32379 Signed-off-by: chyezh <chyezh@outlook.com> Fix compaction fail (milvus-io#32473) Signed-off-by: wayblink <anyang.wang@zilliz.com>

sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Apr 1, 2024

sre-ci-robot requested review from congqixia and czs007 April 1, 2024 04:02

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Apr 1, 2024

chyezh mentioned this pull request Apr 1, 2024

enhance: add metric with database and resource group label at querynode. #31562

Closed

chyezh force-pushed the enhance_rg_level_metrics_3 branch from 837fe3b to 83b19ef Compare April 1, 2024 06:54

mergify bot added the ci-passed label Apr 1, 2024

jaime0815 reviewed Apr 2, 2024

View reviewed changes

longjiquan reviewed Apr 2, 2024

View reviewed changes

sre-ci-robot assigned jaime0815 Apr 2, 2024

sre-ci-robot added the lgtm label Apr 2, 2024

mergify bot removed the ci-passed label Apr 3, 2024

mergify bot mentioned this pull request Apr 3, 2024

enhance: add more metrics for cache and search #31867

Closed

chyezh force-pushed the enhance_rg_level_metrics_3 branch from 83b19ef to 250cb76 Compare April 3, 2024 07:56

sre-ci-robot removed the lgtm label Apr 3, 2024

mergify bot added the ci-passed label Apr 3, 2024

czs007 reviewed Apr 8, 2024

View reviewed changes

chyezh force-pushed the enhance_rg_level_metrics_3 branch from 250cb76 to 65f9316 Compare April 8, 2024 08:53

mergify bot removed the ci-passed label Apr 8, 2024

chyezh force-pushed the enhance_rg_level_metrics_3 branch from 65f9316 to 8c09950 Compare April 8, 2024 11:35

chyezh added 3 commits April 9, 2024 14:04

enhance: add more metrics for cache and search

fadce61

Signed-off-by: chyezh <chyezh@outlook.com>

fix: use millisecond, add global histo

4f0e247

Signed-off-by: chyezh <chyezh@outlook.com>

enhance: add missing for DoWait

ee01639

Signed-off-by: chyezh <chyezh@outlook.com>

chyezh force-pushed the enhance_rg_level_metrics_3 branch from 8c09950 to ee01639 Compare April 9, 2024 06:09

mergify bot added the ci-passed label Apr 10, 2024

sre-ci-robot assigned czs007 Apr 10, 2024

sre-ci-robot added the lgtm label Apr 10, 2024

sre-ci-robot added the approved label Apr 10, 2024

sre-ci-robot merged commit c9faa6d into milvus-io:master Apr 10, 2024
15 checks passed

chyezh deleted the enhance_rg_level_metrics_3 branch April 10, 2024 02:56

chyezh added a commit to chyezh/milvus that referenced this pull request Apr 10, 2024

enhance: add more metrics for cache and search (milvus-io#31777)

1e863e2

issue: milvus-io#30931 --------- Signed-off-by: chyezh <chyezh@outlook.com>

jaime0815 pushed a commit that referenced this pull request Apr 10, 2024

enhance: add more metrics for cache and search (#31777) (#32097)

84da1bf

issue: #30931 Signed-off-by: chyezh <chyezh@outlook.com>

chyezh mentioned this pull request Apr 15, 2024

enhance: add more metrics for cache and search (#31777) #32097

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: add more metrics for cache and search #31777

enhance: add more metrics for cache and search #31777

chyezh commented Apr 1, 2024

mergify bot commented Apr 1, 2024

codecov bot commented Apr 1, 2024 •

edited

chyezh commented Apr 1, 2024

jaime0815 Apr 2, 2024

chyezh Apr 2, 2024

longjiquan left a comment

jaime0815 commented Apr 2, 2024

chyezh commented Apr 3, 2024

czs007 Apr 8, 2024

chyezh Apr 8, 2024

czs007 Apr 8, 2024

chyezh commented Apr 9, 2024

mergify bot commented Apr 9, 2024

chyezh commented Apr 9, 2024

chyezh commented Apr 9, 2024

czs007 commented Apr 10, 2024

sre-ci-robot commented Apr 10, 2024

chyezh commented Apr 15, 2024 •

edited

enhance: add more metrics for cache and search #31777

enhance: add more metrics for cache and search #31777

Conversation

chyezh commented Apr 1, 2024

mergify bot commented Apr 1, 2024

codecov bot commented Apr 1, 2024 • edited

Codecov Report

chyezh commented Apr 1, 2024

jaime0815 Apr 2, 2024

Choose a reason for hiding this comment

chyezh Apr 2, 2024

Choose a reason for hiding this comment

longjiquan left a comment

Choose a reason for hiding this comment

jaime0815 commented Apr 2, 2024

chyezh commented Apr 3, 2024

czs007 Apr 8, 2024

Choose a reason for hiding this comment

chyezh Apr 8, 2024

Choose a reason for hiding this comment

czs007 Apr 8, 2024

Choose a reason for hiding this comment

chyezh commented Apr 9, 2024

mergify bot commented Apr 9, 2024

chyezh commented Apr 9, 2024

chyezh commented Apr 9, 2024

czs007 commented Apr 10, 2024

sre-ci-robot commented Apr 10, 2024

chyezh commented Apr 15, 2024 • edited

codecov bot commented Apr 1, 2024 •

edited

chyezh commented Apr 15, 2024 •

edited