enhance: add the related data size for the read apis #31816

SimFG · 2024-04-02T04:13:09Z

issue: #30436
origin pr: #30438
related pr: #31772

codecov · 2024-04-02T09:23:02Z

Codecov Report

Attention: Patch coverage is 93.29268% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 81.64%. Comparing base (089c805) to head (56fa3b6).
Report is 10 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #31816      +/-   ##
==========================================
+ Coverage   81.58%   81.64%   +0.06%     
==========================================
  Files         987      987              
  Lines      121316   121394      +78     
==========================================
+ Hits        98976    99113     +137     
+ Misses      18529    18480      -49     
+ Partials     3811     3801      -10

Files	Coverage Δ
internal/datacoord/metrics_info.go	`94.15% <100.00%> (ø)`
internal/proxy/impl.go	`86.45% <100.00%> (+0.11%)`	⬆️
internal/proxy/task_query.go	`81.44% <100.00%> (+0.63%)`	⬆️
internal/proxy/task_search.go	`77.76% <100.00%> (+0.07%)`	⬆️
internal/querynodev2/segments/count_reducer.go	`100.00% <100.00%> (ø)`
internal/querynodev2/segments/retrieve.go	`84.76% <100.00%> (+0.44%)`	⬆️
internal/querynodev2/services.go	`90.56% <100.00%> (-0.69%)`	⬇️
internal/querynodev2/tasks/query_task.go	`92.94% <100.00%> (+0.34%)`	⬆️
internal/querynodev2/tasks/search_task.go	`90.82% <100.00%> (+6.38%)`	⬆️
pkg/metrics/proxy_metrics.go	`100.00% <100.00%> (ø)`
... and 3 more

... and 32 files with indirect coverage changes

pkg/metrics/metrics.go

pkg/metrics/proxy_metrics.go

internal/querynodev2/tasks/query_task.go

jaime0815 · 2024-04-09T02:31:07Z

/lgtm

Signed-off-by: SimFG <bang.fu@zilliz.com>

mergify · 2024-04-09T09:23:49Z

@SimFG E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

SimFG · 2024-04-09T11:01:42Z

/run-cpu-e2e

mergify · 2024-04-09T11:41:52Z

@SimFG E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

SimFG · 2024-04-09T11:47:41Z

/run-cpu-e2e

mergify · 2024-04-09T12:07:58Z

@SimFG E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

SimFG · 2024-04-09T12:10:03Z

/run-cpu-e2e

SimFG · 2024-04-09T12:23:52Z

/run-cpu-e2e

samhuang-z · 2024-04-09T12:48:46Z

/run-cpu-e2e

samhuang-z · 2024-04-09T12:55:45Z

/run-cpu-e2e

samhuang-z · 2024-04-09T13:05:21Z

/run-cpu-e2e

internal/proxy/impl.go

czs007 · 2024-04-10T06:50:25Z

/approve

sre-ci-robot · 2024-04-10T06:50:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, SimFG

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

czs007 · 2024-04-10T07:05:23Z

/lgtm

cherry-pick: #31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: chyezh <chyezh@outlook.com> Add metric for lru and fix lost delete data when enable lazy load (milvus-io#31868) Signed-off-by: chyezh <chyezh@outlook.com> feat: Support stream reduce v1 (milvus-io#31873) related: milvus-io#31410 --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Change do wait lru dev (milvus-io#31878) Signed-off-by: sunby <sunbingyi1992@gmail.com> enhance: add config for disk cache (milvus-io#31881) fix config not initialized (milvus-io#31890) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix error handle in search (milvus-io#31895) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: thread safe vector (milvus-io#31898) fix: insert record cannot reinsert (milvus-io#31900) enhance: cancel concurrency restrict for stream reduce and add metrics (milvus-io#31892) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: bit set (milvus-io#31905) fix bitset clear to reset (milvus-io#31908) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Fix 0404 lru dev (milvus-io#31914) fix: 1. sealed_segment num_rows reset to std::null opt 2. sealed_segment lazy_load reset to true after evicting to avoid shortcut --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix possible block due to unpin fifo activating principle (milvus-io#31924) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Add lru reloader lru dev (milvus-io#31952) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix query limit (milvus-io#32060) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: lru cache lost delete and wrong mem size (milvus-io#32072) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add more metrics for cache and search (milvus-io#31777) (milvus-io#32097) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com> fix:panic due to empty search result when stream reducing(milvus-io#32009) (milvus-io#32083) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: sealed segment may not exist when throw (milvus-io#32098) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> Major compaction 1st edition (milvus-io#31804) (milvus-io#32116) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Signed-off-by: chasingegg <chao.gao@zilliz.com> Co-authored-by: chasingegg <chao.gao@zilliz.com> fix: inconsistent between state lock and load state (milvus-io#32171) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: Throw error instead of crash when index cannot be built (milvus-io#31844) issue: milvus-io#27589 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> (cherry picked from commit 1b76766) Signed-off-by: jaime <yun.zhang@zilliz.com> update knowhere to support clustering (milvus-io#32188) Signed-off-by: chasingegg <chao.gao@zilliz.com> fix: segment release is not sync with cache (milvus-io#32212) issue: milvus-io#32206 Signed-off-by: chyezh <chyezh@outlook.com> fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136) (milvus-io#32238) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: possible panic when stream reducing (milvus-io#32247) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enhance: [lru-dev] add the related data size for the read apis (milvus-io#32274) cherry-pick: milvus-io#31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com> add debug log (milvus-io#32303) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Refine code for analyze task scheduler (milvus-io#32122) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: memory leak on stream reduce (milvus-io#32345) related: milvus-io#32304 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> feat: adding cache stats support (milvus-io#32344) See milvus-io#32067 Signed-off-by: Ted Xu <ted.xu@zilliz.com> Fix bug for version (milvus-io#32363) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: remove sub entity in load delta log, update entity num in segment itself (milvus-io#32350) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: clear data when loading failure (milvus-io#32370) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: stream reduce memory leak for failing to release stream reducer(milvus-io#32345) (milvus-io#32381) related: milvus-io#32345 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Keep InProgress state when getting task state is init (milvus-io#32394) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> add log for search failed (milvus-io#32367) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enable asan by default (milvus-io#32423) Signed-off-by: sunby <sunbingyi1992@gmail.com> Major compaction refactoring (milvus-io#32149) Signed-off-by: wayblink <anyang.wang@zilliz.com> Lru dev debug (milvus-io#32414) Co-authored-by: wayblink <anyang.wang@zilliz.com> fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock (milvus-io#32436) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: use Get but not GetBy of SegmentManager (milvus-io#32438) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: return growing segment when sealed (milvus-io#32460) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add request resource for lru loading process(milvus-io#32205) (milvus-io#32452) related: milvus-io#32205 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: unexpected deleted index files when lazy loading(milvus-io#32136) (milvus-io#32469) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: reference count leak cause release blocked (milvus-io#32465) issue: milvus-io#32379 Signed-off-by: chyezh <chyezh@outlook.com> Fix compaction fail (milvus-io#32473) Signed-off-by: wayblink <anyang.wang@zilliz.com>

sre-ci-robot requested review from aoiasd and bigsheeper April 2, 2024 04:13

sre-ci-robot added size/L Denotes a PR that changes 100-499 lines. area/internal-api area/test sig/testing test/integration integration test labels Apr 2, 2024

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Apr 2, 2024

SimFG force-pushed the master branch 2 times, most recently from e210195 to 80795b5 Compare April 2, 2024 07:04

mergify bot added the ci-passed label Apr 2, 2024

jaime0815 reviewed Apr 2, 2024

View reviewed changes

pkg/metrics/metrics.go Outdated Show resolved Hide resolved

jaime0815 reviewed Apr 2, 2024

View reviewed changes

pkg/metrics/proxy_metrics.go Show resolved Hide resolved

jaime0815 reviewed Apr 2, 2024

View reviewed changes

internal/querynodev2/tasks/query_task.go Show resolved Hide resolved

SimFG force-pushed the master branch from 80795b5 to 1f8f62b Compare April 7, 2024 02:27

mergify bot added ci-passed and removed ci-passed labels Apr 7, 2024

sre-ci-robot assigned jaime0815 Apr 9, 2024

sre-ci-robot added the lgtm label Apr 9, 2024

SimFG force-pushed the master branch from 1f8f62b to d9f408d Compare April 9, 2024 06:21

sre-ci-robot removed the lgtm label Apr 9, 2024

mergify bot removed the ci-passed label Apr 9, 2024

SimFG added 2 commits April 9, 2024 14:41

enhance: add the related data size for the read apis

0bcde31

Signed-off-by: SimFG <bang.fu@zilliz.com>

fix: handle the pr conflict of the hybrid search

56fa3b6

Signed-off-by: SimFG <bang.fu@zilliz.com>

SimFG force-pushed the master branch from d9f408d to 56fa3b6 Compare April 9, 2024 07:57

mergify bot added the ci-passed label Apr 9, 2024

czs007 reviewed Apr 10, 2024

View reviewed changes

internal/proxy/impl.go Show resolved Hide resolved

sre-ci-robot added the approved label Apr 10, 2024

sre-ci-robot assigned czs007 Apr 10, 2024

sre-ci-robot added the lgtm label Apr 10, 2024

sre-ci-robot merged commit 90bed1c into milvus-io:master Apr 10, 2024
15 checks passed

SimFG mentioned this pull request Apr 15, 2024

enhance: [lru-dev] add the related data size for the read apis #32274

Merged

jaime0815 pushed a commit that referenced this pull request Apr 15, 2024

enhance: [lru-dev] add the related data size for the read apis (#32274)

be3d4c1

cherry-pick: #31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: add the related data size for the read apis #31816

enhance: add the related data size for the read apis #31816

SimFG commented Apr 2, 2024

codecov bot commented Apr 2, 2024 •

edited

jaime0815 commented Apr 9, 2024

mergify bot commented Apr 9, 2024

SimFG commented Apr 9, 2024

mergify bot commented Apr 9, 2024

SimFG commented Apr 9, 2024

mergify bot commented Apr 9, 2024

SimFG commented Apr 9, 2024

SimFG commented Apr 9, 2024

samhuang-z commented Apr 9, 2024

samhuang-z commented Apr 9, 2024

samhuang-z commented Apr 9, 2024

czs007 commented Apr 10, 2024

sre-ci-robot commented Apr 10, 2024

czs007 commented Apr 10, 2024

enhance: add the related data size for the read apis #31816

enhance: add the related data size for the read apis #31816

Conversation

SimFG commented Apr 2, 2024

codecov bot commented Apr 2, 2024 • edited

Codecov Report

jaime0815 commented Apr 9, 2024

mergify bot commented Apr 9, 2024

SimFG commented Apr 9, 2024

mergify bot commented Apr 9, 2024

SimFG commented Apr 9, 2024

mergify bot commented Apr 9, 2024

SimFG commented Apr 9, 2024

SimFG commented Apr 9, 2024

samhuang-z commented Apr 9, 2024

samhuang-z commented Apr 9, 2024

samhuang-z commented Apr 9, 2024

czs007 commented Apr 10, 2024

sre-ci-robot commented Apr 10, 2024

czs007 commented Apr 10, 2024

codecov bot commented Apr 2, 2024 •

edited