enhance: Speed up target recovery after query coord restart #31240

weiliu1031 · 2024-03-13T12:19:25Z

after querycoord restart, it will pull a new target, which include channel and segment list. when segments loaded on querynode has reached the target, the collection could provide search/query. but if segment list changes by time, ater querycoord pull a new target, it will takes a few minutes to catch up the target's segment distribution. and before that, query/search will fail due to lack of segments.

This PR save the current loaded target to meta storein querycoord's stop progress, and recover it when query coord starts, to speed up the target recovery time.

mergify · 2024-03-13T13:23:27Z

@weiliu1031 ut workflow job failed, comment rerun ut can trigger the job again.

jaime0815 · 2024-03-13T14:24:33Z

internal/querycoordv2/meta/target_manager.go

+	mgr.rwMutex.Lock()
+	defer mgr.rwMutex.Unlock()
+	if mgr.current != nil {
+		for id, target := range mgr.current.collectionTargetMap {


collection target Map may be very large which is not friendly for etcd IO？

saveTarget and recoverTarget only happens when qc restarts, the trigger frequency is low.

the pbTarget only contains (channel Names, segment, cp), which won't be too larger. for example, a collection with 30w segment's target is about 3MB.

use zstd to compress and decompress the pbTarget, for example, the 3MB target could be compressed to 600KB.

codecov · 2024-03-14T04:24:35Z

Codecov Report

Attention: Patch coverage is 85.23490% with 22 lines in your changes are missing coverage. Please review.

Project coverage is 81.08%. Comparing base (056e9f0) to head (5f725f7).
Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #31240      +/-   ##
==========================================
+ Coverage   81.02%   81.08%   +0.05%     
==========================================
  Files         978      978              
  Lines      142852   143065     +213     
==========================================
+ Hits       115750   116004     +254     
+ Misses      23224    23188      -36     
+ Partials     3878     3873       -5

Files	Coverage Δ
internal/metastore/catalog.go	`100.00% <ø> (ø)`
tests/integration/minicluster_v2.go	`63.42% <100.00%> (ø)`
internal/distributed/querycoord/service.go	`73.64% <0.00%> (-1.26%)`	⬇️
internal/querycoordv2/meta/target.go	`93.87% <95.52%> (+3.55%)`	⬆️
internal/querycoordv2/server.go	`80.95% <62.50%> (-0.52%)`	⬇️
internal/metastore/kv/querycoord/kv_catalog.go	`69.65% <88.23%> (+3.78%)`	⬆️
internal/querycoordv2/meta/target_manager.go	`84.38% <70.58%> (-0.47%)`	⬇️

... and 31 files with indirect coverage changes

jaime0815 · 2024-03-14T11:43:57Z

Do we have any testing for recovery time with a large of number write or delete requests, and checking if downtime is 0 during the rolling upgrade/restart period?

internal/querycoordv2/meta/target_manager.go

internal/metastore/kv/querycoord/kv_catalog.go

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

jaime0815 · 2024-03-15T06:03:09Z

/lgtm

congqixia · 2024-03-15T06:18:06Z

/approve

sre-ci-robot · 2024-03-15T06:18:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, weiliu1031

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [congqixia]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…o#31240) issue: milvus-io#28491 after querycoord restart, it will pull a new target, which include channel and segment list. when segments loaded on querynode has reached the target, the collection could provide search/query. but if segment list changes by time, ater querycoord pull a new target, it will takes a few minutes to catch up the target's segment distribution. and before that, query/search will fail due to lack of segments. This PR save the current loaded target to meta storein querycoord's stop progress, and recover it when query coord starts, to speed up the target recovery time. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>

…31449) issue: #28491 pr: #31240 after querycoord restart, it will pull a new target, which include channel and segment list. when segments loaded on querynode has reached the target, the collection could provide search/query. but if segment list changes by time, ater querycoord pull a new target, it will takes a few minutes to catch up the target's segment distribution. and before that, query/search will fail due to lack of segments. This PR save the current loaded target to meta storein querycoord's stop progress, and recover it when query coord starts, to speed up the target recovery time. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>

See also milvus-io#28491 milvus-io#31240 When colleciton number is large, querycoord saves collection target one by one, which is slow and may block querycoord exits. In local run, 500 collections scenario may lead to about 40 seconds saving collection targets. This PR changes the `SaveCollectionTarget` interface into batch one and organizes the collection in 16 per bundle batches to accelerate this procedure. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

See also #28491 #31240 When colleciton number is large, querycoord saves collection target one by one, which is slow and may block querycoord exits. In local run, 500 collections scenario may lead to about 40 seconds saving collection targets. This PR changes the `SaveCollectionTarget` interface into batch one and organizes the collection in 16 per bundle batches to accelerate this procedure. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

See also milvus-io#28491 milvus-io#31240 When colleciton number is large, querycoord saves collection target one by one, which is slow and may block querycoord exits. In local run, 500 collections scenario may lead to about 40 seconds saving collection targets. This PR changes the `SaveCollectionTarget` interface into batch one and organizes the collection in 16 per bundle batches to accelerate this procedure. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Cherry-pick from master pr: #31616 See also #28491 #31240 When colleciton number is large, querycoord saves collection target one by one, which is slow and may block querycoord exits. In local run, 500 collections scenario may lead to about 40 seconds saving collection targets. This PR changes the `SaveCollectionTarget` interface into batch one and organizes the collection in 16 per bundle batches to accelerate this procedure. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

See also milvus-io#28491 milvus-io#31240 When colleciton number is large, querycoord saves collection target one by one, which is slow and may block querycoord exits. In local run, 500 collections scenario may lead to about 40 seconds saving collection targets. This PR changes the `SaveCollectionTarget` interface into batch one and organizes the collection in 16 per bundle batches to accelerate this procedure. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

…31655) Cherry-pick from master pr: #31616 See also #28491 #31240 When colleciton number is large, querycoord saves collection target one by one, which is slow and may block querycoord exits. In local run, 500 collections scenario may lead to about 40 seconds saving collection targets. This PR changes the `SaveCollectionTarget` interface into batch one and organizes the collection in 16 per bundle batches to accelerate this procedure. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Mar 13, 2024

sre-ci-robot requested review from congqixia and godchen0212 March 13, 2024 12:19

sre-ci-robot added area/internal-api area/test sig/testing test/integration integration test labels Mar 13, 2024

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Mar 13, 2024

jaime0815 reviewed Mar 13, 2024

View reviewed changes

mergify bot added the ci-passed label Mar 14, 2024

jaime0815 reviewed Mar 14, 2024

View reviewed changes

internal/querycoordv2/meta/target_manager.go Outdated Show resolved Hide resolved

congqixia reviewed Mar 14, 2024

View reviewed changes

internal/metastore/kv/querycoord/kv_catalog.go Outdated Show resolved Hide resolved

weiliu1031 added 3 commits March 15, 2024 10:38

enhance: Speed up target recovery after query coord restart

aeaa858

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

fix ut

5090ffc

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

fix review comment

5f725f7

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

weiliu1031 force-pushed the speed_up_target_recover branch from e1b9d64 to 5f725f7 Compare March 15, 2024 02:39

mergify bot added ci-passed and removed ci-passed labels Mar 15, 2024

sre-ci-robot assigned jaime0815 Mar 15, 2024

sre-ci-robot added the lgtm label Mar 15, 2024

sre-ci-robot added the approved label Mar 15, 2024

sre-ci-robot merged commit d79aa58 into milvus-io:master Mar 15, 2024
14 checks passed

weiliu1031 mentioned this pull request Mar 20, 2024

enhance: Speed up target recovery after query coord restart (#31240) #31449

Merged

congqixia mentioned this pull request Mar 26, 2024

enhance: Save collection targets by batches #31616

Merged

congqixia mentioned this pull request Mar 27, 2024

enhance: [2.4] Save collection targets by batches (#31616) #31632

Merged

congqixia mentioned this pull request Mar 27, 2024

enhance: [Cherry-pick] Save collection targets by batches (#31616) #31655

Merged

chyezh mentioned this pull request Apr 18, 2024

[Bug]: [benchmark][standalone][LRU] milvus restart and search raises errorcollection not loaded in serial search scene #32398

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: Speed up target recovery after query coord restart #31240

enhance: Speed up target recovery after query coord restart #31240

weiliu1031 commented Mar 13, 2024

mergify bot commented Mar 13, 2024

jaime0815 Mar 13, 2024 •

edited

Loading

weiliu1031 Mar 14, 2024

codecov bot commented Mar 14, 2024 •

edited

Loading

jaime0815 commented Mar 14, 2024

jaime0815 commented Mar 15, 2024

congqixia commented Mar 15, 2024

sre-ci-robot commented Mar 15, 2024

enhance: Speed up target recovery after query coord restart #31240

enhance: Speed up target recovery after query coord restart #31240

Conversation

weiliu1031 commented Mar 13, 2024

mergify bot commented Mar 13, 2024

jaime0815 Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

weiliu1031 Mar 14, 2024

Choose a reason for hiding this comment

codecov bot commented Mar 14, 2024 • edited Loading

Codecov Report

jaime0815 commented Mar 14, 2024

jaime0815 commented Mar 15, 2024

congqixia commented Mar 15, 2024

sre-ci-robot commented Mar 15, 2024

jaime0815 Mar 13, 2024 •

edited

Loading

codecov bot commented Mar 14, 2024 •

edited

Loading