Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: Make applyDelete work in paralell in segment level #32291

Merged
merged 3 commits into from
Apr 24, 2024
Merged

enhance: Make applyDelete work in paralell in segment level #32291

merged 3 commits into from
Apr 24, 2024

Conversation

congqixia
Copy link
Contributor

applyDelete used to be serial for delete entries on each segments. This PR make it work in parallel with errgroup to improve performance

@sre-ci-robot sre-ci-robot added the size/M Denotes a PR that changes 30-99 lines. label Apr 15, 2024
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Apr 15, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 15, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@codecov Codecov
Copy link

codecov bot commented Apr 15, 2024 •
edited

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.79%. Comparing base (dcc15e3) to head (92c14b8).
Report is 2 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #32291      +/-   ##
==========================================
- Coverage   81.83%   81.79%   -0.04%     
==========================================
  Files         999      999              
  Lines      124070   124081      +11     
==========================================
- Hits       101529   101491      -38     
- Misses      18666    18720      +54     
+ Partials     3875     3870       -5
Files Coverage Δ
internal/querynodev2/delegator/delegator_data.go 85.90% <100.00%> (+1.42%) ⬆️

... and 35 files with indirect coverage changes

@congqixia
Copy link
Contributor Author

/run-cpu-e2e

@mergify mergify bot added the ci-passed label Apr 16, 2024
@congqixia
Copy link
Contributor Author

/hold

@mergify mergify bot removed the ci-passed label Apr 23, 2024
@congqixia
Copy link
Contributor Author

/unhold

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 23, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 23, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia congqixia force-pushed the enhance/delete_segment_parallel branch from 51c30d4 to 8df2c91 Compare April 23, 2024 14:05
@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 23, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia
enhance: Make applyDelete work in paralell in segment level
bd5a042 
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia
Use pool instead of unlimited errgroups
eaca57f 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia force-pushed the enhance/delete_segment_parallel branch from 8df2c91 to eaca57f Compare April 24, 2024 02:16
@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 24, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia
Copy link
Contributor Author

/run-cpu-e2e

1 similar comment
@congqixia
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 24, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

1 similar comment
@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 24, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia
Copy link
Contributor Author

/run-cpu-e2e

@congqixia
Fix closure capture iteration var
92c14b8 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@mergify mergify bot added the ci-passed label Apr 24, 2024
@weiliu1031
Copy link
Contributor

/lgtm

@sre-ci-robot sre-ci-robot merged commit faa5595 into milvus-io:master Apr 24, 2024
15 checks passed
congqixia added a commit to congqixia/milvus that referenced this pull request Apr 25, 2024
@congqixia
enhance: Make applyDelete work in paralell in segment level (milvus-i…
5a45920 
…o#32291)

`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this pull request Apr 25, 2024
Merged
sre-ci-robot pushed a commit that referenced this pull request Apr 26, 2024
@congqixia
enhance: [2.4] Make applyDelete work in paralell in segment level (#3…
193f88b 
…2291) (#32604)

Cherry-pick from master
pr: #32291
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aoiasd aoiasd Awaiting requested review from aoiasd

@yah01 yah01 Awaiting requested review from yah01

Assignees

@weiliu1031 weiliu1031

Labels
approved ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement lgtm size/M Denotes a PR that changes 30-99 lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@congqixia @sre-ci-robot @weiliu1031