Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix datanode panic due to concurrent compaction and delete processing #27158

Closed
wants to merge 1 commit into from
Closed

Fix datanode panic due to concurrent compaction and delete processing #27158

wants to merge 1 commit into from

Conversation

xiaofan-luan
Copy link
Contributor

fix #27145

the datanode processing delete while compaction happened so flush can not succeed.
For fixing:

  1. bring back blockall, ,make sure compaction sync segments and delete processing are mutexed.
  2. change ttNode handle updateCheckPoint async, make sure all the flowgraph node is processed fast.
  3. change some error handling

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Sep 17, 2023
@mergify mergify bot added the dco-passed DCO check passed. label Sep 17, 2023
@mergify
Copy link
Contributor

mergify bot commented Sep 17, 2023

@xiaofan-luan ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Sep 17, 2023

@xiaofan-luan E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaofan-luan
Fix datanode panic due to concurrent compaction and delete processing
867dfcb 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
@mergify
Copy link
Contributor

mergify bot commented Sep 17, 2023

@xiaofan-luan ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Sep 17, 2023

@xiaofan-luan E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@yanliang567
Copy link
Contributor

/run-cpu-e2e

@mergify
Copy link
Contributor

mergify bot commented Sep 18, 2023

@xiaofan-luan E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@bigsheeper bigsheeper mentioned this pull request Sep 18, 2023
Merged
@bigsheeper bigsheeper mentioned this pull request Sep 18, 2023
Merged
bigsheeper added a commit to bigsheeper/milvus that referenced this pull request Sep 18, 2023
@xiaofan-luan @bigsheeper
Fix datanode panic due to concurrent compaction and delete processing m…
60fccfc 
…ilvus-io#27158

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
sre-ci-robot pushed a commit that referenced this pull request Sep 18, 2023
@bigsheeper @xiaofan-luan
Fix datanode panic due to concurrent compaction and delete processing #…
dde2771 
…27158 (#27162)

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
@czs007 czs007 closed this Sep 18, 2023
@bigsheeper bigsheeper mentioned this pull request Nov 2, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@godchen0212 godchen0212 Awaiting requested review from godchen0212

@sunby sunby Awaiting requested review from sunby

Assignees
No one assigned
Labels
approved dco-passed DCO check passed. size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: Standalone restarted once during test
4 participants
@xiaofan-luan @sre-ci-robot @yanliang567 @czs007