Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove datanode timetick mq, use rpc to report instead #23156

Merged
merged 1 commit into from
Jun 14, 2023
Merged

Remove datanode timetick mq, use rpc to report instead #23156

merged 1 commit into from
Jun 14, 2023

Conversation

wayblink
Copy link
Contributor

@wayblink wayblink commented Mar 31, 2023 •
edited
Loading

Remove timetick mq logic in datacoord and datanode, use RPC instead.

datacoord:
1, Remove DataNodeTtLoop and related msgstream consumer
2, Add ReportDataNodeTtMsgs interface, ReportDataNodeTtMsgs call handleTimetickMessage

datanode:
1, Remove related msgstream producer in flow graph
2, Introduce a timetick_sender in datanode. All insert_buffer_nodes on datanode updates channel states to timetick_sender.
3, timetick_sender call datacoord.ReportDataNodeTtMsgs to send DataNodeTTMsg periodically, perform like heartbeat

related: #22953

@sre-ci-robot sre-ci-robot added do-not-merge/work-in-progress Don't merge even CI passed. area/dependency Pull requests that update a dependency file area/internal-api size/XXL Denotes a PR that changes 1000+ lines. labels Mar 31, 2023
@mergify mergify bot added the dco-passed DCO check passed. label Mar 31, 2023
@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@wayblink
Copy link
Contributor Author

/run-cpu-e2e

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@wayblink wayblink force-pushed the remove-tts-master2 branch 2 times, most recently from e2c0ad3 to 87e3b58 Compare March 31, 2023 10:31
@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

@codecov
Copy link

codecov bot commented Mar 31, 2023 •
edited
Loading

Codecov Report

Merging #23156 (31cf5c9) into master (4387f36) will decrease coverage by 0.33%.
The diff coverage is 82.70%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #23156      +/-   ##
==========================================
- Coverage   82.44%   82.11%   -0.33%     
==========================================
  Files         779      780       +1     
  Lines      102453   102681     +228     
==========================================
- Hits        84466    84320     -146     
- Misses      14972    15316     +344     
- Partials     3015     3045      +30
Impacted Files Coverage Δ
internal/util/mock/grpc_datacoord_client.go 89.01% <0.00%> (-2.01%) ⬇️
internal/datacoord/server.go 62.71% <20.00%> (-11.15%) ⬇️
internal/distributed/datacoord/client/client.go 92.61% <40.00%> (-2.71%) ⬇️
internal/datacoord/services.go 85.15% <67.92%> (-0.68%) ⬇️
internal/datanode/flow_graph_insert_buffer_node.go 78.81% <80.00%> (-15.13%) ⬇️
internal/datanode/timetick_sender.go 95.65% <95.65%> (ø)
internal/datanode/data_node.go 73.74% <100.00%> (+0.01%) ⬆️
internal/datanode/data_sync_service.go 78.78% <100.00%> (-1.90%) ⬇️
internal/datanode/flow_graph_manager.go 95.17% <100.00%> (-4.83%) ⬇️
internal/distributed/datacoord/service.go 92.30% <100.00%> (+0.07%) ⬆️
... and 1 more

... and 23 files with indirect coverage changes

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@wayblink
Copy link
Contributor Author

/run-cpu-e2e

@wayblink
Copy link
Contributor Author

/assign @XuanYang-cn

@mergify
Copy link
Contributor

mergify bot commented Mar 31, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify mergify bot removed the ci-passed label Jun 11, 2023
@mergify
Copy link
Contributor

mergify bot commented Jun 11, 2023

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Jun 12, 2023

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Jun 12, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@wayblink
Copy link
Contributor Author

/run-cpu-e2e

@mergify mergify bot added the ci-passed label Jun 12, 2023
@wayblink
Remove datanode timetick mq, use rpc to report instead
31cf5c9 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
@mergify mergify bot removed the ci-passed label Jun 13, 2023
@mergify
Copy link
Contributor

mergify bot commented Jun 13, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@wayblink
Copy link
Contributor Author

/run-cpu-e2e

@wayblink
Copy link
Contributor Author

/run-gpu-e2e

@mergify
Copy link
Contributor

mergify bot commented Jun 13, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@wayblink
Copy link
Contributor Author

/run-cpu-e2e

@mergify
Copy link
Contributor

mergify bot commented Jun 13, 2023

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@wayblink
Copy link
Contributor Author

/run-cpu-e2e

@wayblink
Copy link
Contributor Author

/run-gpu-e2e

@mergify mergify bot added the ci-passed label Jun 13, 2023
@wayblink
Copy link
Contributor Author

/run-gpu-e2e

@xiaofan-luan
Copy link
Collaborator

/lgtm
/approve

@xiaofan-luan
Copy link
Collaborator

/lgtm
/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wayblink, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot merged commit bfae6b4 into milvus-io:master Jun 14, 2023
@wayblink wayblink mentioned this pull request Jul 7, 2023
Merged
zhuwenxing pushed a commit to mmga-lab/milvus that referenced this pull request Jul 25, 2023
@wayblink @zhuwenxing
Remove datanode timetick mq, use rpc to report instead (milvus-io#23156)
44dfe71 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaofan-luan xiaofan-luan xiaofan-luan left review comments

@XuanYang-cn XuanYang-cn XuanYang-cn left review comments

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@congqixia congqixia Awaiting requested review from congqixia

Assignees

@XuanYang-cn XuanYang-cn

@xiaofan-luan xiaofan-luan

Labels
approved area/dependency Pull requests that update a dependency file area/internal-api ci-passed dco-passed DCO check passed. lgtm size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wayblink @xiaofan-luan @XuanYang-cn @sre-ci-robot