Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: implement wal managerment on streaming node #34153

Merged

Conversation

chyezh
Copy link
Contributor

@chyezh chyezh commented Jun 25, 2024

issue: #33285

  • add lifetime control for wal.
  • implement distributed-safe wal manager on streaming node.

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign czs007 after the PR has been reviewed.
You can assign the PR to them by writing /assign @czs007 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label Jun 25, 2024
@sre-ci-robot sre-ci-robot added the area/dependency Pull requests that update a dependency file label Jun 25, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Jun 25, 2024
@chyezh chyezh force-pushed the feat_streaming_service_wal_manager branch from 04540a1 to 993b7ca Compare June 26, 2024 08:49
Copy link

codecov bot commented Jun 26, 2024

Codecov Report

Attention: Patch coverage is 87.27273% with 35 lines in your changes missing coverage. Please review.

Project coverage is 84.30%. Comparing base (7611128) to head (fcfdc6b).
Report is 4 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #34153      +/-   ##
==========================================
+ Coverage   80.92%   84.30%   +3.37%     
==========================================
  Files        1113      850     -263     
  Lines      138345   114624   -23721     
==========================================
- Hits       111960    96629   -15331     
+ Misses      22145    13749    -8396     
- Partials     4240     4246       +6     
Files Coverage Δ
...ernal/streamingnode/server/walmanager/wal_state.go 100.00% <100.00%> (ø)
.../streamingnode/server/walmanager/wal_state_pair.go 100.00% <100.00%> (ø)
pkg/streaming/walimpls/impls/rmq/builder.go 35.71% <ø> (ø)
...streamingnode/server/wal/utility/reorder_buffer.go 87.50% <60.00%> (-12.50%) ⬇️
...treamingnode/server/wal/adaptor/scanner_adaptor.go 85.00% <25.00%> (-9.45%) ⬇️
...al/streamingnode/server/walmanager/wal_lifetime.go 89.55% <89.55%> (ø)
...al/streamingnode/server/walmanager/manager_impl.go 87.14% <87.14%> (ø)
internal/util/streamingutil/util/wal_selector.go 60.71% <60.71%> (ø)

... and 309 files with indirect coverage changes

@mergify mergify bot added ci-passed and removed ci-passed labels Jun 26, 2024
@mergify mergify bot added ci-passed and removed ci-passed labels Jun 26, 2024
@chyezh chyezh force-pushed the feat_streaming_service_wal_manager branch from 993b7ca to a629dd4 Compare June 27, 2024 08:32
@mergify mergify bot removed the ci-passed label Jun 27, 2024
Signed-off-by: chyezh <chyezh@outlook.com>
@chyezh chyezh force-pushed the feat_streaming_service_wal_manager branch from a629dd4 to 9c38b5c Compare July 4, 2024 08:00
Signed-off-by: chyezh <chyezh@outlook.com>
@chyezh chyezh force-pushed the feat_streaming_service_wal_manager branch from 9c38b5c to 89e22f3 Compare July 4, 2024 08:08
Signed-off-by: chyezh <chyezh@outlook.com>
@chyezh chyezh force-pushed the feat_streaming_service_wal_manager branch from 83abef5 to fcfdc6b Compare July 4, 2024 11:48
@mergify mergify bot added the ci-passed label Jul 4, 2024

// backgroundTask is the background task for wal manager.
// wal open/close operation is executed in background task with single goroutine.
func (w *walLifetime) backgroundTask() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest using one background task to process life time of all WALs that is goruntines effective

// because new Current state after these operation must be same or greater than expected state.

// term must be increasing or available -> unavailable, close current term wal is always applied.
term := currentState.Term()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following code snippet can move into renew function

@jaime0815
Copy link
Contributor

jaime0815 commented Jul 5, 2024

fix unresolved issues in the next PR.

@jaime0815
Copy link
Contributor

/lgtm

@jaime0815 jaime0815 merged commit ba04981 into milvus-io:master Jul 5, 2024
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/compilation area/dependency Pull requests that update a dependency file ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement lgtm size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants