Basic node decommission progress API #8167
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mmaslankaprv
changed the title
mmaslankaprv
force-pushed
the
node-decommission-monitoring
branch
3 times, most recently
from
e42038e
to
9acee6a
Compare
mmaslankaprv
force-pushed
the
node-decommission-monitoring
branch
from
9acee6a
to
48141be
Compare
bharathv
reviewed
Jan 11, 2023
Comment on lines
+385
to
+388
| if (moving_to) { | ||
| it->second.already_transferred_bytes.emplace_back( | ||
| replica_bytes{ | ||
| .node = node_report.id, .bytes = p.size_bytes}); |
src/v/redpanda/admin_server.cc
Outdated
| status.moving_to = moving_to; | ||
| size_t left_to_move = 0; | ||
| for (auto as : p.already_transferred_bytes) { | ||
| left_to_move += (p.current_partition_size - as.bytes); |
There was a problem hiding this comment.
q: should we pair left_to_move with either already_transferred_bytes or current_total_size or both (even better) because if data is getting appended to the partition at a faster pace than recovery, left_to_move may be going up and that is not intuitive to the operator. Think if they see the other two metrics, they can see that the partition is being written to.
There was a problem hiding this comment.
very good point, i will add this information to the response
mmaslankaprv
force-pushed
the
node-decommission-monitoring
branch
3 times, most recently
from
75dd68b
to
b2e336e
Compare
bharathv
previously approved these changes
Jan 12, 2023
| "type": "long", | ||
| "description": "bytes left to move to new replica" | ||
| }, | ||
| "target_size": { |
There was a problem hiding this comment.
nit: consider using a "total_bytes_to_move" analogous to "bytes_left_to_move".. I think "target" will be a bit vague to the operator. because there are two parties here.
There was a problem hiding this comment.
good point, i will include current partition size and amount of data that were transfer to target node
dotnwat
reviewed
Jan 12, 2023
Comment on lines
+119
to
+124
| def wait_for_removal(self): | ||
| self.last_update = time.time() | ||
| # wait for removal only if progress was reported | ||
| while self._made_progress(): | ||
| try: | ||
| decommission_status = self.admin.get_decommission_status( |
Introduced `cluster::reconfiguration_state` a top level type indicating the reconfiguration state as the ongoing reconfiguration may be result of partition move or its cancellation. Signed-off-by: Michal Maslanka <michal@redpanda.com>
Signed-off-by: Michal Maslanka <michal@redpanda.com>
Added method allowing caller to query node decommissioning status. When node is decommission Redpanda automatically move all the replicas from the decommissioned node and reassign them to another nodes. This process may take a long time as it involves data transfer between nodes. Added API will allow user to observe progress of decommissioning process. Signed-off-by: Michal Maslanka <michal@redpanda.com>
Added `GET /v1/brokers/<id>/decommission` endpoint that returns basic information about node decommissioning progress. Signed-off-by: Michal Maslanka <michal@redpanda.com>
Signed-off-by: Michal Maslanka <michal@redpanda.com>
Added simple helper that waits for node removal and checks progress. It will be used in tests to detect stuck decommissioning operations. Signed-off-by: Michal Maslanka <michal@redpanda.com>
Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv
force-pushed
the
node-decommission-monitoring
branch
from
b2e336e
to
35fe043
Compare
|
ci failure: #7758 |
bharathv
approved these changes
Jan 13, 2023
|
This is great and helpful! Can we backport this to 22.3? |
This was referenced Jan 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added REST endpoint allowing caller to query node decommissioning status.
When node is being decommission Redpanda automatically move all the replicas from
the decommissioned node to other nodes. This process
may take a long time as it involves data transfer between nodes. Added
REST API will allow user to observe progress of decommissioning process.
Fixes: #7874
Backports Required
UX Changes
Release Notes
Improvements
GET /v1/brokers/<id>/decommission