k8s: Improve ghost nodes removal #9750
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RafalKorepta
force-pushed
the
rk/improve-ghost-nodes-removal
branch
2 times, most recently
from
f6b6f20
to
fbb16a1
Compare
alejandroEsc
previously approved these changes
Mar 31, 2023
| nodeIDs[brokers[i].NodeID] = nil | ||
| } | ||
|
|
||
| pods, err := r.podList(ctx, rp) |
There was a problem hiding this comment.
I see one question here now that i think about it a bit more. Here you are relying on the Cluster resource to give you the podlist? Is this something that is populated into or dynamically retrieved from the state of the world in the namespace? That would lead to a race condition right? Would it be better to just retrieve all pods that have the right annotations? Those that do not exist with these will obviously be left out and you can start removing things not in the same list.
There was a problem hiding this comment.
I changed the implementation as it was inaccurate. Eventual consistency of k8s api sometimes returned old annotation.
RafalKorepta
force-pushed
the
rk/improve-ghost-nodes-removal
branch
from
fbb16a1
to
f745014
Compare
joejulian
approved these changes
Apr 3, 2023
7 tasks
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jun 19, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jun 21, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jun 28, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jul 2, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jul 2, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On top of #9749 add ghost nodes removal function that is run after:
Thanks to all the prerequisites when the
removeGhostNodeIDsfunctionis called the replica specification number should match with the running pods,
so that all PODs have latest Node IDs.
This use case can be triggered in GKE environment where Redpanda is backed by
local SSDs defined by PersistentVolume resource and the VM is recreated. The
recreation can be triggered by deleting VM or by other factors.
Backports Required
Release Notes
Features