Decommission ghosted brokers using the Cluster controller #13132
Assignees
Labels
Comments
7 tasks
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this issue
Jun 19, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this issue
Jun 21, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this issue
Jun 28, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this issue
Jul 2, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this issue
Jul 2, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Who is this for and what problem do they have today?
When a broker's storage is deleted, it comes back with a new broker id. There's no indication as to why the data is gone or whether it can be recovered. Since the Cluster resource is deprecated and the cloud team would just like to have a method to delete it regardless, the consensus is that we should assume it's not recoverable and decommission the old broker id immediately.
What are the success criteria?
Why is solving this problem impactful?
After months of running on GKE, a cluster may have enough ghosted brokers that the cluster will fail to reach quorum. Cloud has tests that cause this behavior artificially and they would like that test to pass.
Additional notes
This is dangerous and shouldn't be used. There are circumstances in which brokers with valid data can be decommissioned and even the potential for all the brokers to be decommissioned.
The text was updated successfully, but these errors were encountered: