Set Pod Management to 'Parallel' and disallow cluster scale down entirely #621
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- use cluster_formation.randomized_startup_delay_range to avoid race condition between nodes during initial cluster formation - this change solves the problem of failing to restart the cluster when all pods are deleted and recreated at once; with podManagementPolicy set to 'OrderedReady', pod 0 is also the first one getting recreated, but it may not be the last node to shut down.
- prevent cluster scale down from happening by checking current number of replicas vs desired number of replicas after running statefulSetBuilder.Update() - return errors, logs, publish events and set ReconcileSuccess to false if scale down request detected
MirahImage
approved these changes
Mar 3, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This closes #298
Note to reviewers: remember to look at the commits in this PR and consider if they can be squashed
Summary Of Changes
set
podManagementPolicytoParallelUse cluster_formation.randomized_startup_delay_range to avoid race condition between nodes during initial cluster formation
This change solves the problem of failing to restart the cluster when all pods are deleted and recreated at once; with podManagementPolicy set to 'OrderedReady', pod 0 is also the first one getting recreated, but it may not be the last node to shut down. This problem was first reported in community slack
Disallow cluster scale down entirely After talking to @mkuratczyk @MirahImage @yaronp68, we agreed that the cluster operator should prevent people from scaling down because it's not a properly supported and tested operation. Once
Reconcile()has detect a scale down request, it will error, publish events, and set ReconcileSuccess to false.PodManagementPolicy is immutable For existing clusters, operators won't be able to update the policy successfully. Users would need to manually delete the statefulset with
cascading=falsefirst, and then the operator can recreate the statefulSet with the correct settings. This needs to be mentioned in release notesAdditional Context
It would have been a cleaner and more elegant solution if we can mark CRD requirements on
spec.replicaspeople from updating it to a less number. That would involve a webhook which adds more component for us to maintain. The current solution is easier to achieve and considering we want to support scale down in the future, it's an OK temporary fix.controller-gen was updated in a previous PR, but the crd tag was not updated. I've included the change in this PR since it's a one liner.