Reuse the same upscaleState across StatefulSets #2339
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We had a bug where the second StatefulSet containing master nodes would have its replicas set to 0, because it is not allowed to add additional masters. Turns out that's because we passed a new upscaleState (by value), whose innitialization to allow masters creation only happens once. The first StatefulSet is allowed master nodes creation thanks to the upscale initialization, but the second StatefulSet isn't because its upscaleState is zeroed-out and not re-initialized. This commit fixes it by passing around a StatefulSet pointer, and adds a unit test that fails without the fix.
nkvoll
reviewed
Jan 3, 2020
barkbay
pushed a commit
to barkbay/cloud-on-k8s
that referenced
this pull request
Jan 3, 2020
* Reuse the same upscaleState across StatefulSets We had a bug where the second StatefulSet containing master nodes would have its replicas set to 0, because it is not allowed to add additional masters. Turns out that's because we passed a new upscaleState (by value), whose innitialization to allow masters creation only happens once. The first StatefulSet is allowed master nodes creation thanks to the upscale initialization, but the second StatefulSet isn't because its upscaleState is zeroed-out and not re-initialized. This commit fixes it by passing around a StatefulSet pointer, and adds a unit test that fails without the fix. * Add a comment to adjustStatefulSetReplicas * Unit test upscaleState mutation in adjustStatefulSetReplicas
mjmbischoff
pushed a commit
to mjmbischoff/cloud-on-k8s
that referenced
this pull request
Jan 13, 2020
* Reuse the same upscaleState across StatefulSets We had a bug where the second StatefulSet containing master nodes would have its replicas set to 0, because it is not allowed to add additional masters. Turns out that's because we passed a new upscaleState (by value), whose innitialization to allow masters creation only happens once. The first StatefulSet is allowed master nodes creation thanks to the upscale initialization, but the second StatefulSet isn't because its upscaleState is zeroed-out and not re-initialized. This commit fixes it by passing around a StatefulSet pointer, and adds a unit test that fails without the fix. * Add a comment to adjustStatefulSetReplicas * Unit test upscaleState mutation in adjustStatefulSetReplicas
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We had a bug where the second StatefulSet containing master nodes would
have its replicas set to 0, because it is not allowed to add additional
masters.
Turns out that's because we pass the upscaleState by value, so any mutation
we do on it is not preserved at the next call for the next StatefulSet. Moreover
it contains a sync.Once initialization not reused the second time. The first
StatefulSet is allowed master nodes creation thanks to the upscale state
initialization, but the second StatefulSet isn't because its
upscaleState is zeroed-out and not re-initialized.
This commit fixes it by passing around a StatefulSet pointer, and adds a
unit test that fails without the fix.
Fixes #2338.