New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ScyllaDB version upgrade being stuck on Pod staying in maintenance mode #1267
Conversation
9193ace
to
73b44b1
Compare
73b44b1
to
0d2c8e8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well done tracking the issue. As for the fix - it's imperative and prone to fail with disruptions, e.g. deleting the service during the upgrade would result in unexpected behaviour. Having said that I don't see a different approach to fixing this without changing the upgrade procedure, but that's something to keep in mind.
oldService: &corev1.Service{ | ||
ObjectMeta: metav1.ObjectMeta{ | ||
Labels: map[string]string{ | ||
naming.NodeMaintenanceLabel: "42", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: s/"42"/""
Looks unnecessarily obfuscated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
value doesn't matter, ""
is as good as "42"
. Leaving as is to save CI respins.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/approve
/lgtm
0d2c8e8
to
d201b7b
Compare
d201b7b
to
1c05996
Compare
…e mode Service controller copied maintenance mode label into required object from cached informer when it was present. Labels are also rewritten from existing objects in all Apply methods. If maitenance mode label was removed in between when required object is constructed, and Apply method is called, it would make it persitently available in the Service object. Instead adding it to required object, it should be carried over from existing object only in Apply method, because this label is managed by patches. Fixes scylladb#1252
1c05996
to
78f142f
Compare
Service controller copied maintenance mode label into required object from cached informer when it was present.
Labels are also rewritten from existing objects in all Apply methods. If maitenance mode label was removed in between when required object is constructed, and Apply method is called, it would make it persitently available in the Service object.
Instead adding it to required object, it should be carried over from existing object only in Apply method, because this label is managed by patches.
Fixes #1252