-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ceph: Cleanup mon canary pvc if the failover failed #6224
Conversation
When the mons are based on pvcs, the pvc_name label was not being set as expected. Now the label will always be set on both the canary and the daemon pods. The mon pvcs were also never being deleted during canary cleanup. This code is now removed since we must be very careful when the pvcs are removed. Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit
logger.Info("checking for orphaned mon resources") | ||
|
||
if c.spec.Mon.VolumeClaimTemplate == nil { | ||
logger.Info("skipping check for orphaned mon pvcs") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
logger.Info("skipping check for orphaned mon pvcs") | |
logger.Info("skipping check for orphaned mon pvcs") |
Did you mean non-pvc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's skipping the check for mon pvcs, it looks correct to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But if the VolumeClaimTemplate
is nil this means non-pvc right? 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's right. There are no mon pvcs, so we skip checking for them. What about this?
logger.Info("skipping check for orphaned mon pvcs since using the host path")
assert.NoError(t, err) | ||
assert.Equal(t, 4, len(pvcs.Items)) | ||
|
||
// pvc "b" should be removed and the others should remain |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you mean "c" instead?
902d7a8
to
50db4ac
Compare
If the mon failover fails, if the mon is on a pvc, the pvc could remain orphaned. This is likely to happen if there is not a node available for scheduling the mon. Therefore, the operator will ensure after a failover that any orphaned mon pvc is cleaned up. Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
50db4ac
to
7a6620f
Compare
ceph: Cleanup mon canary pvc if the failover failed (bp #6224)
Description of your changes:
If the mon failover fails and the mon is backed by a pvc, the pvc could remain orphaned. This is likely to happen if there is not a node available for scheduling the mon. Therefore, the operator will ensure after a failover that any orphaned mon pvc is cleaned up.
On a related change, the pvc_name label was not being set as expected on canary deployments. Now the label will always be set on both the canary and the daemon pods.
This replaces #6199.
Checklist:
make codegen
) has been run to update object specifications, if necessary.