-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Cut-off client traffic when backups are not healthy #252
Comments
@timuthy Why? Please don't. We have had already an outage when we lost blob store access and needlessly caused cluster outages. Back then we agreed that we should not do that (though it wasn't changed, yet). We should keep running, raise an alert, possibly add another shoot state condition (for backup), but the one thing we should absolutely not do is to shut down the cluster. It's our main job/responsibility. If ETCD is healthy, we keep running. We had a similar discussion with @stoyanr when we discussed the "bad case scenario". Shutting down a healthy control plane is definitely causing an outage while not taking backups or not reaching Gardener for some time is very bad, but not a neck-breaker, if recovered. Please do not terminate healthy ETCDs/control planes. No managed service out there will ever do that. Being available is the single most important SLI/KPI that we have. Besides, treating backup readiness different from pod readiness should make things hopefully also simpler/less tightly coupled and thereby less complex (clustered ETCD is already complex enough). |
There is also a open issue which was basically opened Not to block the client traffic and to decouple the readiness probe of etcd with snapshotter and it also suggested to enhance the multi-node etcd proposal to address this new requirement. |
Feature (What you would like to be added):
Concerning the multi-node etcd proposal (ref) client traffic should be cut-off as soon as a failure in the backup procedure is detected.
Motivation (Why is this needed?):
Approach/Hint to the implement solution (optional):
Consult
BackupReady
condition to check if backups are healthy. In case of unhealthy backups, the client traffic is supposed to be cut-off by changing thespec.selector
of the clientservice
object.Depends on #158, #251
The text was updated successfully, but these errors were encountered: