You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, our service processes connect to a single DB on startup. If that DB enters a bad state while running, the process continues to run, serving 500s.
Instead, we should pick from a set of healthy DBs on each request.
Alternative to picking a healthy DB connection per request:
On startup, we connect to one of the configured DBs. All requests use the same connection. (This is the way it works now).
We implement a health check that pings the DB.
This will cause the health check watcher (AppEngine or the GKE load balancer) to quickly kill processes that are connected to a bad DB, and start new ones.
That approach will work well assuming a failure mode where one database becomes completely unavailable. If instead all the databases become 5% unavailable, e.g. because of a bug that overloads them all, they won't find a database that works all the time. Depending on the exact characteristics of the failure, how fast the instances get killed, and how fast they restart, that could end up making a minor outage into a very large one.
There's also some risk with pointing each instance to a single database; after an outage on one replica, all the frontend instances will be pointed at the other replica.
Since the failures we've seen so far are (AFAIK) total outages of one replica, this might be fine. But an approach that doesn't crash would be better in all respects, except perhaps for development ease.
Activity
gopherbot commentedon Jul 28, 2020
Change https://golang.org/cl/244603 mentions this issue:
internal/frontend: get a DataSource on each requestinternal/frontend: get a DataSource on each request
jba commentedon Sep 11, 2020
Alternative to picking a healthy DB connection per request:
This will cause the health check watcher (AppEngine or the GKE load balancer) to quickly kill processes that are connected to a bad DB, and start new ones.
heschi commentedon Sep 14, 2020
That approach will work well assuming a failure mode where one database becomes completely unavailable. If instead all the databases become 5% unavailable, e.g. because of a bug that overloads them all, they won't find a database that works all the time. Depending on the exact characteristics of the failure, how fast the instances get killed, and how fast they restart, that could end up making a minor outage into a very large one.
There's also some risk with pointing each instance to a single database; after an outage on one replica, all the frontend instances will be pointed at the other replica.
Since the failures we've seen so far are (AFAIK) total outages of one replica, this might be fine. But an approach that doesn't crash would be better in all respects, except perhaps for development ease.