x/pkgsite: transient 500 errors on frontend during DB replica maintenance

[hyangah]

When a DB replica is offline for maintenance, frontend instances bound to that replica fail all requests, resulting in a high request failure rate across the frontend pool.

• Cause:

  • Frontend instances randomize their primary DB host choice at startup.
  • The current fallback mechanism to the secondary DB host is only executed during startup connection check.
  • Once running, the connection pool remains static and cannot route around the dead replica.

• Potential solution to look into:
  The driver supports multi-host DSN. Configure connection strings with all the replica hosts.
  Configure the connect timeout in the DSN, so connection attempts to an offline host time out quickly and fail over to the healthy one.

[gabyhelp]

Related Issues

• x/pkgsite: select healthy DB per request #40444

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

CC @golang/pkgsite.

[gopherbot]

Change https://go.dev/cl/791947 mentions this issue: internal/config: enable conditional failover for DB replicas