/health/ returned 200 with {"database": "healthy"} in 0.44s at the same moment Postgres was refusing new connections with FATAL: sorry, too many clients already, the login endpoint was failing, and the deploy pipeline could not run manage.py.
Why it could not fail
The check ran SELECT 1 on the request's own connection. That connection is already established, so it keeps answering however saturated the server is. The endpoint was structurally incapable of failing for the condition that was taking the service down.
A green health check was therefore worse than no health check: ci-deploy.sh gates on /health/ with curl -f, and the view returns 503 when a required service is unhealthy — so the deploy gate, the rollback trigger and anything paging on this endpoint all stayed green throughout an outage.
Fixed
Also opens a fresh connection and closes it immediately. The two failure modes are independent, and it is the second that catches exhaustion.
With the check now able to detect it, the existing deploy gate catches this fault class and rolls back automatically — the chain was already correct, just blind.
PR: DataSpaceBackend#135
/health/returned200with{"database": "healthy"}in 0.44s at the same moment Postgres was refusing new connections withFATAL: sorry, too many clients already, the login endpoint was failing, and the deploy pipeline could not runmanage.py.Why it could not fail
The check ran
SELECT 1on the request's own connection. That connection is already established, so it keeps answering however saturated the server is. The endpoint was structurally incapable of failing for the condition that was taking the service down.A green health check was therefore worse than no health check:
ci-deploy.shgates on/health/withcurl -f, and the view returns 503 when a required service is unhealthy — so the deploy gate, the rollback trigger and anything paging on this endpoint all stayed green throughout an outage.Fixed
Also opens a fresh connection and closes it immediately. The two failure modes are independent, and it is the second that catches exhaustion.
With the check now able to detect it, the existing deploy gate catches this fault class and rolls back automatically — the chain was already correct, just blind.
PR: DataSpaceBackend#135