Graceful shutdown: SIGTERM, in-flight request draining, /ready health semantics, clean pool close

[coroluca]

Motivation

smoothdb currently doesn't survive a standard rolling deploy (or a plain restart) without dropping work:

• A raw SIGTERM — the default stop signal of Docker, Kubernetes and systemd — terminated the process immediately (only SIGINT was handled), cutting in-flight requests mid-response and resetting Postgres connections.
• Even on SIGINT, the shutdown context was hardcoded to 1 second, while ReadTimeout/WriteTimeout allow requests to run for 60 — so every restart killed any request slower than ~1s.
• /live and /ready were the same unconditional 200 handler, so a load balancer had no way to deregister an instance before its listener closed.
• Database pools were never closed on shutdown (s.DBE.Close() was commented out as "now blocks"), so Postgres saw connection resets on every stop.
• A clean stop exited with status 1, which supervisors read as a crash.

PostgREST recently addressed the same gaps — shutdown waiting for in-flight requests (PostgREST/postgrest#4689, fixed in #4702) and a pre-shutdown wait for load-balancer deregistration (#4580, server-shutdown-wait-period). This PR follows the same model, and where they differ, kube-apiserver's --shutdown-delay-duration / traefik's requestAcceptGraceTimeout shape.

What changes

One commit per fix/feature, in dependency order:

1. fix: shut down gracefully on SIGTERM, not only SIGINT — registers SIGTERM alongside SIGINT, synchronously in Run() before the listener starts (so an early signal can't hit the default disposition).
2. fix: exit 0 after a graceful shutdown — http.ErrServerClosed is a clean stop, not an error.
3. feat: wait for in-flight requests on shutdown, with optional bound — shutdown now waits for in-flight requests indefinitely by default (they're already bounded per-request by ReadTimeout/WriteTimeout; the supervisor's stop grace period is the final backstop — same model as PostgREST). A new GracefulShutdownTimeout config key (seconds, 0 = wait until done) bounds the wait. Because the wait is no longer trivially short: a second signal forces an immediate exit, Shutdown() is idempotent (sync.Once), and an expired graceful window is logged.
4. feat: report draining on /ready during shutdown, with optional drain window — /ready flips to 503 {"status":"draining"} as soon as shutdown begins while /live keeps answering 200 (the standard probe contract). A new DrainDelay config key (seconds, 0 = disabled) keeps the listener serving that long after readiness flips, so load balancers can observe the 503 and deregister before connections start being refused — the k8s preStop-sleep pattern internalized. The delay applies to SIGTERM only (orchestrated stops); an interactive Ctrl-C skips it, and a second signal cuts it short. Same semantics as PostgREST's server-shutdown-wait-period.
5. test: honor SMOOTHDB_DATABASE_URL in database package tests — CI already exports it; this was the only suite ignoring it.
6. fix: close database pools and notification listener on shutdown — un-comments the pool close and fixes both reasons it used to block: the 1s window abandoned handlers still holding connections, and NotificationListener.Stop() didn't interrupt a pending WaitForNotification (the goroutine and its dedicated connection survived Stop). The listener now cancels its loop context, closes the connection with a bounded fresh context (Postgres receives a proper Terminate instead of a reset), and Stop waits for the goroutine to exit. Server.Shutdown closes the engine after the HTTP drain, bounded by the shutdown context plus a 5s backstop.
7. test: run the server package in make test — the new tests live in ./server, which make test (and CI) didn't run.

New config keys

Key	Meaning	Default
GracefulShutdownTimeout	Max seconds to wait for in-flight requests on shutdown (0 = wait until done)	0
DrainDelay	Seconds to keep serving after /ready starts reporting 503 on a SIGTERM shutdown (0 = disabled)	0

Both default to inert values: with an empty config, behavior only changes in that shutdown is now actually graceful.

Shutdown sequence (with both keys set)

SIGTERM → /ready starts answering 503 (listener still serving)
        → wait DrainDelay (LB observes the 503 and deregisters; second signal skips the wait)
        → stop accepting, wait for in-flight requests (≤ GracefulShutdownTimeout; second signal forces exit)
        → close notification listener + connection pools (clean disconnects in Postgres)
        → exit 0

Compatibility notes

• api.Helper gained an IsDraining() bool method — a compile-time break for external implementers of that interface (in-repo, *server.Server is the only one).
• The exported api.HealthHandler was replaced by LiveHandler and ReadyHandler(Helper).
• stopHandler now takes the signal channel (unexported, no API impact).

Testing

• 6 new integration tests in server/shutdown_test.go (run with -race): SIGTERM graceful path + pools closed, in-flight request completing across shutdown, /ready 503 + /live 200 during the drain window, SIGINT skipping the drain, a second signal skipping the drain, and a listener Stop-interrupts-WaitForNotification test in database/.
• Full suite passes: SMOOTHDB_DATABASE_URL=... make test (CI-equivalent env; verified on Postgres 16).
• Verified end-to-end on the binary: kill -TERM → drain → "Stopped." → exit 0; second SIGTERM during a 30s drain window → immediate graceful stop.

Out of scope / follow-ups

• /ready verifying the main connection pool (PostgREST checks pool + schema cache state) — natural next step.
• Two pre-existing issues found while testing, will file separately: the ACL parser doesn't know PG17's MAINTAIN privilege (m), so TestGrants fails on Postgres 17; and InitDbEngine writes the package-global dbe, which races when two engines exist in one process.