fix(worker): rotate STS secret on a side conn, off the activation lock

[benben]

Summary

• Rotate the DuckDB S3 secret on a side connection that shares the same DuckDB instance as the main client-query DB but has its own Go connection pool — so CREATE OR REPLACE SECRET never queues behind a long-running client query.
• Run the rotation outside p.mu.Lock in reuseExistingActivation , so a slow refresh can't starve concurrent p.mu.RLock acquirers (the gRPC health-check goroutine snapshotting sessions for stall detection).

Why

mw-prod-us 2026-05-04: ~10 worker pods/hour killed with K8s worker unresponsive, deleting pod , every kill ~10 s after a Refreshing S3 credentials... log. Workers were idle on CPU/memory — pure deadlock.

Root cause:

Step	What	Where
1	Credential refresh sends ActivateTenant	controlplane/multitenant.go (now per-CP after #474)
2	Worker enters reuseExistingActivation , takes p.mu.Lock	duckdbservice/activation.go
3	Calls RefreshS3Secret → db.Exec(\"CREATE OR REPLACE SECRET ...\") on the activation DB	server/server.go
4	Activation DB has MaxOpenConns=1 ; an active client query holds the conn → db.Exec queues indefinitely	—
5	gRPC health-check needs p.mu.RLock → starved behind p.mu.Lock	duckdbservice/flight_handler.go:212
6	3 × 3 s health-check timeouts = 9 s of unresponsive → CP force-deletes the pod	controlplane/worker_mgr.go

Fix

Side-connection for control DDL

server/duckdb_pair.go introduces server.OpenDuckDBPair :

• One *duckdb.Connector (one DuckDB Database)
• Two *sql.DB s built via sql.OpenDB , each MaxOpenConns(1)
• Wrapped in nonClosingConnector so *sql.DB.Close doesn't kill the underlying instance

SessionPool now stores a *DuckDBPair ; controlDB is exposed for credential rotation. Both DBs see the same SecretManager and attached catalogs. The running query picks up the new STS token on its next S3 request — strictly better than today's "no rotation possible while a query holds the conn" behavior.

Lock-released refresh

reuseExistingActivation is now three phases:

1. Locked: inspect activation, decide if refresh is needed, capture refresh fn / DB / sem.
2. Unlocked: run RefreshS3Secret on controlDB .
3. Locked: re-validate (activation may have changed) and commit the new payload.

Health-check RLock acquisition no longer waits for the secret to commit.

Test plan

• TestReuseExistingActivationDoesNotBlockHealthChecks — injects a blocking refreshS3Secret stub, asserts a concurrent p.mu.RLock acquirer returns within 100 ms during the slow refresh, and asserts the rotation runs on controlDB (not Main).
• Confirmed the test FAILS on the pre-fix code path (locally simulated by reverting the unlock — RLock blocked, test fired its deadlock-regression message).
• All existing ./duckdbservice/ , ./server/ , ./controlplane/ tests pass with -tags kubernetes .
• Deploy to mw-dev, watch Refreshing S3 credentials... complete cleanly while a long query is in flight.
• Deploy to mw-prod-us, confirm K8s worker unresponsive rate drops to zero on managed-warehouse workers.