fix(pool): reap orphaned assigned/failed pool_items (sweep #8)

[mastermanas805]

What

Adds a reaper to the hot-pool Manager (internal/pool/manager.go) so leaked pool_items no longer strand backing infra. Closes sweep-backlog finding #8 (P2).

On each maintenance tick reapStale now:

1. Reaps failed items older than failedReapGrace (10m), bounded to reapBatchLimit (50) per pass: routes Deprovision through the resource-type backend using the row's pool_token as the naming token, then deletes the row. A Deprovision failure leaves the row for the next tick — the tracking row is never deleted before its infra is freed. Deprovision is idempotent (DROP ... IF EXISTS), so reaping an item whose infra is already gone is a safe no-op. A failed row (set only by Discard, before the item is ever returned to api) has by construction no owning resources row, so this is a pure leak with no live owner.

2. Reports assigned items older than stuckAssignedGrace (30m) on the new instant_pool_stuck_assigned gauge — but does not deprovision them.

Why assigned is reported, not reaped (scope decision)

The finding asked to reap orphaned assigned items (crashed-claim). I verified this cannot be done safely from the provisioner:

• The pool item lifecycle has only two status writers: Claim (→assigned) and Discard (→failed). There is no write-back when api successfully binds a claimed item to a resources row.
• So from the provisioner's own DB, an orphaned assigned row is indistinguishable from one a live api request bound to a resources row. A bound item's infra is owned by that resource row and reaped by the worker's resource-TTL path. Deprovisioning by age here would destroy live customer infra — the truehomie-db DROP incident class.
• pool_items lives in the provisioner's own standalone Postgres (PROVISIONER_DATABASE_URL); resources lives in platform_db. No current service has both, so a safe orphan-assigned anti-join reaper has no correct home today. The failed-drain half is squarely a provisioner job; the assigned half is surfaced as an operator signal + documented follow-up rather than forced unsafely.

Metrics / Rule 25 follow-up

New metrics: instant_pool_reap_total{resource_type,status,outcome}, instant_pool_stuck_assigned{resource_type}.

The alert + dashboard tile + METRICS-CATALOG.md row mandated by rule 25 live in the infra repo, which is out of scope for this provisioner-only PR. Follow-up required in infra: Prom rule + NR alert (P2 observability: rising instant_pool_stuck_assigned = claim-path leak; non-zero instant_pool_reap_total{outcome="deprovision_err"} rate = wedged reaper), dashboard tile, catalog rows.

Coverage

Symptom:        pool_items stuck 'failed' (leaked infra) / 'assigned' (orphaned by crashed claim)
Enumeration:    rg -n "UPDATE pool_items|status =" internal/pool/manager.go  → 2 status writers (Claim, Discard); no reaper, no bound write-back
Sites found:    1 (the maintenance loop ticker arm in run())
Sites touched:  1 (reapStale wired into the ticker arm)
Coverage test:  TestReapFailed_ReapsOrphanedPastGrace (orphaned past grace IS reaped + correct pool_token);
                TestReapFailed_DeprovisionErrorLeavesRow (no orphaned infra on Deprovision failure);
                TestReapStale_NeverDeprovisionsAssigned (truehomie guard — assigned never deprovisioned);
                TestReapFailed_BatchBounded; gauge reset-to-zero; fakeDB/fakeRows seam for Query/Scan/Rows.Err/DELETE error arms
Live verified:  N/A pre-merge (provisioner is in-cluster gRPC, no public URL). reaper functions 100% covered,
                package 97.6%, full `go build ./... && go vet ./... && go test ./... -short -p 1` GREEN + -race clean
                against a local Postgres (DB-gated tests ran, not skipped). CI runs the same with TEST_PROVISIONER_DATABASE_URL set.

🤖 Generated with Claude Code