test: serialize cross-tenant sweep tests + stop defaulting tests at the live DB

[0xmanhnv]

Two test-isolation problems in the only two packages that open a real database.

1. Global sweeps race across packages (latent, not observed)

internal/infra/postgres and tests/integration are the only packages that open a DB, and go test ./... runs their binaries concurrently against the same one. Several things they exercise are deliberately global — RecoverStuckJobs, ExpireOldPlatformJobs, MarkStaleAsOffline, and the recover_stuck_* SQL functions all sweep every tenant, because that is what a reaper does.

platform_job_lifecycle_db_test seeds a non-platform command — acknowledged, 120 minutes old, with a tenant agent — and asserts the platform sweep leaves it alone. recover_stuck_tenant_commands(10, 3), called from tests/integration/command_recovery_test, selects on exactly those columns:

is_platform_job = FALSE
AND status = 'acknowledged'
AND agent_id IS NOT NULL
AND acknowledged_at < NOW() - (p_stuck_threshold_minutes || ' minutes')::INTERVAL
AND dispatch_attempts < p_max_retries

Seeding that row by hand and running the function:

--- BEFORE (the assertion requires acknowledged / 0)
 acknowledged | 0
--- the other package's global sweep runs:
 rows_recovered: 1
--- AFTER
 pending      | 1

That is precisely what the assertion forbids.

What I did not find: any actual failure. 120 concurrent rounds of both packages produced zero failures, because each test seeds, sweeps and asserts inside ~20-80ms. So this is a latent hazard, not CI noise anyone is currently suffering — I would rather say that than sell it as a flake fix. It is still worth closing: the window widens with -race (which CI uses), on a loaded runner, and with every test added to either package.

The fix

A Postgres session-level advisory lock, taken for the whole test rather than just the sweep call — the race is between one package's seed and its assertion, so locking only the call would leave exactly that window open.

defer lockGlobalSweep(ctx, t, db)()

Rejected alternatives: -p 1 would serialize 91 packages to discipline two; weakening the assertions would remove the thing they exist to catch.

The helper is duplicated in both packages rather than extracted to a shared testutil — it is nine lines, and the lock key is the contract between them, which is easier to see when both files state it.

Proof the lock is wired, not just present: holding it from psql and running a locked test →

lock	result
held	exit=124 (timed out)
free	exit=0, 0.08s

2. Three helpers defaulted to the production database name

setupCommandTestDB and the two RLS helpers fell back to postgres://openctem@localhost:5432/openctem when DATABASE_URL was unset. An unconfigured go test ./... therefore wrote rows and ran cross-tenant sweeps against whatever openctem resolves to on that machine — in a dev environment, the real database. They now t.Skip, matching openPlatformJobDB.

Verification

• go test ./... with DATABASE_URL=…/app_test — all green
• go test -race on both DB packages — green
• make lint-ci — clean
• Tests confirmed to run, not skip (-test.v shows === RUN / --- PASS)