test harness: env-overridable container ports + deterministic host load

[WilfordGrimley]

Test infrastructure only. No production module is touched; no envelope threshold is changed.

Several agents share this box and keep colliding in the test harness. Two distinct causes, both reproduced live tonight.

Problem 1 — fixed container host ports

conftest.POSTGRES_PORT/ELASTICSEARCH_PORT were hardcoded to 47000/9300. A second concurrent suite dies at container start with Bind for 0.0.0.0:9300 failed: port is already allocated, which surfaces as ~2500 collection ERRORs — reading as catastrophic breakage rather than a port clash.

Both are now environment-overridable, defaulting to the historical values so CI and a single local run are byte-identical:

POSTGRES_PORT = int(os.environ.get("TEST_POSTGRES_PORT", 47000))
ELASTICSEARCH_PORT = int(os.environ.get("TEST_ELASTICSEARCH_PORT", 9300))

TEST_POSTGRES_PORT=47010 TEST_ELASTICSEARCH_PORT=9310 pytest cardpicker/tests/

The Elasticsearch half works too. The old comment warned that 9300 "is the default expected by elasticsearch_nooproc". pytest_elasticsearch.config.get_config reads config.getoption("elasticsearch_port") or config.getini("elasticsearch_port") and falls back to a hardcoded 9300, so a new conftest.pytest_configure threads our value into the plugin's own config. Nothing in this suite currently resolves elasticsearch_nooproc (the local elasticsearch fixture shadows the plugin's and never requests the noproc process), so this is belt-and-braces — but without it, TEST_ELASTICSEARCH_PORT would leave a latent 9300 behind for the first test that eventually does.

pytest-xdist worker-id offsets were considered and skipped: xdist is not installed here, and a second mechanism silently competing with the env override would be worse than either alone.

Problem 2 — tests sampled the real host load average

stage_e_dispatch._sample_envelope_signals samples the real os.getloadavg()[0] before every dispatch decision — correct in production, but it made every Stage E dispatch test a function of whatever else was running on the machine. Above HOST_LOAD_CEILING (7.0) they return halted-new-trip and fail; below it they pass. These are the worst kind of flake: plausible, specific, and entirely about the neighbouring process.

New autouse deterministic_host_load fixture in conftest.py pins os.getloadavg() to 0.5 for every test, with a @pytest.mark.real_host_load opt-out (registered in pytest.ini; nothing uses it yet).

The seam is os.getloadavg itself, not _sample_envelope_signals, deliberately. Stubbing the whole sampler would also flatten _window.failures_and_total() and get_process_rss_mb(), which several dispatch tests legitimately drive (the fetch-failure-rate bar in particular). Only the ambient host signal is pinned; every other envelope input still comes from the real code path.

HOST_LOAD_CEILING, RSS_MB_PER_WORKER_CEILING and every other ratified threshold are untouched, as is production sampling. No production code has a new seam.

Which tests were affected

Found by pinning the load high (TEST_HOST_LOAD_AVG=8.67, the value observed in the wild) rather than waiting for a busy box — 37 tests fail halted-new-trip, more than the 18 originally reported:

file	count
test_stage_e_dispatch.py	25
test_stage_e_shakedown.py	6
test_stream_full_catalog.py	6

All 37 pass with the fixture's default. No test was found that deliberately asserts real-load sampling — the tests that pin envelope inputs (test_operating_envelope.py, test_resolve_envelope_trip.py, and the existing monkeypatch.setattr(..., "_sample_envelope_signals", ...) sites) construct EnvelopeSignals directly and never reach os.getloadavg. test_process_metrics.py samples real RSS, not load, and is unaffected.

New tests

cardpicker/tests/test_harness_isolation.py (9 tests) exists so neither fix can rot into a silent no-op — a harness fix that has stopped taking effect is worse than no fix, because the suite stays green and the flakes come back looking like product bugs. Notably it asserts the pinned value is what the production sampler returns (_sample_envelope_signals().load_avg), not merely that the fixture ran; that the Django DB settings and the ES DSL actually follow the overridden ports; and that the real_host_load opt-out genuinely restores real sampling (compared against /proc/loadavg, so a stub that merely forwards is still distinguishable).

Verification

run	result
origin/master, default ports, real host load ~2.0	2503 passed, 7 skipped, 0 failed
this branch, ports 47010/9310	2512 passed, 7 skipped, 0 failed (+9 new harness tests)
this branch, TEST_HOST_LOAD_AVG=8.67	38 failed (37 load-sensitive + the guard test asserting the shipped default; that one now skips when the variable is set)
two full suites concurrently, 47010/9310 and 47020/9320	both 2512 passed, 7 skipped, 0 failed — zero port collisions

Two independent live confirmations of problem 1 during this work:

• An unmodified origin/master run started while another agent held 9300 produced 29 errors, every one Bind for 0.0.0.0:9300 failed: port is already allocated.
• A later run of this branch deliberately left on the default ports was killed the same way mid-session when another agent grabbed 9300 — while its sibling on 9320 completed clean. Same commit, same host, same minute; only the port differed.

All pre-commit hooks pass (ruff, isort, black, mypy, prettier).

🤖 Generated with Claude Code

https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN