Skip to content

fix(tests): poll for on_timeout middleware in prefork test#154

Merged
pratyush618 merged 1 commit into
masterfrom
fix/prefork-timeout-middleware-race
May 8, 2026
Merged

fix(tests): poll for on_timeout middleware in prefork test#154
pratyush618 merged 1 commit into
masterfrom
fix/prefork-timeout-middleware-race

Conversation

@pratyush618
Copy link
Copy Markdown
Collaborator

Summary

Master CI run 25539963938 failed on tests/worker/test_prefork.py::test_prefork_kills_hung_task (Python 3.10 / Ubuntu) with:

AssertionError: on_timeout middleware did not fire
assert '019e063d-…' in []

The job was killed correctly (status dead, error "timed out", well under the 12s budget), but the on_timeout spy hadn't fired yet.

Race: the worker thread, on receiving the timeout result, runs

let outcome = handle_result(result);   // writes status='dead' to the DB
dispatch_outcome(py, outcome);         // fires on_timeout middleware

_wait_for_terminal polls the DB and returns as soon as it sees dead — which can happen between those two calls. On a fast Ubuntu 3.10 runner the test thread observes dead, races back, and asserts on timeouts_seen before dispatch_outcome reaches the middleware call.

Fix: wait for the side effect we actually care about. The poll_until fixture in tests/conftest.py already exists for this exact pattern (used in test_prefork_cancel_running_job_stops_quickly). Replace the bare assert job.id in timeouts_seen with poll_until(lambda: job.id in timeouts_seen, timeout=5, …).

Five seconds is well above the worst-case window between the DB write and the middleware call (which is microseconds in practice), but it gives slow CI runners enough headroom to never flake again.

The middleware ordering itself isn't a bug — dispatch_outcome deliberately runs after handle_result so the DB is the source of truth for status. The test was simply asserting on a moving target.

Test plan

  • uv run python -m pytest tests/worker/test_prefork.py::test_prefork_kills_hung_task -v passes locally
  • ruff check, mypy clean
  • CI green on master after merge

@github-actions github-actions Bot added the tests label May 8, 2026
handle_result() flips the DB status to 'dead' before dispatch_outcome
fires on_timeout, so a fast test thread can observe 'dead' and assert on
timeouts_seen before the worker reaches the middleware call. Poll for
the spy with a small budget instead.
@pratyush618 pratyush618 force-pushed the fix/prefork-timeout-middleware-race branch from b27c6ae to 4f88e58 Compare May 8, 2026 06:44
@pratyush618 pratyush618 merged commit 5b4d7bd into master May 8, 2026
19 checks passed
@pratyush618 pratyush618 deleted the fix/prefork-timeout-middleware-race branch May 8, 2026 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant