Summary
The background worker can initialize a duroxide runtime against one extension epoch, then publish readiness and write its epoch sentinel after DROP EXTENSION / CREATE EXTENSION has replaced that epoch. The sentinel is written into the new df schema, so the stale runtime is incorrectly treated as current and is never restarted.
This surfaced as a test-shutdown.sh Scenario B readiness timeout, but the failure occurs before active work is launched or shutdown begins. It is an extension lifecycle ordering race, not a shutdown-latency regression.
Evidence
Observed in PR #332 CI run:
The uploaded PostgreSQL log shows this ordering on Scenario B:
00:03:49.663: PostgreSQL becomes ready.
00:03:49.688: worker detects the existing extension.
00:03:49.696: worker begins runtime initialization against _duroxide.
00:03:49.752: duroxide runtime starts.
- The test's concurrent
DROP EXTENSION / CREATE EXTENSION replaces the provider objects.
00:03:49.938: readiness publication fails because _duroxide._worker_ready no longer exists.
00:03:49.944: the worker writes an epoch sentinel into the newly created df schema.
- The stale runtime continues polling and repeatedly fails because
_duroxide.fetch_work_item, _duroxide.fetch_orchestration_item, and their backing tables no longer exist.
- The readiness probe times out after 60 seconds.
The exact failed commit subsequently passed the test five times locally, confirming that the race is timing-dependent.
Root cause
run_duroxide_runtime() currently writes the epoch sentinel only after runtime initialization and readiness publication. The sentinel therefore identifies whichever extension epoch exists at the end of initialization, not necessarily the epoch against which the runtime was initialized.
If drop/recreate occurs during initialization, the new epoch receives a sentinel for the stale runtime. run_until_extension_dropped_or_shutdown() then sees that sentinel as valid and cannot detect the replacement.
Expected behavior
Readiness must only be published for a runtime initialized against the current extension/provider epoch. If the epoch changes during initialization, the runtime should be torn down and initialization retried against the new epoch.
Suggested direction
- Establish an epoch identity before initializing the duroxide runtime.
- Revalidate that same identity after initialization and before publishing readiness or processing work.
- If validation fails, shut down the just-created runtime and retry initialization.
- Ensure a drop/recreate between readiness publication and entering the processing loop is also detected without certifying a stale runtime.
Acceptance criteria
- Add a deterministic regression test that pauses the worker between epoch capture/runtime initialization and readiness publication, then drops and recreates the extension.
- Verify the stale runtime is torn down and a new runtime initializes successfully.
- Verify
_worker_ready only describes the current provider epoch.
- Verify the worker does not continuously poll removed provider functions or tables.
- Keep shutdown latency behavior unchanged.
Summary
The background worker can initialize a duroxide runtime against one extension epoch, then publish readiness and write its epoch sentinel after
DROP EXTENSION/CREATE EXTENSIONhas replaced that epoch. The sentinel is written into the newdfschema, so the stale runtime is incorrectly treated as current and is never restarted.This surfaced as a
test-shutdown.shScenario B readiness timeout, but the failure occurs before active work is launched or shutdown begins. It is an extension lifecycle ordering race, not a shutdown-latency regression.Evidence
Observed in PR #332 CI run:
The uploaded PostgreSQL log shows this ordering on Scenario B:
00:03:49.663: PostgreSQL becomes ready.00:03:49.688: worker detects the existing extension.00:03:49.696: worker begins runtime initialization against_duroxide.00:03:49.752: duroxide runtime starts.DROP EXTENSION/CREATE EXTENSIONreplaces the provider objects.00:03:49.938: readiness publication fails because_duroxide._worker_readyno longer exists.00:03:49.944: the worker writes an epoch sentinel into the newly createddfschema._duroxide.fetch_work_item,_duroxide.fetch_orchestration_item, and their backing tables no longer exist.The exact failed commit subsequently passed the test five times locally, confirming that the race is timing-dependent.
Root cause
run_duroxide_runtime()currently writes the epoch sentinel only after runtime initialization and readiness publication. The sentinel therefore identifies whichever extension epoch exists at the end of initialization, not necessarily the epoch against which the runtime was initialized.If drop/recreate occurs during initialization, the new epoch receives a sentinel for the stale runtime.
run_until_extension_dropped_or_shutdown()then sees that sentinel as valid and cannot detect the replacement.Expected behavior
Readiness must only be published for a runtime initialized against the current extension/provider epoch. If the epoch changes during initialization, the runtime should be torn down and initialization retried against the new epoch.
Suggested direction
Acceptance criteria
_worker_readyonly describes the current provider epoch.