fix: keep processing jobs when the notification listener fails - #677
Merged
MoonBoi9001 merged 22 commits intoJul 20, 2026
Conversation
A worker loop polls the queue every second and also holds a database connection that wakes it sooner when a job arrives. An error on that connection used to stop every loop and halt all job processing, so the loop now drops it, keeps polling, and reconnects on a later tick.
A worker keeps a database notification connection to wake sooner than its 1-second job poll. A failed wait returns instantly, so re-subscribing on that same tick to a connection that fails just as fast spun the loop at full CPU. Re-subscribing now waits for the next poll-paced tick.
A worker loop gave up entirely if it could not open its database notification connection at startup, and because the loops are supervised together that stopped every one of them. It now starts in poll-only mode and picks the notifications up once the database is reachable.
The rule that stops a broken notification connection from being retried in a tight loop lived inline in the worker loop, where no test could reach it. It now sits in a small function the wait step feeds, so the pacing is covered by unit tests.
A worker that loses its database notification connection falls back to checking for jobs on a timer, but only said so once, then reported each retry at debug. It now repeats a warning about every 60 failed attempts, so an outage that never recovers stays visible in normal logs.
Waiting on a job notification alongside a stop signal and a timer drops the losing wait, which looks like it could discard a notification. The database driver's read step is built to survive exactly that, so the note explains why, to save the next reader the same investigation.
The test that checks a worker still wakes on its timer when notifications are unavailable slept for real, making it slow and dependent on machine timing. It now advances a simulated clock, so it proves the same thing instantly and repeatably.
The code deciding when to repeat a warning tested a remainder by hand, which the project's lint settings reject in favour of the standard library call that says the same thing.
The note saying no job notification is ever lost claimed more than holds. It is true while the connection is up, but one can still go missing if the connection drops mid-reconnect, which the 1 second fallback turns into a short delay.
The rule spacing out attempts to reopen the job notification connection named a case it has no way to observe: a dropped connection is retried inside the database driver, never surfacing as an error, so the pacing never applies to it.
The interval between warnings about an unavailable job notification connection was described as about a minute, which only holds when attempts fail quickly. A database that accepts but never answers stretches each attempt to 30 seconds, and the gap to half an hour.
The database driver quietly rebuilds a dropped notification connection and waits again, so a proxy killing those connections could churn them indefinitely with nobody the wiser. Repeated drops in quick succession are now reported to the caller, which can pace and log the retry.
Counting failures misjudged how often to repeat a warning: a connection recovering between each failure reset the count, so every cycle looked like the first and warned again. Warnings now follow a 60 second cooldown, and each one is answered by exactly one recovery message.
…ng-when-job-notifications-fail
A lost LISTEN connection was rebuilt inside sqlx before it reported the loss, so a caller that cancels the wait on a timer discarded that work every second and never learned the connection was down. Reconnect here instead, under a 30 second deadline, counting attempts.
Each of the 8 worker loops retried the subscription once a second for as long as it kept failing, so a struggling database was asked for 8 new connections a second. Space attempts out from 1 second, doubling to 30, and start over once one survives 30 seconds.
A connection that fails dozens of times a minute logged the same single line every 60 seconds as a one-off blip would. Carry the number of failures the warning covers, and the number of attempts a recovery took, so the two are told apart at a glance.
The loss was blamed on a cancellation landing inside the re-subscription. It happens whether or not the wait is cancelled, because PostgreSQL only holds notifications for sessions that are listening at the time one is sent.
The loop's subscribe handling had no test, so removing the re-subscribe step or making a failed startup subscribe fatal both went unnoticed. Move that handling into a Subscription type the tests can drive with a queue that has no database behind it.
Worker loops space out attempts to re-open their job-available notification connection, 1 second doubling to 30, but a failed attempt never fed that pacing. A database refusing connections was asked again on every 1 second tick by all 8 loops instead of less and less often.
The wording said 3 reconnects within 60 seconds, but each failure only has to land within 60 seconds of the previous one to continue the run, so 3 of them can span nearly 120 seconds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR keeps a worker processing jobs when its database notification connection fails, and makes the failure visible. Each worker loop polls the queue every second and separately holds a Postgres LISTEN/NOTIFY connection that only wakes it sooner when work arrives, so losing that connection should cost about a second of latency and nothing else.
It did not. An error on that connection crashed the loop, and one crash brought down all of them: a single dropped connection halted every job while the process still looked healthy. Failing to open it at startup did the same. Underneath, the database driver quietly rebuilt dropped connections forever, with nothing above debug to show for it.
A worker now degrades to polling when the connection fails, starts that way if it cannot subscribe at startup, and retries on a pace that backs off to 30 seconds while the database stays unhealthy. Rebuild attempts are bounded in time and counted, so a connection that keeps dropping surfaces as an error and a warning instead of being rebuilt forever.