Skip to content

Fix Watcher blocking during slow pool termination#343

Merged
v0idpwn merged 7 commits intomasterfrom
fix/slow-watcher
Apr 20, 2026
Merged

Fix Watcher blocking during slow pool termination#343
v0idpwn merged 7 commits intomasterfrom
fix/slow-watcher

Conversation

@v0idpwn
Copy link
Copy Markdown
Member

@v0idpwn v0idpwn commented Apr 20, 2026

The Watcher calls DynamicSupervisor.terminate_child/2 synchronously,
blocking all other watch/terminate operations until the pool finishes
shutting down.

This can cause applications running multiple pools to have degraded
performance when connections take long to terminate (observed in production
with Postgrex).

Added a test reproducing the issue, and a proposed fix via a Task calling
GenServer.stop/1.

v0idpwn added 6 commits April 20, 2026 08:08
The Watcher calls DynamicSupervisor.terminate_child/2 synchronously,
blocking all other watch/terminate operations until the pool finishes
shutting down. This test reproduces the issue with a slow disconnect.
@impl true
def terminate(_, {_, started_refs}) do
for {_, {caller_pid, _}} <- started_refs do
for {_, {caller_pid, _}} when caller_pid != nil <- started_refs do
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was necessary because now we keep ref => {nil, nil} when a termination is on flight

@v0idpwn v0idpwn changed the title Fix Watcher contention during slow pool termination Fix Watcher blocking during slow pool termination Apr 20, 2026
Comment thread lib/db_connection/watcher.ex Outdated
@v0idpwn v0idpwn merged commit 7ea461e into master Apr 20, 2026
2 checks passed
@v0idpwn v0idpwn deleted the fix/slow-watcher branch April 20, 2026 18:38
@v0idpwn
Copy link
Copy Markdown
Member Author

v0idpwn commented Apr 20, 2026

Thank you @josevalim! 💚
We tested it with Postgrex and it seems to have resolved the issue. We will be rolling it out in production and I'll report back confirming the results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants