Worker writes in atepg (CreateWorker/UpdateWorker/DeleteWorker) carry a pg_notify inside the write transaction for WatchWorkers(code). Postgres serializes all notifying commits through a global lock, which caps worker writes at ~600/s on any instance size — latency is fine at low rates but collapses past the ceiling:
Latency results for workerUpdates when 1000 workers preloaded.
| worker updates/s |
p50 |
p99 |
| 250 |
4.7 ms |
7.0 ms |
| 500 |
4.7 ms |
16 ms |
| 1,000 |
610 ms |
1,163 ms |
The same update path without the notify (UpdateActor) stays flat at 2,000/s (p99 = 6.5 ms), so the write itself is not the problem. This is a known Postgres limitation (Recall.ai, DBOS;. The requirements target O(10K) worker updates/s. NOTIFY's 8 KB payload limit also fails writes outright.
Proposed fix
Replace pg_notify with a worker_changes outbox table written in the same transaction (delivery-iff-commit preserved), and have WatchWorkers poll it every 100 ms instead of LISTEN (well within the ≤1 s watch target).
Reference articles: https://microservices.io/patterns/data/transaction-log-tailing.html
https://oltionzefi.com/en/blog/scaling-outbox-postgres-part-3/
Worker writes in atepg (
CreateWorker/UpdateWorker/DeleteWorker) carry apg_notifyinside the write transaction forWatchWorkers(code). Postgres serializes all notifying commits through a global lock, which caps worker writes at ~600/s on any instance size — latency is fine at low rates but collapses past the ceiling:Latency results for workerUpdates when 1000 workers preloaded.
The same update path without the notify (
UpdateActor) stays flat at 2,000/s (p99 = 6.5 ms), so the write itself is not the problem. This is a known Postgres limitation (Recall.ai, DBOS;. The requirements target O(10K) worker updates/s. NOTIFY's 8 KB payload limit also fails writes outright.Proposed fix
Replace
pg_notifywith aworker_changesoutbox table written in the same transaction (delivery-iff-commit preserved), and haveWatchWorkerspoll it every 100 ms instead of LISTEN (well within the ≤1 s watch target).Reference articles: https://microservices.io/patterns/data/transaction-log-tailing.html
https://oltionzefi.com/en/blog/scaling-outbox-postgres-part-3/