Repository navigation
CurlShot: per-account render concurrency for a screenshot API with BullMQ #4957
SergiuPogor
started this conversation in
Show and tell
Replies: 1 comment
|
Really clean pattern — using DelayedError + Redis counter for per-account concurrency without multiplying queues is something I hadn’t seen done this way before. The backoff scaling for deeper backlogs is a nice touch too. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi all, sharing a production use of BullMQ in case the pattern is useful to someone.
CurlShot is a screenshot API: one HTTP request turns a URL, HTML or Markdown into a PNG, JPEG, WebP, PDF or a scrolling video, rendered in headless Chromium. Every render (sync, async and bulk) goes through a BullMQ queue, and webhooks for async jobs go through a second queue.
A few things that worked well for us:
Per-account concurrency without extra queues. Each plan has a limit on parallel renders per account. Instead of one queue per account, the processor takes a slot from a Redis-backed counter keyed by account. If the account is already at its limit, the job is put back with
job.moveToDelayed(...)and the processor throwsDelayedError, so the job does not burn one of its attempts. Jobs deeper in an account's backlog wait a little longer before asking again, so a long backlog is not polled every second and the next jobs in line get the freed slots.Failure budget the same way. If an account keeps sending pages that fail, its jobs are deferred with the same
moveToDelayed+DelayedErrortrick instead of being failed.Worker concurrency equals the render budget. The screenshot worker's
concurrencyis the same number as the instance's browser render limit, so a job only leaves the queue when that instance can actually run it.lockDurationis 120 s withmaxStalledCount: 2, so a crashed renderer's jobs go back to the queue.Separate webhook queue. Webhook delivery has its own worker and concurrency, so a slow customer endpoint never holds a render slot.
Thanks for BullMQ. The
DelayedErrorpath in particular made fair scheduling across accounts much simpler than we expected.If you want to try the API, the free plan has 100 screenshots a month with no card: https://curlshot.com
All reactions