You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Go port of BullMQ — would you consider taking it in-tree?
#5006
We have a working Go port of BullMQ running in a fork, and before growing it any
further we would rather ask the question it depends on: would you take it in-tree as a
sibling to rust/, python/ and the others — and on what terms? We are happy to own
it; what we cannot decide alone is whether you want another language in the repo.
It is not a weekend experiment: it has been running in our own system for a couple of
months while we move some services from Node.js to Go. Those services share one Redis and
the same queues with the Node ones that stay, which is why interop in both directions is
something we test rather than assume.
github.com/klerick/bullmq/go, Go 1.25, built on go-redis/v9 with the client
injectable (WithClient) so it fits a DI container.
Queue, Worker, Job, FlowProducer, JobScheduler, QueueEvents; retries and backoff
(including custom strategies), rate limiting, global concurrency, stalled recovery and
lock renewal, deduplication, metrics, and OpenTelemetry through a dependency-free Telemetry interface (the W3C metadata is wire-compatible with BullMQOtel, so traces
span Go and Node).
~4.5k lines of implementation and ~5.6k of tests; 146 tests, the integration ones
against a real Redis, plus cross-runtime interop tests in both directions (Node
produces → Go consumes and back, with job counts and cron scheduling compared against
Node on the same Redis), pinned to bullmq@6.4.1. CI runs all of it.
The part that decides everything: script drift
The port does not reimplement any Lua. It vendors rawScripts through copy:lua:go, mirroring copy:lua:rust, and embeds the directory with go:embed. The
scripts are committed, unlike in rust/ and python/ — a Go module is served straight
from a git tag, and anything untracked never reaches the consumer, so an unvendored
script would break go build rather than publish.
Drift is caught mechanically: a test
pins the KEYS count of every script the port binds
(36 of the 63), so a renamed -N fails the build. It earned its keep on the 6.4 bump — moveToDelayed 12 → 11, getCountsPerPriority 4 → 2 and reprocessJob 8 → 9 all
surfaced there rather than in production.
On keeping up: we moved the port from 6.0.10 to 6.4.1 ourselves, bindings and interop
included, in a day. If you would want a CI job, a CODEOWNERS entry, or a release cadence
of your choosing, say so and we will match it.
How it was built
To be straightforward about it: the port was written with heavy AI assistance, with a
human reviewing and making the calls. Every decision was taken from the sources in this
repo rather than invented — rust/ for the layout, python/ for the readable API
surface, the TypeScript for semantics — and we did not trust that process on its own.
Hence the tests, the interop runs against Node on the same Redis, and the KEYS guard
described above: a translation can look right and be wrong, and those are what catch it.
The two bugs below are a by-product of the same discipline — they turned up while
comparing your code with what the port does, line by line.
Two upstream bugs we ran into while porting
Both are in the TypeScript side, both still present in 6.4.1, and we would gladly open
issues and PRs for them — independently of what you decide about the port.
1. Flow nodes declared without opts lose the trace context. In addNode the
propagation metadata is attached only when node.opts exists
(src/classes/flow-producer.ts:418 — if (srcPropagationMetadata && opts)). A node
written as { name, queueName, data } therefore stores no tm, and the worker that
picks it up opens a new root trace instead of continuing the producer's. No other
producing path has that condition: queue.ts:324 (add), queue.ts:409 (addBulk)
and job-scheduler.ts:196 all inject regardless. Dropping && opts looks like the
whole fix — the metadata does not depend on user options. Our port injects for every
node, and an interop test pins the difference against Node, which is how we noticed.
2. The telemetry metadata written on failure is never read back.Job.moveToFailed
puts tm into fieldsToUpdate, and updateJobFields HMSETs it as a top-level hash
field — but every reader takes tm out of the stored opts JSON
(optsFromJSON; raw2jobData has no raw.tm, and JobJsonRaw declares none). So the
failure span's context never reaches the next attempt: a retried job continues from the
original producer span. Either map it back in raw2jobData or stop writing it; we chose
not to mirror the write in the port.
What we deliberately left out
relay* (experimental, and no port implements it yet), the Postgres backend, sandboxed
processors (a goroutine makes them moot), the administrative internal spans around queue
and worker lifecycle, and Redis Cluster/Sentinel. Where the port diverges on purpose —
the flow tm injection above, and telemetry.omitContext, which it expresses
differently — it is written down in FEATURE_PARITY.md
rather than left to be discovered.
If the answer is yes
Would you prefer one PR with the subtree, or a stack — scaffold plus the Lua sync
mechanism first, so the piece that affects your repo can be reviewed on its own, then
the features in waves? We are fine either way, and equally fine with "no, keep it as a
fork" — in that case we will say so plainly in the README so nobody mistakes it for an
official port.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We have a working Go port of BullMQ running in a fork, and before growing it any
further we would rather ask the question it depends on: would you take it in-tree as a
sibling to
rust/,python/and the others — and on what terms? We are happy to ownit; what we cannot decide alone is whether you want another language in the repo.
It is not a weekend experiment: it has been running in our own system for a couple of
months while we move some services from Node.js to Go. Those services share one Redis and
the same queues with the Node ones that stay, which is why interop in both directions is
something we test rather than assume.
What exists today
go/on thego-portbranch(README,
FEATURE_PARITY,
CI workflow),
released as
go/v0.7.1.github.com/klerick/bullmq/go, Go 1.25, built ongo-redis/v9with the clientinjectable (
WithClient) so it fits a DI container.(including custom strategies), rate limiting, global concurrency, stalled recovery and
lock renewal, deduplication, metrics, and OpenTelemetry through a dependency-free
Telemetryinterface (the W3C metadata is wire-compatible with BullMQOtel, so tracesspan Go and Node).
against a real Redis, plus cross-runtime interop tests in both directions (Node
produces → Go consumes and back, with job counts and cron scheduling compared against
Node on the same Redis), pinned to
bullmq@6.4.1. CI runs all of it.The part that decides everything: script drift
The port does not reimplement any Lua. It vendors
rawScriptsthroughcopy:lua:go, mirroringcopy:lua:rust, and embeds the directory withgo:embed. Thescripts are committed, unlike in
rust/andpython/— a Go module is served straightfrom a git tag, and anything untracked never reaches the consumer, so an unvendored
script would break
go buildrather than publish.Drift is caught mechanically: a test
pins the KEYS count of every script the port binds
(36 of the 63), so a renamed
-Nfails the build. It earned its keep on the 6.4 bump —moveToDelayed12 → 11,getCountsPerPriority4 → 2 andreprocessJob8 → 9 allsurfaced there rather than in production.
On keeping up: we moved the port from 6.0.10 to 6.4.1 ourselves, bindings and interop
included, in a day. If you would want a CI job, a CODEOWNERS entry, or a release cadence
of your choosing, say so and we will match it.
How it was built
To be straightforward about it: the port was written with heavy AI assistance, with a
human reviewing and making the calls. Every decision was taken from the sources in this
repo rather than invented —
rust/for the layout,python/for the readable APIsurface, the TypeScript for semantics — and we did not trust that process on its own.
Hence the tests, the interop runs against Node on the same Redis, and the KEYS guard
described above: a translation can look right and be wrong, and those are what catch it.
The two bugs below are a by-product of the same discipline — they turned up while
comparing your code with what the port does, line by line.
Two upstream bugs we ran into while porting
Both are in the TypeScript side, both still present in 6.4.1, and we would gladly open
issues and PRs for them — independently of what you decide about the port.
1. Flow nodes declared without
optslose the trace context. InaddNodethepropagation metadata is attached only when
node.optsexists(
src/classes/flow-producer.ts:418—if (srcPropagationMetadata && opts)). A nodewritten as
{ name, queueName, data }therefore stores notm, and the worker thatpicks it up opens a new root trace instead of continuing the producer's. No other
producing path has that condition:
queue.ts:324(add),queue.ts:409(addBulk)and
job-scheduler.ts:196all inject regardless. Dropping&& optslooks like thewhole fix — the metadata does not depend on user options. Our port injects for every
node, and an interop test pins the difference against Node, which is how we noticed.
2. The telemetry metadata written on failure is never read back.
Job.moveToFailedputs
tmintofieldsToUpdate, andupdateJobFieldsHMSETs it as a top-level hashfield — but every reader takes
tmout of the stored opts JSON(
optsFromJSON;raw2jobDatahas noraw.tm, andJobJsonRawdeclares none). So thefailure span's context never reaches the next attempt: a retried job continues from the
original producer span. Either map it back in
raw2jobDataor stop writing it; we chosenot to mirror the write in the port.
What we deliberately left out
relay*(experimental, and no port implements it yet), the Postgres backend, sandboxedprocessors (a goroutine makes them moot), the administrative internal spans around queue
and worker lifecycle, and Redis Cluster/Sentinel. Where the port diverges on purpose —
the flow
tminjection above, andtelemetry.omitContext, which it expressesdifferently — it is written down in
FEATURE_PARITY.md
rather than left to be discovered.
If the answer is yes
Would you prefer one PR with the subtree, or a stack — scaffold plus the Lua sync
mechanism first, so the piece that affects your repo can be reviewed on its own, then
the features in waves? We are fine either way, and equally fine with "no, keep it as a
fork" — in that case we will say so plainly in the README so nobody mistakes it for an
official port.
All reactions