Repository navigation
Releases
Releases · danish-a1/goqueue
v0.3.6
Added
goqueue.Permanent(err)(alsojob.Permanent) marks a job error as non-retryable, andgoqueue.IsPermanent(err)detects the mark. When a handler returns a permanent error (also when wrapped again with%w), the worker skips the remaining retries and dead-letters the job at once instead of retrying a request that cannot succeed, such as one an API rejected with a 4xx. The DLQ entry keeps the original error chain.Permanent(nil)is nil.
v0.3.5
Fixed
- Jobs lost when they failed during shutdown. A job that failed on its last attempt after the worker context was canceled was dead-lettered with that canceled context, so network DLQs (Redis, Postgres, MySQL) refused the push; the job was then acked and gone. DLQ pushes now detach from the worker context's cancellation (
context.WithoutCancel) with their own 5s timeout (dlqPushTimeout). Regression test:TestDLQPushSurvivesWorkerCtxCancel.
v0.3.4
Fixed
- Graceful shutdown abandoned in-flight jobs. Apps cancel the worker context (their signal context) before calling
Shutdown, andprocessJobSafelystopped waiting for the handler as soon as that context was done:Shutdownreturned while the job was still running, and the job was counted as failed (retried or dead-lettered withcontext canceled). The worker now waits for the handler to return when the worker context is canceled; the handler still sees the cancel, andShutdown's own deadline bounds the wait. Per-job timeouts still abandon the handler as before. Regression test:TestShutdownWaitsForInFlightJobAfterCtxCancel.
v0.3.3
Fixed
- Busy spin in the metrics worker. Its
selecthad adefault:branch that calledDequeueMetricsback to back, so whenever the call returned at once (SQS and database stores when empty, the memory store before its first metric, any store on error) the goroutine burned a full CPU core and, on errors, flooded the log. It now polls on a timer: after an empty or failed call it waits 1s (metricsIdleWait), after a delivered metric it polls again immediately to drain. Shutdown and new metrics still interrupt the wait. Regression tests:TestMetricsWorkerDoesNotSpinWhenIdle,TestMetricsWorkerDeliversMetric. TestVersionno longer hard-codes a version string, which had it failing since 0.3.2.
v0.3.2
Fixed
- Data race between
Worker.StartandWorker.Shutdown.StartlaunchedstartMetricsWorkerwithgo, so itswg.Add(1)could run afterShutdownhad enteredwg.Wait()(reported by-race; anAddracing aWaiton a zero counter is WaitGroup misuse).startMetricsWorkeris now called synchronously: it still runs in its own goroutine, but itsAddhappens beforeStartreturns. Regression test:TestWorkerShutdownImmediatelyAfterStart.
Docs
- README: Features list and comparison table now mention the web dashboard; version badge corrected (it was stuck at 0.0.1, so
scripts/release.shcould not bump it).
v0.3.1
Fixed
- Redis ACL username is now sent to Redis.
NewRedisConfighas accepted a username since 0.2.3 and stored it since 0.2.5, but the Redis client was built without it, so connections authenticated as the default user and failed on servers that require an ACL user (WRONGPASS invalid username-password pair).
Full Changelog: v0.3.0...v0.3.1
v0.3.0
Added
dashboardpackage: a password-protected web UI served as a plainhttp.Handler(Redis driver only, stdlib only, no new dependencies). Shows totals, per-day history, per-queue counts, job IDs, busy workers, retries, dead and deleted jobs, with Retry / Kill / Delete / Restore (single and batch, atomic Lua moves, nothing is hard-deleted). Mount it withdashboard.New(dashboard.Options{...}). An empty access token disables it (404 on every path); tokens must be at least 32 bytes. Seedocs/dashboard.md.- Tracking keys (Redis):
goqueue:queuesregistry,goqueue:stat:*processed/failed counters (global, daily, per queue),goqueue:process:*worker heartbeats and busy-job hashes,goqueue:deleted:<queue>deleted-job lists. Key names live inadapter/redis/keys.go.NewQueueregisters its queue withHSETNX(a producer never overwrites an existing dead key);Worker.Startregisters the consumer's real dead key withHSET, so a producer on the default DLQ cannot hide a consumer's custom DLQ list. - Dashboard actions return
{"Done","NotFound","Skipped"}.Skippedholds entries still at the source that cannot be moved (missing job name or unreadable data); the UI keeps their rows. examples/dashboard: runnable demo on miniredis.
Changed
- DLQ entries (Redis) now share one shape (
dlq.Entry): fieldqueue_nameis renamedqueue,job_nameis added, and the error is capped at 256 bytes.dlq.DecodeEntrystill reads legacyqueue_name. RedisDLQ.Pushnow usesRPUSH(newest entry at the tail instead of the head), keeps at most 1000 entries per queue and expires the list after 30 days.- Retry envelopes carry
error_messageandfailed_at(last failed attempt);Popexposes it asJobContext.LastError. Old envelopes without these fields still decode.
Breaking
RedisDLQordering: the newest dead job is now at the tail of the list, not the head. Anything that reads the list by index (LINDEX 0,LRANGE 0 n) now sees the oldest entries first.RedisDLQfield rename:queue_nameis nowqueue. Readers must usedlq.DecodeEntryor accept both names. Entries already stored withqueue_namestay readable throughDecodeEntry.
Upgrade notes
Legacy RedisDLQ lists (0.2.9 and older) were written with LPUSH (newest entry at the head) and never capped. 0.3.0 pushes with RPUSH and then LTRIM -1000 -1, which keeps the tail. On the first dead job after the upgrade:
- a legacy list longer than 1000 entries loses its newest entries (they sit at the head), and
- the remaining legacy entries show in reverse order in the dashboard (oldest as newest).
Before upgrading, with the old workers stopped:
- Find long lists:
redis-cli --scan --pattern 'dlq:*' | while read k; do echo "$(redis-cli LLEN "$k") $k"; done | sort -rn - Archive or drain every
dlq:*list longer than 1000 entries, for exampleredis-cli COPY dlq:<queue> dlq-archive:<queue>(Redis 6.2+) and process or drop the archive yourself. - Reverse each legacy list in place once, so the newest entry is at the tail. Run it exactly once per list (a second run reverses it back):
redis-cli EVAL "local v = redis.call('LRANGE', KEYS[1], 0, -1); redis.call('DEL', KEYS[1]); for i = #v, 1, -1 do redis.call('RPUSH', KEYS[1], v[i]) end; return #v" 1 dlq:<queue>Then start the 0.3.0 workers. Custom DLQ adapters that already RPUSH with a cap (for example fleetlery's) are unaffected.
Fixed
<queue>:metrics:ackno longer grows without bound:DequeueMetricsusesBRPOPinstead ofBRPOPLPUSHand never wrote an ack for metrics.MetricsAckQueueSuffixis deprecated. Clean up existing lists with:redis-cli --scan --pattern '*:metrics:ack' | xargs redis-cli del.
Full Changelog: v0.2.9...v0.3.0
v0.2.9
Fixed
- Idle workers now stop promptly on shutdown. The Redis
Popblocked forever on an empty queue (BRPOPLPUSHwith timeout 0), soShutdownalways waited the full timeout and returnedcontext.DeadlineExceeded.Popnow blocks for at most 1 second and returns an empty job, letting the worker loop re-check for shutdown. - The metrics goroutine now also exits on
Shutdown, not only when theStartcontext is canceled, soShutdownno longer times out when the caller keeps that context alive.
Full Changelog: v0.2.8...v0.2.9
v0.2.8
Added
- Reclaim jobs left behind by a crashed process. A Redis job sits in
processing:<queue>while a worker runs it. If the process is killed (SIGKILL, OOM, host loss), the job used to stay there and never run again.Config.WithReclaimProcessingOnStart(true)now moves those jobs back onto the queue whenStartWorkersruns.- Opt-in: only enable it when a single consumer instance serves the queue. With several instances it would also move jobs another instance is still running.
- Stores without a processing list are skipped; a reclaim failure is logged and does not stop the workers.
v0.2.7
Fixed
- Redis DLQ never received jobs.
RedisDLQ.Pushredeclarederrwhile marshalling the job, soerr.Error()panicked on nil for every job that exhausted its retries. - Caller DLQ adapter ignored.
NewQueuereplaced a configuredDLQAdapterwith the driver default for the Redis and database drivers. The default now applies only when no adapter is set. - Database driver init.
NewQueueused the database store before checking its init error, so a failed connection panicked. - Panicking jobs crashed the process. The job handler runs on its own goroutine, out of reach of the worker's
recover. A panic now becomes a job error and follows the normal retry / DLQ path.
Also includes v0.2.6: the Redis health check no longer closes the shared client after a transient ping failure.