Releases: efureev/go-outbox
Release list
v1.7.1
v1.7.0
No new features. This one is about whether the previous six releases do what they say, and the
answer turned out to be "mostly" — a shipped recipe did not work at all, and a piece of advice
credited the wrong thing.
Fixed
-
The grant in use case 8 was wrong, and a deployment
that followed it could not deliver a single message. The recipe saidGRANT INSERTand, in a
comment, "and nothing else: no SELECT". The driver inserts withON CONFLICT (id) DO NOTHING,
and PostgreSQL requires read access on a table whenON CONFLICTnames a target: every batch
came back42501 permission denied.Measured rather than reasoned about, because the two forms differ in more than their grant:
Form Requires Suppresses ON CONFLICT (id) DO NOTHINGINSERTandSELECTa repeat of the same id ON CONFLICT DO NOTHINGINSERTany unique violation, silently The target-less form would have kept the narrow grant. It also swallowed a message with a fresh
id and a taken business key and reported success — three inserts, one row. So the conflict target
stays and the grant widens toGRANT INSERT, SELECT: a lost message is worse than a read
privilege. The recipe, the driver's comment and a test that asserts the role still cannot
UPDATEorDELETEnow say the same thing. -
outbox migrate <typo>answered a mistyped action with two lines aboutOUTBOX_DB_USER. The
configuration was loaded before the action was checked, so an operator who typedmigrate down
on a machine without database credentials was told about the credentials. Checking the action
costs nothing and now comes first. -
Use case 6 credited pgx with the batch protocol's win. It
advised a producer writing hundreds of messages per transaction to move topkg/outboxclientand
pgx. Measured in three arms — pgx batched, pgx looped,database/sqllooped — the two loops are
indistinguishable at 116 against 116 microseconds per message at a hundred, and 109 against 111 at
five hundred, with the spread within a single arm wider than the gap between them. Switching
clients without callingEnqueueBatchchanges nothing. The rest of the recipe held: at one
message per transaction all three arms sit inside the commit's shadow, and at five hundred a loop
adds 43 ms to a transaction a user is waiting on.
Added
-
docs/Benchmarks.md — every figure this changelog and the README quote,
each with the benchmark that produced it and the caveats that make it honest. Three benchmarks
were missing, and each was propping up a claim that had no number behind it.BenchmarkDrainDestinationruns one drain into three destinations. A table reaches ~17 700
msg/s against RabbitMQ's ~6 500, or 44% of the dispatcher's own ceiling against the broker's
16%. That is the measurement behind "a broker is not a mandatory dependency", which until now was
an argument. Both destinations are containers on the same host, and the inbox is in the same
database as the outbox — the cheapest arrangement, not the representative one.BenchmarkStore*isolates claim and write-back from the pipeline, the workers, the bus and every
destination: the floor no driver can go below. The write-back costs about as much as the
claim — roughly half the throughput at every batch size, which is what a confirmed publication
costs and is not a knob. Nacking is 20–30% dearer than acking at a batch of 500 and
indistinguishable below 200, which is one more thing the circuit breaker is buying: during an
outage the failure path is the one under load.BenchmarkEnqueuemeasures the producer's side, inside the business transaction, which is the
cost a user waits on.New targets:
make bench-destination,make bench-enqueue,make bench-store. -
make mutation— mutation testing on the three packages where it pays, pinned to
gremlins v0.6.0. One trap is baked into the target with its reason: gremlins derives its timeout
from a baseline run, and for packages whose tests take milliseconds it comes out so tight that
every mutant reportsTIMED OUTand the summary reads "efficacy 0%" — which looks like a suite
that catches nothing rather than a tool that ran nothing.
Changed
- The administrative commands take an interface and a writer rather than
*store.Storeand
os.Stdout. The interface is the same four operations the admin API needs, which is the claim
the file's own comment already made — that neither path can drift into being the one that does it
correctly — now written where the compiler can see it. No behaviour changes; the commands became
testable, going from 13.3% covered to 83%.
Testing
Nothing here changes what the dispatcher does. It changes what is known about it.
| Before | After | |
|---|---|---|
Coverage, internal/... and pkg/... |
— | 79.7% |
internal/app |
1.7% | 49.4% |
internal/observability |
18.8% | 90.1% |
cmd/outbox |
13.3% | 83.0% |
internal/dispatch |
72.6% | 95.9% |
internal/core |
84.3% | 97.4% |
Mutation efficacy, internal/config |
68.4% | 96.1% |
| Mutation efficacy, the circuit breaker | 70.0% | 90.0% |
Two findings are worth more than the numbers.
A whole class of validation test was wrong. Forty of the fifty-four mutants surviving in
internal/config were one mistake repeated: every rule was fed a value outside its boundary and
never the boundary itself. BATCH_SIZE=0 is rejected, and nothing asserted that BATCH_SIZE=1 is
accepted, so a comparison shifted by one survived everywhere. That is the more expensive direction —
a rule refusing its own documented minimum refuses the configuration the documentation told the
operator to write. Thirty rules are now exercised twice, past the boundary and exactly on it. None
turned out to be misplaced; nothing was holding them there.
A test that passed while covering nothing. TestShutdownReleasesUnattemptedClaims asserted
attempted + released == 9, which held with released == 0: the release path it was named for ran
zero times. It now cancels from inside Claim, and fails when the release call is removed.
Fourteen mutants survive across the three packages and each is written down in the test file for
its package, with the reason it is not worth killing — equivalent mutants, os.Hostname(), and
guards that only suppress a duplicate complaint about a value another rule has already rejected.
Requirements
- Go 1.26
- PostgreSQL 13 or newer
- RabbitMQ 3.8+ and/or Kafka 2.4+
v1.6.0
The broker stops being mandatory. A dispatcher can now deliver into a table — the consumer's inbox
— and a producer that is not on pgx can write to the outbox.
Added
-
A
postgresdriver: delivery into a table instead of to a broker. The destination is the
consumer's inbox, and the dispatcher only inserts into it — it does not create the table, read
it, update it or clean it up. That boundary is what keeps this a destination rather than the
beginnings of a consumer framework.It buys a guarantee no broker offers. Delivery stays at-least-once, because the insert and the
write-back marking the rowsentare two commits and a replica can die between them, but the
inbox's primary key makes the repeat harmless: the driver inserts withON CONFLICT (id) DO NOTHINGand reports a conflict as a delivery. Deduplication stops being an obligation on the
consumer's code and becomes a property of its schema.It also costs nothing to carry. pgx is already linked, so the driver adds zero modules —
the only candidate on the driver spec of which that is true, against
+1.58 MB for NATS and +6.33 MB for Redis Streams. And it removes the broker from the list of
things a deployment must run at all.An empty
DSNmeans the database the dispatcher already reads its outbox from, which is the
modular-monolith case. Two configurations are refused at startup: a table that does not exist,
and a destination that is the dispatcher's own outbox table — which would deliver to itself,
every published message becoming a new message to publish.A batch is one statement on the happy path. When the database refuses, the batch is replayed one
message at a time, because a positional error slice is the contract and one malformed message
must not condemn the rest; that second pass is skipped when the database could not be reached at
all. Classification reads the server's own SQLSTATE, which makes this the least guessy of the
three drivers.Example schema and the reasoning behind it:
migrations/inbox/messages.sql. Use cases and the flows:
docs/InboxSpec.ru.md; whether it was worth building at all:
docs/PostgresDestination.ru.md.Three properties are measured rather than asserted. A batch is one statement — proved by a
statement-level trigger on the inbox, which fires once per statement whatever the row count,
rather than by timing. On the failure path the errors land positionally, including on the first
and last message, where an off-by-one in the isolating pass would show. And there is no
payload-size class at all: 32 MiB in one message and 64 MiB in one batch go in and come back byte
for byte, where every broker driver has a permanent failure for exceeding a frame or a
message.max.bytes.Cleaning the inbox is the consumer's. The janitor sweeps the outbox and never touches the
destination, so the inbox grows until its owner cleans it — quietly, because nothing on the
dispatcher's side is looking. It is the one failure of this driver that surfaces six months later
rather than immediately, which is why it is stated in three places rather than one.No fan-out. An inbox is point-to-point, so one event reaching three consumers means the
producer writing three rows. Where a fan-out is wanted, a broker is still the right tool, and the
documentation says so rather than leaving it to be discovered on the second subscriber. -
Three use cases for it, one page each:
a modular monolith with no broker at all,
two services delivering into each other's inbox, and
dead letters in a table rather than a topic — which needs no code
change, because the forwarder already publishes through the router.
Fixed
- CI builds against the current Go patch release.
go.modsaysgo 1.26, and without
check-latestthe action takes whatever patch the runner had cached — which is how the same
commit went green on 1.26.6 and red on 1.26.5. Standard library advisories are fixed by patch
releases, so which one a run gets decided whethergovulncheckhad anything to report. It now
applies to the release workflow too, where the stake is higher: a release could otherwise ship
binaries built against a known-vulnerablecrypto/tlson a day the cache happened to be stale,
and nothing in the pipeline would have said so.
Changed
-
config.DBConfig.ConnStringreplaces the unexported DSN assembly ininternal/store. The pool
is no longer the only thing that needs it: a driver delivering into PostgreSQL has to be able to
say "the same database the dispatcher reads from" without repeating how that database is
described. -
pkg/outboxsql, the producer client for everybody not on pgx.pkg/outboxclienttakes a
pgx.Tx, which is the wrong dependency to force on a codebase that chosesqlx,gormor the
standard library. The new one takes anything withExecContextand imports no driver at all.It is a package of its own rather than another constructor in
outboxclient, because importing
that package brings pgx with it — which is the thing being avoided. The twoMessagetypes are
therefore duplicated, and a test compares them by reflection so the copies cannot drift apart
unnoticed.It costs the daemon nothing:
go list -deps ./cmd/outboxdoes not contain either package, and
the binary measures 21.87 MB before and after. Integration tests cover both PostgreSQL drivers a
database/sqluser realistically holds —lib/pqandpgx/v5/stdlib— including a payload of
arbitrary non-UTF-8 bytes and ajsonbround trip. The recipe is
use case 6.
Changed
- The use cases are one page each, in docs/usecases, with
docs/UseCases.md as the index. The single document had reached seven hundred
lines and six recipes, which is past the point where anybody reads it end to end. Every recipe is
carried over unchanged.
v1.5.0
Two things a deployment reaches for once it is large enough to need them: a trace that shows where
the time went, and a table shape that keeps up past ten million rows a day.
Added
-
Range partitioning, for deployments past roughly ten million rows a day. At that volume the
retention sweep stops keeping up: a chunkedDELETEcreates dead tuples faster than autovacuum
reclaims them, so the table grows while apparently being cleaned. Partitioning bycreated_at
turns the same work intoDROP TABLE— a catalogue change and an unlink, costing the same
whatever the partition held.It is opt-in and needs no fork of the migration set: apply
migrations/partitioned/messages.sqlto an empty database
and the released migrations run over it unchanged, because 0001 creates the table only
IF NOT EXISTSand its indexes are created on the parent and propagated. The dispatcher notices
the shape of the table by itself and switches retention from deleting rows to dropping
partitions; every query it runs is the same either way, because partitioning is transparent to
DML.A partition is dropped only when everything in it has been delivered and the most recent
delivery is past retention. Neither half follows from the partition's bounds — those are on
created_atwhile retention is ondispatched_at— so a partition full of week-old messages may
still hold one that failed and is waiting for somebody. The shipped schema also carries a default
partition, because a row that fits no partition is a failedINSERTinside the producer's
business transaction: a stopped janitor must cost a warning, not a rolled-back application.OUTBOX_JANITOR_PARTITION_AHEAD(3) is how many days are kept created in front.
outbox_partitions_dropped_totalandoutbox_default_partition_rowsreport what happens.It changes the primary key, which the roadmap said it would not. PostgreSQL requires a unique
constraint on a partitioned table to include the partition key, soidalone cannot be the
primary key and becomes(id, created_at): the database no longer enforces that an id appears
once across the whole table, only once per day. Consumers already deduplicate on the message id
under at-least-once delivery, so nothing breaks, but it is a guarantee given up rather than a
detail. Measured on 405k rows across 31 daily partitions, claiming executes in about 0.25 ms
against 0.18 ms unpartitioned; planning goes from 0.4 ms to 2 ms and is paid once, since pgx
prepares its statements. -
make soak— the resilience scenarios under continuous load for as long as you are willing to
wait, behind its own build tag so it never runs by accident. The ordinary resilience tests break
one thing, observe and heal, which establishes that each failure is handled but not that the
dispatcher survives them overlapping while work keeps arriving. A 45-second run inserted 2,249
messages while both brokers and the database were broken in rotation, and delivered every one of
them. -
An
outbox.publishspan per message, closing the gap in the producer's trace. A producer's
span ends when its transaction commits and a consumer's starts when the broker hands it a
message; between them is an interval exactly the width of the outbox lag, which a metric can size
but not explain. The span is parented to the producer'straceparentand re-injected into the
message's headers, so a trace reads producer →outbox.publish→ consumer, in one trace, with
the wait visible as the space in front of the middle span.A producer that never traced still gets a span and its consumer a header to continue from:
requiring the producer to have traced first would make this useful only where it was needed
least.Configured by
OUTBOX_OTEL_ENDPOINT(OTLP/HTTP), withOUTBOX_OTEL_INSECUREand
OUTBOX_OTEL_SAMPLING. Sampling defers to the producer's decision when there is one, so a trace
sampled at the source does not lose its middle here.
Changed
-
With tracing on, the
traceparentreaching the broker names the dispatcher's span rather
than the producer's. The trace id is unchanged — nothing leaves the producer's trace, only the
parent moves — which is what puts the dispatcher between the two ends instead of beside them.
With tracing off, which is the default, the header is passed through untouched exactly as before. -
The image is 29 MB, up from 21 MB. The OpenTelemetry SDK and its OTLP encoder add 6.3 MB to a
15.5 MB binary whether or not a collector is ever configured, and that is the real price of this
release. What it does not cost is throughput: with no endpoint set the publish loop checks one
boolean and starts no span, at 0 allocations and about 5 ns per message. A recorded span costs
about 1.9 µs and 17 allocations. gRPC appears ingo.modas an indirect requirement of the OTLP
proto module, but no package of it is imported and none of it is linked in.
v1.4.0
The operational round: the tools an operator reaches for, and the evidence that what they are
running is what was built.
Added
-
outbox stats,outbox failedandoutbox requeue— the admin API's operations as
subcommands, over the database connection instead of over HTTP. The HTTP route needs a reachable
pod, a token and a JSON body; what is to hand during an incident is a shell in the container the
binary already lives in. Both paths run the same store calls, so neither can drift into being the
one that does it correctly.Authorisation differs on purpose. The endpoints are guarded by
OUTBOX_HTTP_ADMIN_TOKENbecause
anything that can route to the pod can call them; the commands are guarded by holding the database
credentials, which is a stronger thing to have.outbox failed -stream local,outbox requeue <id>...,outbox requeue -before <RFC3339>, and
-jsonon any of them for a pipe intojq. -
govulncheckon every CI run, andmake vulnso it is the same command locally. It reports
only the vulnerabilities the code actually reaches, so a finding is something to act on rather
than a line in an advisory feed. It runs as its own job: a vulnerable dependency is a fact about
the module, not a failing test, and should not be discovered by whoever happens to be reading a
red test run. -
A CycloneDX SBOM per platform, attached to every release. Generated from the compiled binaries
rather than from the source tree, so each one lists what was actually linked into the artefact it
describes.make dist SBOM=1writes them beside the archives, andmake sbomwrites one for a
local build; the default stays off so a developer'smake distneeds no extra tool. -
Keyless cosign signatures on the release and the image.
SHA256SUMS.cosign.bundlecovers
every archive and every SBOM through the checksum file, and the image is signed by digest — a tag
is a name that can be moved, so a signature against one says nothing about what anybody pulls.
There is no key to distribute or to lose: cosign gets a short-lived certificate against the
workflow's OIDC identity, and what a verifier establishes is that this repository, on this
workflow, produced the artefact. Verification commands are in
docs/UseCases.md.Tool versions are pinned, and cosign's own download is checked against a recorded hash. Fetching a
signing tool over the network without checking what came back would be an odd way to start
signing things. -
A Grafana dashboard,
dashboards/outbox.json. Alert rules shipped
without one, so every adopter built the same panels from the same metric reference. Thirteen
panels in four rows, with astreamvariable for narrowing to one broker when only one of them is
the problem.It is checked against the code rather than against a screenshot: a test walks every query in the
file and fails if it names a metric the dispatcher does not register, or filters on a label that
metric does not carry. Both render an empty panel, which reads as "nothing is happening" and is
indistinguishable from good news until somebody needs it. -
?stream=onGET /api/v1/messages/failed, so working through one broker's backlog does not
mean paging through everybody else's. The CLI needed the filter first; adding it to the endpoint
too is what keeps parity a fact rather than a claim.
Changed
- Administrative commands check only the configuration they use.
migrate,stats,failed
andrequeueneed a database and nothing else, and previously refused to run on a broken routing
table. The moment an operator most needs to see what stopped is often the moment the routing table
is what is wrong, and a tool that answers "your broker is misconfigured" to the question "what
failed?" is useless precisely then. The dispatcher itself is unchanged: it still refuses to start
without a routing table, because it cannot deliver without one.
v1.3.0
Added
-
The dispatcher stops claiming for a stream whose broker is unreachable. Finding a broker gone
used to change nothing about the loop: it kept claiming a batch, failing to publish it and writing
the failure back for the whole outage.What that cost was not the retries. A deferred message is rescheduled a backoff into the future,
so retrying an outage is self-limiting. New messages are not — every insert arriving while the
broker is down wakes the pipeline throughLISTEN/NOTIFYand was claimed, attempted and written
back at once. The load removed is proportional to how busy the producer is rather than to how long
the outage lasts.The pause starts at one poll interval and doubles up to
OUTBOX_DISPATCH_PAUSE_MAX(30s), and
one ordinary claim is let through each time it elapses. The trial is a real batch rather than a
health check on purpose: publishing is the capability that matters, and a health check is only a
proxy for it — one that can be green while the exchange the messages need is not there. A wake-up
is not allowed past the pause, since it carries no information the breaker does not already have.The ceiling matches the delay the RabbitMQ supervisor backs off to between reconnection attempts,
so pausing adds nothing to how soon a returning broker is noticed.0restores the previous
behaviour, which is also how the tests prove the pause is what makes the difference. -
outbox_stream_paused{stream},1while a stream has stopped claiming. This is not
decoration: while claims are held back nothing is published, sooutbox_messages_deferred_total
stops advancing precisely when an outage is most established. The gauge is the signal that
outlives the condition it reports.
Changed
- The shipped
OutboxBrokerUnreachablealert now joinsoutbox_stream_pausedwith the deferral
rate. On the rate alone it would have cleared itself a minute into every outage it exists to
report. Pipeline.RunOncereturns adispatch.Result— claimed, delivered and deferred — instead of a
bare count, so the run loop can tell an outage from a batch that simply failed.
v1.2.0
Added
-
An unreachable broker no longer spends a message's retry budget. Every failure used to
advance the attempt counter, which conflated two events deserving opposite responses. A broker
that looks at a message and refuses it should exhaust a budget — retrying will not change its
mind. A broker that cannot be reached never saw the message, and charging it for that outage
spent the budget on somebody else's problem: at the default backoff the whole budget is gone in
fifteen minutes, so a twenty-minute restart left a table full offailedrows that only ever
needed to wait, and an operator requeueing them by hand.Failures to reach a broker are now classified separately. Such a message returns to
pending
with its attempt counter untouched, marked with a newdeferred_sincecolumn, and is retried on
the ordinary backoff until the broker comes back — however long that takes. The attempt counter
measures rejections, not minutes.The classification is deliberately conservative: a per-message problem mistaken for an outage
would never advance its counter and so never reachfailed, so anything not positively
identified as unreachable stays retryable. For RabbitMQ that means the named connection errors
plus the case that matters most and looks least like an outage — a confirmation deadline expiring
on a connection that is no longer live. For Kafka it means the availability codes the protocol
reports (LeaderNotAvailable,NotEnoughReplicas, and others), network errors, and the write
timeout, but only while the caller is still running so a shutdown is not recorded as an outage. -
OUTBOX_DISPATCH_MAX_DEFERbounds how long an unreachable broker may hold a message back
before it fails anyway, measured from the first deferral rather than from the row's creation — an
old message meeting its first outage has waited none of it. The default is0, meaning
unbounded, because a message delivered late is worth more than one failed by a timeout. A message
failed this way is reported asreason="unreachable"rather thanattempts_exhausted: it was
never rejected, and its attempt counter still reads zero. -
Two metrics for the condition.
outbox_messages_deferred_total{stream,driver}counts
messages put back without spending an attempt, and theoutbox_messages_deferredgauge is how
many are waiting right now. Together withoutbox_oldest_pending_age_secondsthey separate a
backlog that is moving slowly from one that is not moving at all.outbox_broker_errors_total
gains akind="unavailable"label, and a startingOutboxBrokerUnreachablealert ships in
docs/MetricsAndAlerts.md.
Changed
attemptsnow counts times a broker rejected a message, not publish attempts made. A message
that waited out an hour-long outage and then went through records zero attempts.GET /api/v1/statsreportsmessages.deferredandsettings.max_defer, and thereadyline at
startup carriesmax_defer.- The
outbox_publish_errorsalert expression filterskind="retryable", which no longer matches
an outage;OutboxBrokerUnreachablecovers that case.
Removed
OUTBOX_HTTP_PPROF_TOKEN. The field was declared in the configuration and read by nothing —
pprof was never registered — so the variable did nothing whether it was set or not. Behaviour is
unchanged; the name is simply gone from the configuration surface.
Database
- Migration
0004_deferral.sqladds thedeferred_sincecolumn, a partial index over it, and
replaces the tworequeuefunctions so they clear it along with everything else they reset. It
is additive: existing rows readNULL, which is what "nothing is waiting on a broker" means.
v1.1.0
v1.0.0
First release.
A Transactional Outbox dispatcher: a producer writes a message to a database table inside the same
transaction as the business change it describes, and this service reads those rows and publishes
them to RabbitMQ or Kafka. Delivery is at-least-once, a row becomes sent only after the broker
acknowledges it, and any number of replicas may run against one table.
Added
-
Lease ownership on every write. A claim stamps a row with a token; every statement that
finalizes a row requires that token to match. Claiming concurrently is straightforward —
FOR UPDATE SKIP LOCKEDgives disjoint batches — but recording the outcome is not: without the
token, a replica whose lease expired mid-flight overwrites the status of a row another replica
has already reclaimed and delivered, resurrecting a delivered message indefinitely. The
invariant is enforced by the schema as well as the queries: a row is leased exactly while it is
processing, andoutbox_lease_conflicts_totalcounts the times the check fires. -
Confirmed publication. RabbitMQ publishes wait on a per-message deferred confirmation and go
through a pool of channels, so workers publish concurrently rather than queueing behind one
channel and one mutex. Kafka writes the whole batch in oneWriteMessagescall and maps the
positional error slice back onto individual messages, withacks=allby default. -
Permanent failures are distinguished from retryable ones. An unroutable message, an unknown
stream, a payload above the broker's limit or a rejected credential fails at once instead of
spending five attempts and an hour of backoff reaching the same conclusion. Retryable failures
back off exponentially with a ceiling and jitter, so a broker coming back up is not met by every
message that failed while it was down, all due at the same instant. -
LISTEN/NOTIFYwakeups. A trigger announces each insert and the relevant pipeline wakes
within milliseconds. A burst of inserts is coalesced into one wakeup, and a small jitter keeps
replicas from all claiming at the same millisecond. The poll loop stays on as reconciliation,
becauseNOTIFYis best-effort: losing one costs a poll interval, never a message. It can be
disabled entirely where the database role cannot create triggers. -
One pipeline per stream, so a broker that is down delays only its own messages. The loop is
adaptive: a full batch means there is a backlog, so the next iteration starts at once rather
than sleeping out the poll interval. -
Graceful drain. On
SIGTERMpipelines stop claiming, finish the batch in flight, record its
outcome, and hand back anything they never started — so another replica takes that work
immediately instead of waiting out the lease. -
Housekeeping: expired leases returned to the queue, backlog gauges sampled, and delivered
rows swept in bounded chunks after a configurable retention. Each cycle takes a PostgreSQL
advisory lock, so it runs on one replica per cycle however many are deployed. -
Dead-letter forwarding. A message that stops being retried can be forwarded to a destination
a consumer watches, carrying its original topic, stream, attempt count and permanence as
headers. The row stays in the table either way: the dead-letter topic is a signal, not the
record. -
Metrics on an injected registry, not on package globals, so a test can build its own and read
it back. Series for every configured stream and driver are created at startup, so a scrape
before the first message reports a zero rather than nothing — a distinction an alert expression
cannot otherwise make. Label values are bounded by the configuration, so a producer cannot mint
unbounded time series by writing an unknown name into thestreamcolumn.
outbox_oldest_pending_age_secondsis the metric to hold delivery to. -
HTTP endpoints on
net/http:/health,/ready,/api/v1/stats,
/api/v1/messages/failedfor inspecting what stopped and why, and
POST /api/v1/messages/requeuefor putting it back. The mutating endpoint is registered only
when a token is configured to guard it. -
outbox.requeueandoutbox.requeue_failed_beforeas database functions, so an operator in
psql, the admin endpoint and the CLI all take the same path. Requeueing has to reset the attempt
counter and the availability time along with the status; a hand-written UPDATE that changes only
the status leaves a row that is nominally pending and is never selected again. -
pkg/outboxclient, which takes the caller's transaction as an argument so the transactional
part is not something to remember, and generates UUIDv7 identifiers so the primary key index
stays append-ordered. -
Configuration read and validated before anything connects. Every duration is parsed at load
time, and validation reports every problem at once — a misconfigured deployment takes one
restart to diagnose rather than one per mistake. Driver settings are looked up by exact key from
a closed set, so a misspelled key is a startup error rather than a setting that silently does
nothing. -
An embedded migration runner: forward-only, one transaction per file, an advisory lock around
the run so replicas starting together apply each migration exactly once, and a recorded checksum
per file. The checksum is what makes an edited migration an error rather than a silent
divergence between a fresh install and an upgraded one.
Performance
From make bench — PostgreSQL and RabbitMQ in Docker on one machine, Go 1.26, Apple M5 Pro,
batches of 200, medians of three runs. Compare runs on one machine rather than reading the absolute
figures as a capacity plan.
| Result | |
|---|---|
| Drain, dispatcher and PostgreSQL only | ~46 000 msg/s |
| Drain via RabbitMQ, 4 workers over 4 channels | ~7 300 msg/s |
| Drain via RabbitMQ, 8 workers over 8 channels | ~12 000 msg/s |
| Insert to broker, shipped defaults | ~105 ms |
| Insert to broker, debounce and jitter minimised | ~5 ms |
Two things the sweeps say that are worth carrying into a deployment.
Throughput is bounded by the smaller of the worker count and the driver's channel pool. Eight
workers over the default four channels performs the same as four workers over four; widening the
pool to match moves it by around 60%. Raise WORKERS and CHANNELS together or neither. Without a
broker in the way the dispatcher itself never becomes the limit, so what is being tuned is the
publish path, not this process.
Latency at the defaults is the debounce window plus the mean of the replica jitter, and both
are deliberate: they turn a burst of inserts into a couple of claims and keep replicas from all
waking at the same millisecond. Trading them away takes delivery to roughly five milliseconds, at
the cost of both properties.
Distribution
- Container image on
ghcr.io/efureev/go-outbox, built forlinux/amd64and
linux/arm64on every tag. The Dockerfile cross-compiles rather than emulating, so a
multi-platform build needs no QEMU. - Prebuilt archives for Linux and macOS, amd64 and arm64, with
SHA256SUMS, attached to the
GitHub release. The release notes are the changelog entry for the version, so the two cannot
drift. go install github.com/efureev/go-outbox/cmd/outbox@latestfor anyone who has Go, though it
stamps no version.
Requirements
- Go 1.26
- PostgreSQL 13 or newer
- RabbitMQ 3.8+ and/or Kafka 2.4+