Skip to content

v0.23.0 — write-only outbox + event bus (in-memory + Kafka), relay & consumer

Choose a tag to compare

@daniel-garcia daniel-garcia released this 27 Jun 19:04
8012534

Reworks the outbox/events subsystem to a write-only outbox + queue-based event bus (supersedes v0.22.0's poll+lease+mark dispatcher).

Pipeline: aggregate tx → write-only outbox (Append) → RELAY (ReadAfter+cursor → bus, leader-elected) → BUS (in-memory dev | Kafka prod) → CONSUMER (group → handlers, Atomically + idempotency).

Highlights

  • Write-only outbox — only the transactional INSERT + DROP PARTITION retention; the dispatcher never mutates rows (no lease/delivered/attempts). A clean CDC source, no write+delete churn.
  • Queue-based consumption — multi-replica coherence/throughput via Kafka consumer groups (competing consumers, partition ordering, coherent offsets). In-memory bus for dev/in-process; Kafka (franz-go) for prod, Redpanda for integration tests.
  • Relay (outbox→bus) is leader-elected (SingleProcessLeader dev / gormtx.PGAdvisoryLeader prod — crash-safe via the advisory lock as lease) so exactly one replica pumps. Consumer commits the Kafka offset only after the handler+idempotency commit → no loss + exactly-once effect; NACK rewinds in-session (at-least-once).
  • OutboxCDCConsumer seam stays documented for an external WAL→queue project; the SDK ships no WAL engine.

Proven

Real Postgres, MySQL 8, and Redpanda (testcontainers, -race): write-only end-to-end, two-consumers-split-partitions, per-key ordering, exactly-once under redelivery, single-relay under contended PG advisory leader, drop-partition retention. CI guards PG/MySQL/Kafka didn't silently skip.

Clean core

No broker/Kafka/ORM dep in persistence/authz/grpcauthz/events; franz-go only in events/kafkabus; mysql/redpanda only in testdata. Scaffold + all existing suites green.