feat: Add database-backed event processing with Outbox pattern #56

effron · 2025-11-11T19:25:44Z

Summary

This PR introduces a proof of concept for an optional database-backed event processing system using the Outbox pattern as an alternative to ActiveJob-based delivery.

Configuration Example

# config/initializers/journaled.rb
Journaled.delivery_adapter = Journaled::Outbox::Adapter

Extract Kinesis client creation and batch sending logic into reusable components. Introduce a DeliveryAdapter interface to support alternative event delivery mechanisms while maintaining backward compatibility with the existing ActiveJob-based delivery. Changes: - Extract KinesisClientFactory for centralized Kinesis client creation - Extract KinesisBatchSender for shared batch sending logic - Introduce DeliveryAdapter base class defining adapter interface - Create ActiveJobAdapter wrapping existing ActiveJob delivery behavior - Refactor Writer to use delivery_adapter.deliver() instead of direct job enqueueing - Refactor Connection to delegate to delivery_adapter.transaction_connection - Update Engine to use adapter.validate_configuration! instead of detect_queue_adapter! - Update DeliveryJob to use KinesisClientFactory All existing behavior is preserved through the ActiveJobAdapter, which is the default adapter. No breaking changes for existing users.

Add PostgreSQL gem as a development dependency and configure CI to run tests against both SQLite and PostgreSQL databases. This ensures the database-backed event processing works across both database systems. Changes: - Add PostgreSQL service container to GitHub Actions workflow - Configure database matrix to run tests against sqlite3 and postgresql - Add pg gem to Gemfile and Rails-specific gemfiles - Update test database configuration with PostgreSQL connection details

app/models/journaled/outbox/event.rb

jmileham · 2025-11-11T19:56:47Z

README.md

+Journaled.worker_max_attempts = 3        # Retries before moving to DLQ
+```
+
+**Note:** When using the Outbox adapter, you do **not** need to configure an ActiveJob queue adapter (skip step 1 of Installation). The Outbox adapter uses the `journaled_events` table for event storage and its own worker daemons for processing, making it independent of ActiveJob. Transactional batching still works seamlessly with the Outbox adapter.


This is better than enqueuing a "drain the outbox" job we could enqueue with a little bit of a debounce whenever journaled entries are flowing in and not have to manage a daemon?

Fair question! I think we can further decouple the storing of events in an outbox table and the way we process the events to allow both implementations. For now, I'd like to experiment with managing a separate daemon to get a sense of how difficult that is vs keeping the processing in the job queue.

jmileham · 2025-11-11T20:02:33Z

README.md

+```ruby
+# Find failed events
+Journaled::Outbox::DeadLetter.where(stream_name: 'my_stream')
+Journaled::Outbox::DeadLetter.failed_since(1.hour.ago)


I think it might be attractive to block whole shards on failure to enable in order delivery rather than have a deadletter concept.

effron · 2025-11-11T20:28:21Z

Some changes i'll be implementing after talking with @smudge :

No locked at or locked by, do everything in one txn using row level locks
uuidv7 as the event id, use that as the table PK, no more index on created_at, order by id ASC
add failed_at, index failed_at, remove DLQ
Batch updates (delete_all, update_all etc)
Set fill factor to < 1 to encourage hot-updates
move incrementing of attempts to temporary errors instead of during pickup
emit events as part of work loop (maybe every 100x loops or something or time based, threaded)

lib/journaled/outbox/batch_processor.rb

lib/journaled/kinesis_batch_sender.rb

README.md

smudge

TAFN -- left some thoughts, but I think the main thing is ensuring that we don't fail events for reasons that have nothing to do with the actual event payload.

…nsert events

app/models/journaled/outbox/event.rb

Co-authored-by: Nathan G <nathan@ngriffith.com>

db/migrate/2_create_journaled_events.rb

lib/journaled/outbox/batch_processor.rb

lib/journaled/outbox/metric_emitter.rb

smudge

TAFN - nothing major, just leaving this here so I can get a ping for re-review.

…t instead of custom case stement, use fewer queries to emit metrics

smudge · 2025-11-17T20:37:07Z

lib/journaled/outbox/adapter.rb

+
+        records = events.map do |event|
+          # Exclude the application-level id - the database will generate its own using uuid_generate_v7()
+          event_data = event.journaled_attributes.except(:id)


Confirmed that we're injecting the true PK ID into the payload later & that tests cover that, but I'm wondering what else might even care about the id prior to record insertion -- maybe we can set id to nil or remove it from the Event API surface entirely if Journaled is configured to use the outbox adapter... 🤔

smudge · 2025-11-17T20:43:12Z

lib/journaled/outbox/worker.rb

+          break if shutdown_requested
+
+          # Only sleep if no events were processed to prevent excessive polling on empty table
+          sleep(Journaled.worker_poll_interval) if events_processed.zero?


lib/journaled/kinesis_batch_sender.rb

…ors as transient

smudge

domain LGTM && platform LGTM!

effron added 2 commits November 11, 2025 14:17

effron requested a review from a team as a code owner November 11, 2025 19:25

effron changed the title ~~Add database-backed event processing with Outbox pattern~~ POC: Add database-backed event processing with Outbox pattern Nov 11, 2025

effron force-pushed the effron/main/custom-delivery-adapter-joint-point branch from dabc0cc to e25b6c6 Compare November 11, 2025 19:28

medlefsen reviewed Nov 11, 2025

View reviewed changes

app/models/journaled/outbox/event.rb Outdated Show resolved Hide resolved

jmileham reviewed Nov 11, 2025

View reviewed changes

effron marked this pull request as draft November 11, 2025 20:28

effron force-pushed the effron/main/custom-delivery-adapter-joint-point branch 6 times, most recently from 9d0f399 to f2f42cd Compare November 12, 2025 19:26

effron requested a review from medlefsen November 12, 2025 19:26

effron force-pushed the effron/main/custom-delivery-adapter-joint-point branch 5 times, most recently from 63d0a8a to b28d747 Compare November 13, 2025 16:15

smudge reviewed Nov 13, 2025

View reviewed changes

lib/journaled/outbox/batch_processor.rb Outdated Show resolved Hide resolved

smudge reviewed Nov 13, 2025

View reviewed changes

lib/journaled/outbox/batch_processor.rb Outdated Show resolved Hide resolved

smudge reviewed Nov 13, 2025

View reviewed changes

lib/journaled/kinesis_batch_sender.rb Show resolved Hide resolved

smudge reviewed Nov 13, 2025

View reviewed changes

lib/journaled/kinesis_batch_sender.rb Outdated Show resolved Hide resolved

smudge reviewed Nov 13, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

smudge reviewed Nov 13, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

smudge requested changes Nov 13, 2025

View reviewed changes

effron force-pushed the effron/main/custom-delivery-adapter-joint-point branch from b28d747 to 77497e8 Compare November 13, 2025 21:26

effron force-pushed the effron/main/custom-delivery-adapter-joint-point branch 2 times, most recently from af3c103 to b64f68e Compare November 14, 2025 16:50

effron requested review from jmileham and smudge November 14, 2025 16:51

effron force-pushed the effron/main/custom-delivery-adapter-joint-point branch from b64f68e to 7b380de Compare November 14, 2025 16:58

simplify migration generator, add created_at to outbox events, bulk i…

1c4d40d

…nsert events

effron force-pushed the effron/main/custom-delivery-adapter-joint-point branch from 7b380de to 1c4d40d Compare November 14, 2025 20:52

effron added 3 commits November 14, 2025 16:40

move log out of trap handler

9fffcf1

only sleep if there were no events processed, update defaults

3b053cf

bump version to 6.2.0

659f740

effron changed the title ~~POC: Add database-backed event processing with Outbox pattern~~ feat: Add database-backed event processing with Outbox pattern Nov 17, 2025

effron marked this pull request as ready for review November 17, 2025 17:58

remove deployment instructions from readme

7e25ac2