Skip to content

Separate publisher process#85

Merged
gsmetal merged 3 commits intomasterfrom
feat/separate-publisher-process
Mar 26, 2026
Merged

Separate publisher process#85
gsmetal merged 3 commits intomasterfrom
feat/separate-publisher-process

Conversation

@gsmetal
Copy link
Copy Markdown
Member

@gsmetal gsmetal commented Mar 26, 2026

Problem

Concurrent transactions produce artery messages with sequential IDs, but after_commit callbacks fire in unpredictable order. This causes incorrect _previous_index values — consumers see a broken chain and drop valid messages. The previous fix (#82) used FOR UPDATE locks on artery_model_infos to serialize publishing, which worked correctly but caused high lock contention under high concurrency.

Solution

Replace inline publishing with an outbox pattern:

  • Messages are inserted into artery_messages directly in model callbacks (after_create/after_update/after_destroy), inside the model transaction. No locks needed.
  • A separate Publisher process (bundle exec artery-publisher) polls artery_messages and publishes to NATS in strict id order, guaranteeing a correct _previous_index chain.
  • Publisher tracks progress via last_published_id on artery_model_infos, protected by FOR UPDATE (only between publisher instances, not on the hot path).
  • Each model gets its own thread in a concurrent-ruby thread pool.

Configuration

Two publishing modes controlled by inline_publish (default: true):

  • Inline — publishes from after_commit, no extra process needed. Suitable for development.
  • Publisher — set config.inline_publish = false and run artery-publisher. Recommended for production.

@gsmetal gsmetal self-assigned this Mar 26, 2026
@gsmetal gsmetal marked this pull request as ready for review March 26, 2026 13:50
@gsmetal gsmetal merged commit da147bf into master Mar 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant