Skip to content

Fix race conditions when calculating previous_index#82

Merged
gsmetal merged 4 commits intomasterfrom
fix/race-conditions-in-previous-index
Mar 23, 2026
Merged

Fix race conditions when calculating previous_index#82
gsmetal merged 4 commits intomasterfrom
fix/race-conditions-in-previous-index

Conversation

@gsmetal
Copy link
Copy Markdown
Member

@gsmetal gsmetal commented Mar 23, 2026

Problem

When multiple model transactions run concurrently, they INSERT into artery_messages and receive sequential auto-increment IDs. However, transactions can COMMIT in arbitrary order. Since previous_index is computed in after_commit (and only sees committed rows), a later-committed message with a lower ID can end up with the same previous_index as an already-committed message with a higher ID. This causes the consumer to silently drop the out-of-order message, leading to data loss.

Example: Two concurrent transactions produce messages with IDs 100 and 101. Transaction for ID 101 commits first. Both messages end up with previous_index = 99. If the consumer processes 101 first, message 100 is skipped as "already handled".

Approach

  1. New artery_model_infos table — stores latest_index per model, used both as a lock target (SELECT ... FOR UPDATE) and as a fast lookup for latest_index / previous_index.

  2. Deferred message creation via before_commitafter_create / after_update / after_destroy callbacks now queue pending notifications instead of creating artery messages immediately. A before_commit callback flushes the queue, minimizing the lock window to just INSERT + COMMIT.

  3. around_create :lock_on_model on Artery::ActiveRecord::Message — acquires a FOR UPDATE lock on the model's row in artery_model_infos, caches previous_index from latest_index, creates the message, then updates latest_index. This serializes message creation per model, preventing the race.

  4. Configurable model_info_class — follows the same pattern as message_class and subscription_info_class.

Deployment

  • Run migration first (creates and seeds artery_model_infos from existing artery_messages data)
  • Rolling deploy is safe: old processes ignore the new table, new processes use it. latest_index self-corrects once all processes are on the new code
  • New models are handled automatically via acquire_lock! (creates the lock row on first use)

@gsmetal gsmetal requested a review from IgRich March 23, 2026 09:24
@gsmetal gsmetal self-assigned this Mar 23, 2026
@gsmetal gsmetal marked this pull request as ready for review March 23, 2026 13:17
* Refactor logging with ActiveSupport::Instrumentation
@gsmetal gsmetal merged commit a54d14d into master Mar 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants