Active Record Streams

A small library to stream ActiveRecord's create/update/delete events to AWS SNS topics, Kinesis streams or HTTP listeners.

# config/initializers/active_record_streams.rb

require 'active_record_streams'

ActiveRecordStreams.configure do |config|
  # Listen to all tables and publish events to the specified SNS topic
  config.streams << ActiveRecordStreams::Publishers::SnsStream.new(
    topic_arn: 'arn:aws:sns:...'
  )
  
  # Publish only events coming from `users` table to the specified SNS topic
  config.streams << ActiveRecordStreams::Publishers::SnsStream.new(
    table_name: 'users',
    topic_arn: 'arn:aws:sns:...'
  )
  
  # Publish events for all the tables but `updates`
  config.streams << ActiveRecordStreams::Publishers::SnsStream.new(
    ignored_tables: %w[updates],
    topic_arn: 'arn:aws:sns:...'
  )
end

Error handling

It might happen that the message delivery fails. In such case you might want to retry sending the message or even use a background processor like Sidekiq to perform retires in automated way.

Every stream has an error_handler option which accepts lambda/proc with error_handler :: (Stream, TableName, Message, Error) -> * signature.

NOTE that your consumer has to take care of duplicated messages. Each message contains EventID which may help to identify duplications.

# config/initializers/active_record_streams.rb

require 'active_record_streams'

class SampleHttpReSender
  include Sidekiq::Worker
  
  def perform(table_name, message_json)
    message = ActiveRecordStreams::Message.from_json(message_json)
    ActiveRecordStreams.config.streams.each do |stream|
      # Use `handle_error: false` to raise exceptions instead of
      # calling `error_handler`, this will let he Sidekiq to
      # perform retries on it's own 
      stream.publish(table_name, message, handle_error: false)
    end
  end
end

ActiveRecordStreams.configure do |config|
  config.streams << ActiveRecordStreams::Publishers::HttpStream.new(
    url: 'https://posteventshere.test',
    error_handler: lambda do |stream, table_name, message, error|
      # Do whatever you want here, you may also try to start a new
      # thread or re-try publishing directly to stream using
      # stream.publish(table_name, message)
 
      # Try to schedule re-publishing with Sidekiq 
      SampleHttpReSender.perform_async(table_name, message.json)
    end
  )
end

Supported targets