Skip to content
DataBuffer provides an efficient way to maintain persistable lists of data.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


Build Status DataBuffer Version

DataBuffer is an efficient way to maintain a local buffer list associated with a given key that can later be flushed to persistent storage. In fast moving systems, this provides a scalable way keep track of data without putting heavy loads on a database.


The package can be installed by adding data_buffer to your list of dependencies in mix.exs:

def deps do
    {:data_buffer, "~> 0.1"}


Please see HexDocs for additional documentation.

Getting Started

We can start off by creating our buffer. This is simply a module that uses DataBuffer and implements the handle_flush/2 as well as the optional handle_error/2 callback.

defmodule Buffer do
  use DataBuffer

  @impl DataBuffer
  def handle_flush(key, data) do
    # do database stuff...

    # we must return an :ok atom

We then must add the buffer to our supervision tree.

children = [

There are some configruable options available for our buffers. You can read more about them here. These options can be passed when creating our buffer.

use DataBuffer, partitions: 4, interval: 10_000

With our buffer started, we can now insert data. A key can be any valid term.

Buffer.insert("mykey", "myval1") # Adds myval1 to mykey
Buffer.insert("mykey", "myval2") # Adds myval2 to mykey

Our buffer will be flushed using our handle_flush/2 callback after the default interval period. Dead counters are automatically removed.

def handle_flush(key, data) do
  IO.inspect(key) # returns "mykey"
  IO.inspect(data) # returns ["myval1", myval2"]

If the handle_flush/2 returns an invalid value or raises an exception, the flush operation will be retried after the configurable :retry_delay up to the maximum :retry_max. If the operation still fails, the handle_error/2 callback will be called. It is then left up to the developer how to handle the data.

def handle_error(key, data) do
  # Put the data back into the buffer...
  # Or put the data to local disk...
You can’t perform that action at this time.