💾 KVS: Abstract Term Database
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
include Fix ITER Oct 2, 2017
man kvs Sep 4, 2018
.gitignore fix typo May 30, 2013
CNAME CNAME Sep 15, 2017
LICENSE clean repo May 13, 2014
README.md traverse API via fold Apr 26, 2017
index.html nav Sep 4, 2018
mix.exs merge Mar 24, 2015
rebar.config kvs css Sep 4, 2018


KVS: Erlang Abstract Term Database

Online Presentation: http://slid.es/maximsokhatsky/kvs


  • Polymorphic Tuples
  • Managing Linked-Lists
  • Various Backends Support: Mnesia, Riak, KAI, Redis, MongoDB
  • Sequential Consistency via Feed Server
  • Basic Schema for Social Sites and Accounting
  • Extendable Schema
  • Supports Secondary Indexes for KAI, Mnesia, Riak and MongoDB
  • Change Backends on-the-fly
  • Supports Multiple backends at the same time
  • Xen Ready


In rebar.config:

{kvs, ".*", {git, "git://github.com/synrc/kvs", "HEAD"}}

Redis also need to add:

{eredis, ".*", {git, "git://github.com/wooga/eredis", {tag, "v1.0.6"} }}

MongoDB also need to add:

{mongodb, ".*", {git, "git://github.com/comtihon/mongodb-erlang", {tag, "master"} }},
{poolboy, ".*", {git, "git://github.com/devinus/poolboy", {tag, "master"} }}

In the above example poolboy is optional. MongoDB and Poolboy config example:

{kvs, [
  {mongo, [
    {connection, [{database,<<"kvs">>}]},
    {pool, [{size,10},{max_overflow,20}]}

To disable poolboy exclude {pool, ...} from your sys.config. More information on the configuring MongoDB and Poolboy can be found here: https://github.com/comtihon/mongodb-erlang, https://github.com/devinus/poolboy.


We have built with KVS a number of applications and came up with schema samples. We grouped schemas by three category. KVS hides database access behind backend drivers and provides high-level rich API to stored and extend following data:

  • Core — Acl, Users, Subscriptions, Feeds, Entries, Comments
  • Banking — Account, Customer, Transaction, Item, Currency, Program, Card, Cashback
  • Social — Group, Meeting, Payment, Product, Purchase


This Framework provides also a feed application for sequential consistency and cr application for chain replication database on top of kvs. All write requests with given object key will be handled by single processes so you may not worry about concurrent changes of user feed tops.

All write operations that are made to data with secondary indexes, i.e. not like linked lists could be potentially handled without feed_server. But some KV storages are not supporting secondary indexes so use these backends carefully.

Store Backends

Currently kvs includes following store backends:

  • Mnesia
  • Riak
  • KAI
  • Filesystem
  • Redis
  • MongoDB


First of all you need to tune your backend in the kvs application:

{kvs, [{dba,store_mnesia}]},

Try to check it:

1> kvs:config(dba).

2> kvs:version().
{version,"KVS KAI PURE XEN"}

Create database for single node:

3> kvs:join().
[kvs] Mnesia Init

You can also create database by joining to existing cluster:

3> kvs:join('kvs@synrc.com').

In that case you don't need to initialize the database to check table packages included into the schema:

4> kvs:dir().


Try to add some data:

1> rr(kvs_user).
2> kvs:put(#user{id="maxim@synrc.com"}).
3> kvs:get(user,"maxim@synrc.com").
#user{id = "maxim@synrc.com",container = feed,...}
4> kvs:put(#user{id="doxtop@synrc.com"}).
5> length(kvs:all(user)).

Polymorphic Records

The data in KVS represented as plain Erlang records. The first element of the tuple as usual indicates the name of bucket. And the second element usually corresponds to the index key field. Additional secondary indexes could be applied for stores that supports 2i, e.g. kai, mnesia, riak, mongodb.


All record could be chained into the double-linked lists in the database. So you can inherit from the ITERATOR record just like that:

-record(iterator, {id,version,

The layout of iterators are following:

> lists:zip(lists:seq(1,length((kvs:table(operation))#table.fields)),


This means your table will support add/remove linked list operations to lists.

1> kvs:add(#user{id="mes@ua.fm"}).
2> kvs:add(#user{id="dox@ua.fm"}).

Read the chain (undefined means all)

3> kvs:entries(kvs:get(feed, user), user, undefined).

Read flat values by all keys from table:

4> kvs:all(user).

Table can be traversed from any element in any directions with fold/6 processing function.

Function entries/3 which read the elements from top of container can be implemented by reversing the fold/6 result list:

4> {ok,F2} = kvs:get(feed,"f").

5> lists:reverse(kvs:fold(fun(A,Acc)->[A|Acc] end, [], entry, element(#container.top, F2),2,#iterator.prev)).
[#entry{id = 2,container = feed,feed_id = "f",prev = 1, next = [],...},
 #entry{id = 1,container = feed,feed_id = "f",prev = [], next = 2,...}]


If you are using iterators records this automatically means you are using containers. Containers are just boxes for storing top/heads of the linked lists. Here is layout of containers:

-record(container, {id,top,count}).
> lists:zip(lists:seq(1,length((kvs:table(feed))#table.fields)),

Extending Schema

Usually you need only specify custom mnesia indexes and tables tuning. Riak, KAI and Redis backends don't need it. Group you table into table packages represented as modules with handle_notice API.


metainfo() -> 
        #table{ name=entry,container=feed,fields=record_info(fields,entry),
                keys=[entry_id,author_id]} ]}.

And plug it into schema config:

{kvs, {schema,[kvs_user,kvs_acl,kvs_feed,kvs_subscription]}},

And on database init

1> kvs:join().

It will create your custom schema.

Using KVS in real applications

Besides using KVS in production in a number of applications we have built on top of KVS several products. The first product is Chain Replication Database wit XA protocol. And second is social Feed Server for web shops and social sites.

Chain Replication Database

The kvs semantic is totally compatible with XA protocol. Adding the object with PUT means only putting to database while ADD operations provides linking to the chain's container. Also linking operation LINK is provided separately.

dispatch({prepare,_,_,Tx}, #state{})  ->
    kvs:info(?MODULE,"KVS PUT ~p:~p~n",[element(1,Tx),element(2,Tx)]),

dispatch({commit,_,_,Tx}, #state{})  ->
    kvs:info(?MODULE,"KVS LINK ~p:~p~n",[element(1,Tx),element(2,Tx)]),

dispatch({rollback,_,_,Tx}, #state{})  ->
    kvs:info(?MODULE,"KVS REMOVE ~p:~p~n",[element(1,Tx),element(2,Tx)]),

See: https://github.com/spawnproc/cr

Feeds Server

Here is Consumer behavior handlers of KVS FEEDS supervised processes

handle_notice(  [kvs_feed,user,Owner,entry,Eid,add],
                #state{feeds=Feeds}) ->

                case lists:keyfind(Fid,2, S#state.feeds) of
                    false -> skip;
                    {_,_} -> add_entry(Eid,Fid,Entry) end,
                {noreply, S};

handle_notice(  [kvs_feed,user,Owner,entry,{Eid,FeedName},edit],
                #state{feeds=Feeds}) ->

                case lists:keyfind(FeedName,1,Feeds) of
                    false -> skip;
                    {_,Fid}-> update_entry(Eid,Fid,Entry) end,
                {noreply, S};

handle_notice(  [kvs_feed,user,Owner,entry,Eid,edit],
                #state{feeds=Feeds}) ->

                case lists:keyfind(Fid, 2, Feeds) of
                    false -> skip;
                    {_,_} -> update_entry(Eid,Fid,Entry) end,
                {noreply, S};

Here is the private implementation

add_entry(Eid,Fid,Entry) ->
    E = Entry#entry{id = {Eid, Fid}, entry_id = Eid, feeds=[comments]},
    Added = case kvs:add(E) of {error, Err} -> {error,Err}; {ok, En} -> En end,
    msg:notify([kvs_feed, entry, {Eid, Fid}, added], [Added]).

update_entry(Eid,Fid,Entry) -> ...

And that is how you can call it

kvs:notify([kvs_feed, user, "maxim@synrc.com", entry, Eid, add],

See: https://github.com/synrc/feeds


  • Maxim Sokhatsky
  • Andrii Zadorozhnii
  • Vladimir Kirillov
  • Alex Kalenuk
  • Sergey Polkovnikov
  • Andrey Martemyanov