Skip to content
No description or website provided.
Erlang Other
  1. Erlang 99.6%
  2. Other 0.4%
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci test and build docker image on otp-21 (#148) Aug 15, 2018
cluster use port 5588 by default (#145) May 29, 2018
config use port 5588 by default (#145) May 29, 2018
docs
helm update chain_name to accept string for use of os vars Feb 15, 2017
include upgrade to support OTP-21 (#147) Aug 10, 2018
src test and build docker image on otp-21 (#148) Aug 15, 2018
test test and build docker image on otp-21 (#148) Aug 15, 2018
.gcloudignore update cloudbuild to automatically use git tags as version (#126) Jan 18, 2018
.gitignore
Dockerfile
LICENSE add apache-2 license Nov 3, 2016
README.md
cloudbuild.yaml update cloudbuild to automatically use git tags as version (#126) Jan 18, 2018
rebar.config test and build docker image on otp-21 (#148) Aug 15, 2018
rebar.lock test and build docker image on otp-21 (#148) Aug 15, 2018

README.md

vonnegut

CircleCI

codecov

Vonnegut is a append-only log that follows the file format and API of Kafka 1.0. The server can be run standalone, with 1 or more chains each with 1 or more replicas, or as part of another Erlang release which can talk to it directly.

Each chain is responsible for a range of the topic space. A read or write to a topic requires finding what chain the topic belongs to and then making a request to the head, in the case of a write, or the tail, in the case of a read.

Configuration

Server

A node in a chain can discover other nodes within the chain through DNS SRV record queries. The replicas configuration tells vonnegut node how many other nodes it needs to connect to to form the required chain length to ack writes.

{vonnegut, [{chain, [{name, "chain-1"},
                     {discovery, {srv, "chain-1.service.cluster.local"}},
                     {replicas, "2"},
                     {port, 5555}]}
            ]}

Client

Clients start a pool of connections to the head and tail of each chain. Chains are found through DNS queries against endpoints:

{vonnegut, [{client, [{endpoints, [{"chain-1.service.cluster.local", 5555}]}]}]}

Erlang Interface

A local interface can be used to create, read and write topics.

$ rebar3 shell
1> vg:create_topic(<<"test_topic">>).
2> vg:write(<<"test_topic">>, [<<"some log message">>, <<"more log message">>]).
3> vg:fetch(<<"test_topic">>).
{ok,#{high_water_mark => 1,partition => 0,
      record_batches =>
          [#{headers => [],key => <<>>,offset => 1,sequence_number => 1,
             timestamp => 1517613646458,value => <<"more log message">>},
           #{headers => [],key => <<>>,offset => 0,sequence_number => 0,
             timestamp => 1517613646458,
             value => <<"some log message">>}]}}

By default index and log files will be written to ./data:

$ ls data/test_topic-0/
00000000000000000000.index  00000000000000000000.log

Kafkaesque Client

$ rebar3 shell
1> vg_client_pool:start().
ok
2> vg_client:produce(<<"my-topic-2">>, [<<"message 1">>, <<"message 2">>]).
{ok,1}
3> vg_client:fetch(<<"my-topic-2">>).
{ok,#{<<"test_topic-2">> =>
          #{0 =>
                #{error_code => 0,high_water_mark => 1,
                  record_batches =>
                      [#{headers => [],key => <<>>,offset => 1,
                         sequence_number => 1,timestamp => 1517616861441,
                         value => <<"message 2">>},
                       #{headers => [],key => <<>>,offset => 0,
                         sequence_number => 0,timestamp => 1517616861441,
                         value => <<"message 1">>}],
                  record_batches_size => 95}}}}

Running Tests

The tests require opening thousands of files and so may require increasing the limit per process on your system with:

$ ulimit -n 63536

Tests also require a nodename:

$ rebar3 ct
You can’t perform that action at this time.