A high performant Erlang client for beanstalkd work queue
Switch branches/tags
Clone or download
Silviu Caragea
Silviu Caragea Fix pool restart
Latest commit 87e1c89 Mar 14, 2017
Permalink
Failed to load latest commit information.
include Add API for starting and stopping pools Mar 12, 2017
src Fix pool restart Mar 14, 2017
test Switch from poolboy to erlpool Mar 12, 2017
.gitignore Add gitignore Aug 17, 2016
Makefile Add support for unit testings Jan 17, 2017
README.md Update readme Mar 12, 2017
rebar.config Fix pool restart Mar 14, 2017

README.md

ebeanstalkd

A high performant Erlang client for beanstalkd work queue

Features

  • Automatically detects when server is down and is trying periodically to reconnect
  • Can provide notifications to another process when connection goes down/up
  • Support for connection pool using erlpool
  • Very big throughput in messages per second that can be sent over one single connection (achieved over 50K/second on a single connection)
  • Protection against OOM
  • The library is working with binary strings but accepts also lists.

Quick start

Getting all deps and compile:

rebar get-deps
rebar compile

The any is very simple. For example the following piece of code is creating a connection, put a job, then reserve, delete it and close the connection:

{ok, Pid} = ebeanstalkd:connect().
{inserted, Id} = ebeanstalkd:put(Pid, <<"job body">>).
{reserved, Id, <<"job body">>} = ebeanstalkd:reserve(Pid).
{deleted} = ebeanstalkd:delete(Pid, Id).
ok = ebeanstalkd:close(Pid).

The connection can also accept arguments by using ebeanstalkd:connect/1 method with a list with the following options:

  • host - Beanstalkd server host. Default {127,0,0,1} example: {host, {127,0,0,1}
  • port - Beanstalkd server port. Default 11300 example: {port, 11300}
  • timeout - Connection timeout in milliseconds. Default 5000 example: {timeout, 5000}
  • tube - Specify what tube(s) to use or watch. Example: {tube, {watch, [<<"tube1">>, <<"tube2">>]}} or {tube, {use, <<"tube1">>}}
  • reconnect_interval - After how many milliseconds should try again to reconnect. Default 5000 example: {reconnect_interval, 5000}
  • monitor - The process pid that should return the connection up and connection down messages. Default is undefined example: {monitor, self()}

In case the monitor parameter is set the specified pid will receive the following messages:

  • {connection_status, {up, ConnectionPid}} - Triggered when connection is ready to be used
  • {connection_status, {down, ConnectionPid}} - Triggered when connection is broken

Usage of connection pool

You can specify in the sys.config the pool specs in the following format:

[
    {ebeanstalkd, [
        {pools, [
            {bk_pool,[
                {size, 50},
                {host, {127,0,0,1}},
                {port, 11300},
                {timeout, 5000},
                {tube, {use, <<"poolname">>}}
            ]}
        ]}
    ]}
].

The API for pool is the same for plain connections but instead of using a pid as the first parameter you are using the name of the pool (atom):

application:ensure_all_started(ebeanstalkd).
{inserted, Id} = ebeanstalkd:put(pool_one, <<"hello">>).
{reserved, ID, <<"hello">>} = ebeanstalkd:reserve(pool_one).
{deleted} = ebeanstalkd:delete(pool_one, ID).

All supported commands from the protocol are exported in ebeanstalkd module.

Performance testing

From the test folder you can use the load_test:run/4 method to run benchmarks. Make sure you define a pool called bk_pool in your sys.config or use the one from the same location.

The load_test/run(Profiling, ClientsNr, ReqNr, PayloadLength) can take the following values:

  • Method - Specify if you want to profile the erlang processes during the test.
  • ClientsNr - how many processes should spawn to send requests
  • ReqNr - how many requests should send during the test.
  • PayloadLength - what length will have the message payload sent to the server

Example:

load_test:run(false, 10, 100000, 32*1024).

Notes

  • The ebeanstalkd:put_in_tube2 works only with my server fork currently. This command put_in_tube is not supported by the official protocol. A lot of clients are sending a use command and then a put but this is decreasing performances. Also ebeanstalkd offers this functionality in this way using ebeanstalkd:put_in_tube