Twitter's fork of Apache BookKeeper (will push changes upstream eventually)
Java C++ Shell M4 Protocol Buffer CMake Other
Latest commit 91c85ab Jan 12, 2017 Yiming Zang committed with sijie Add aggregated read/write queue size metric
Summary: Reduce metric cardinality

Reviewers: psu, jbull, fcuny, drusek, sijieg, mpatidar, lstewart

Reviewed By: sijieg

Subscribers: #rb_ds-messaging

JIRA Issues: PUBSUB-18945

Differential Revision:
Failed to load latest commit information.
bin Add missing configuration files to the package. Apr 29, 2015
bookkeeper-benchmark Bump version to 4.3.6-TWTTR-OSS-SNAPSHOT Dec 13, 2016
bookkeeper-server Add aggregated read/write queue size metric Mar 17, 2017
bookkeeper-stats-providers Allow unregistering gauges and removing subscopes in stats provider Dec 13, 2016
bookkeeper-stats Allow unregistering gauges and removing subscopes in stats provider Dec 13, 2016
compat-deps Bump version to 4.3.6-TWTTR-OSS-SNAPSHOT Dec 13, 2016
doc merge open source changes to latest (includes part2 (multiplexing) & … Dec 13, 2012
hedwig-client BOOKKEEPER-776: Flaky test BookieRecoveryTest (ivank via fpj) Dec 9, 2014
hedwig-protocol BookKeeper: Update JAR version to 4.2.6 Jul 12, 2013
src/assemble First commit for master Jul 23, 2012
.gitignore Port BOOKKEEPER-618: Better resolution of bookie address Jun 8, 2016
CHANGES.txt BookKeeper: Merge NettyReleaseBranch back to master Mar 19, 2015
LICENSE First commit for master Jul 23, 2012
NOTICE First commit for master Jul 23, 2012
OWNERS Improve and fix bookkeeper rereplicator Dec 13, 2016
README add notes on running bookkeeper tests Jun 19, 2014
formatter.xml First commit for master Jul 23, 2012


Build instructions for BookKeeper


* Unix System
* JDK 1.6
* Maven 3.0
* Autotools (if compiling native hedwig client)
* Internet connection for first build (to fetch all dependencies)

The BookKeeper project contains:

 - bookkeeper-server     (BookKeeper server and client)
 - bookkeeper-benchmark  (Benchmark suite for testing BookKeeper performance)
 - hedwig-protocol       (Hedwig network protocol)
 - hedwig-client         (Hedwig client library)
 - hedwig-server         (Hedwig server)

BookKeeper is a system to reliably log streams of records. It is designed to 
store  write ahead logs, such as those found in database or database like 

Hedwig is a publish-subscribe system designed to carry large amounts of data 
across the internet in a guaranteed-delivery fashion from those who produce 
it (publishers) to those who are interested in it (subscribers).

How do I build?

 BookKeeper uses maven as its build system. To build, run "mvn package" from the 
 top-level directory, or from within any of the submodules.

 Useful maven commands are:

 * Clean                     : mvn clean
 * Compile                   : mvn compile
 * Run tests                 : mvn test 
 * Create JAR                : mvn package
 * Run findbugs              : mvn compile findbugs:findbugs
 * Install JAR in M2 cache   : mvn install
 * Deploy JAR to Maven repo  : mvn deploy
 * Run Rat                   : mvn apache-rat:check
 * Build javadocs            : mvn compile javadoc:aggregate
 * Build distribution        : mvn package assembly:single

 Tests options:

 * Use -DskipTests to skip tests when running the following Maven goals:
    'package',  'install', 'deploy' or 'verify'
 * -Dtest.exclude=<TESTCLASSNAME>
 * -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java


BookKeeper uses maven-shade-plugin to build shade packages for old versions for
backward compatibility testing. This shade plugin is only able to run at 'package'
phase. So there are two ways to run bookkeeper tests:

 * Run 'mvn clean package' under bookkeeper root directory
 * Run 'mvn clean test' under bookkeeper/bookkeeper-server directory after you run
   'mvn clean install -DskipTests' under bookkeeper directory

How do I run the services?

 Running a Hedwig service requires a running BookKeeper service, which in turn
 requires a running ZooKeeper service (see To 
 start a bookkeeper service quickly for testing, run:
   $ bookkeeper-server/bin/bookkeeper localbookie 10

 This will start a standalone, ZooKeeper instance and 10 BookKeeper bookies.
 Note that this is only useful for testing. Data is not persisted between runs.

 To start a real BookKeeper service, you must set up a ZooKeeper instance and
 run start a bookie on several machines. Modify bookkeeper-server/conf/bk_server.conf
 to point to your ZooKeeper instance. To start a bookie run:

   $ bookkeeper-server/bin/bookkeeper bookie

 Once you have at least 3 bookies runnings, you can start some Hedwig hubs. A 
 hub is a machines which is responsible for a set of topics in the pubsub 
 system. The service automatically distributes the topics among the hubs. 
 To start a hedwig hub:

   $ hedwig-server/bin/hedwig server

 You can get more help on using these commands by running:

   $ bookkeeper-server/bin/bookkeeper help
   $ hedwig-server/bin/hedwig help