Skip to content
Bloomberg's distributed RDBMS
Branch: master
Clone or download
adizaimi Limit when to record test perf under callgrind
When running tests under callgrind limit recording to the part
of the test after setup/start and before verify/unsetup.
Latest commit 331b3c5 Apr 23, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bbinc Fix pglogs_asof_thread race with clean exit (#1512) Apr 22, 2019
bdb
berkdb Debug berkeley lockers upon lock/unlock (#1484) Apr 8, 2019
cdb2api Fix issue of low load on first node (#1498) Apr 9, 2019
cdb2jdbc Read configuration files only once. (#1232) Nov 28, 2018
cmake Add Tcl bindings... (#911) Jun 14, 2018
comdb2rle fix some compiler warnings pt1 (#1112) Sep 13, 2018
contrib
crc32c Treat warnings as errors (#1343) Jan 3, 2019
csc2 CSC2: Fix csc2 parser to accept named constraint and modifiers togeth… Mar 1, 2019
csc2files Initial import. May 17, 2017
cson
datetime datetime: Fix drifting timezone (#1461) Mar 4, 2019
db Authorizer callback for use with SQLite, based on PR #1305 (v2) (#1517) Apr 23, 2019
dfp cleanup more warnings (#1296) Dec 13, 2018
dlmalloc Treat warnings as errors (#1343) Jan 3, 2019
docs Authorizer callback for use with SQLite, based on PR #1305 (v2) (#1517) Apr 23, 2019
linearizable/ctest Port linearizable tests to testsuite (#328) Nov 1, 2017
lua Authorizer callback for use with SQLite, based on PR #1305 (v2) (#1517) Apr 23, 2019
mem Compiler warnings (#1320) Dec 14, 2018
net Init structs with zeros rather than bzero content (#1455) Mar 14, 2019
pkg
plugins Add ability to whitelist databases allowed to connect to a given db (#… Mar 22, 2019
protobuf
schemachange schemachange: Fix the schema change resumption check for comdb2sc (#1500 Apr 10, 2019
sockpool Init structs with zeros rather than bzero content (#1455) Mar 14, 2019
sqlite Authorizer callback for use with SQLite, based on PR #1305 (v2) (#1517) Apr 23, 2019
tcl
tests Limit when to record test perf under callgrind Apr 23, 2019
tools cdb2sql: Add "restore" and "fuzz" tiers. Apr 11, 2019
util Authorizer callback for use with SQLite, based on PR #1305 (v2) (#1517) Apr 23, 2019
.gitignore Generate and run tests based on options in testopts file (#1094) Sep 5, 2018
CMakeLists.txt cdb2api: Configuring DNS suffix at compile time. (#1438) Apr 5, 2019
CONTRIBUTING.md Add a more visible "contributing" doc. (#89) Jun 5, 2017
INTERNAL_CONTRIBUTORS.md Updating SQLite to the 3.26 release (#953) Dec 7, 2018
LICENSE
PULL_REQUEST_TEMPLATE.md Create PULL_REQUEST_TEMPLATE.md Jan 5, 2018
README.md Update README.md (#1153) Oct 2, 2018

README.md

Overview

Comdb2 is a clustered RDBMS built on Optimistic Concurrency Control techniques. It provides multiple isolation levels, including Snapshot and Serializable Isolation. Read/Write transactions run on any node, with the client library transparently negotiating connections to lowest cost (latency) node which is available. The client library provides transparent reconnect.

Work on Comdb2 was started at Bloomberg LP in 2004 and it has been under heavy development since. More information about the architecture of the project can be found in our VLDB 2016 paper and for more information on usage please look in the Docs.

Build

Documentation

Comdb2 documentation is included in the docs directory. It can be hosted locally with jekyll by running jekyll serve from the docs directory.

Contributing

Please refer to our contribution guide for instructions. We welcome code and idea contributions.

Quick Start

On every machine in the cluster:

  1. Make sure all machines in the cluster can talk to each other via ssh.
    Copy keys around if needed.

  2. Install prerequisites:

    Ubuntu 16.04, 16.10, 17.04, Windows Subsystem for Linux (WSL)

    sudo apt-get install -y \
        bison \
        build-essential \
        cmake \
        flex \
        protobuf-c-compiler \
        liblz4-dev \
        libncurses5-dev \
        libprotobuf-c-dev \
        libreadline-dev \
        libssl-dev \
        libsqlite3-dev \
        libunwind-dev \
        tcl \
        uuid-dev \
        zlib1g-dev
    

    CentOS 7

    sudo yum install -y gcc gcc-c++ cmake3 protobuf-c libunwind libunwind-devel protobuf-c-devel byacc flex openssl openssl-devel openssl-libs readline-devel sqlite sqlite-devel libuuid libuuid-devel zlib-devel zlib lz4-devel gawk tcl epel-release lz4 rpm-build which
    

    macOS High Sierra (experimental)

    Install Xcode and Homebrew. Then install required libraries:

    brew install cmake lz4 openssl protobuf-c readline
    

    To run tests, install following:

    brew install coreutils bash
    export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
    

    It is recommended to increase the open file limits, at least in the sessions which start pmux and server.

  3. Build and Install Comdb2:

    mkdir build && cd build && cmake .. && make && sudo make install
    
  4. Add /opt/bb/bin to your PATH

    export PATH=$PATH:/opt/bb/bin
    
  5. Start pmux:

    pmux -n
    

    Note: on WSL this needs to be run elevated, such as: sudo /opt/bb/bin/pmux -n

  6. (optional) Comdb2 nodes identify each other by their hostnames. If the hostname of each node isn't resolvable from other nodes, we should tell Comdb2 the full domain name to use for the current node. Most setups won't have this issue.

    Tell comdb2 our FQDN.

    vi /opt/bb/etc/cdb2/config/comdb2.d/hostname.lrl
    add current machine's name, e.g.
    hostname machine-1.comdb2.example.com
  7. On one machine (say machine-1), create a database - this example creates a database called testdb stored in ~/db.

    comdb2 --create --dir ~/db testdb
    

    Note: the --dir PATH parameter is optional, and if it is omitted comdb2 uses a default root of /opt/bb/var/cdb2/ for creating a database directory to contain the database files, which is named as per the database name parameter; hence in this case /opt/bb/var/cdb2/testdb.
    The default root will have to be created explicitly with the desired permissions before invoking comdb2 --create for a database.
    In this quick start, we use the home directory to avoid obfuscating the key steps of the process.

  8. Configure the nodes in the cluster:

    vi ~/db/testdb.lrl
    add
    cluster nodes machine-1.comdb2.example.com machine-2.comdb2.example.com
    
  9. On other nodes, copy the database over:

    copycomdb2 mptest-1.comdb2.example.com:${HOME}/db/testdb.lrl
    
  10. On all nodes, start the database.

    comdb2 --lrl ~/db/testdb.lrl testdb
    

    All nodes will say 'I AM READY.' when ready.

    Note: the log dir comdb2 uses by default is /opt/bb/var/log/cdb2/ If this directory does not have permissions allowing the user to create file, there will be diagnostics output such as:

    [ERROR] error opening '/opt/bb/var/log/cdb2/testdb.longreqs' for logging: 13 Permission denied

    This condition will not impact operation of the database for the purposes of this quick start.
     

  11. On any node, start using the database. You don't have any tables yet. You can add them with cdb2sql Example -

    cdb2sql testdb local 'CREATE TABLE t1 {
         schema {
             int a
         }
    }'

    Database can be queried/updated with cdb2sql:

    cdb2sql testdb local 'insert into t1(a) values(1)'
    (rows inserted=1)
    cdb2sql testdb local 'select * from t1'
    (a=1)

Comdb2 Directory Contents

Directory Description
bbinc/ Header & Generic include files
bdb/ Table layer
berkdb/ Btrees layer
cdb2api/ Client code
cdb2jdbc/ JDBC driver
cmake/ cmake configuration files
comdb2rle/ Run length encoding
contrib/ Misc useful programs that aren't part of core Comdb2
crc32c/ Checksum component
csc2/ csc2 processing
csc2files/ csc2 config files
cson/ JSON library
datetime/ Datetime component
db/ Types layer and overall glue
dfp/ Decimal number component
dlmalloc/ Local malloc version
docs/ Documentation
lua/ All things pertaining to lua VM used for stored procedures
mem/ Memory accounting subsystem
net/ Network component
pkg/ deb and rpm packaging rules
plugin/ Plugin subsystem
protobuf/ API to communicate with the server
schemachange/ Code for table create/alter/truncate/etc
sockpool/ sockpool related files
sqlite/ Sqlite VM SQL engine
tcl/ Tcl language bindings
tests/ Comdb2 test suite
tools/ Tools that are part of Comdb2 core
util/ Useful generic modules
You can’t perform that action at this time.