Multicore in-memory storage engine
Switch branches/tags
Nothing to show
Clone or download
stephentu Merge pull request #15 from nathanielherman/masstree_up
update masstree. seems to give a small (1-2%) speedup on tpcc
Latest commit cc11ca1 Mar 23, 2015
Permalink
Failed to load latest commit information.
benchmarks Add a bidding benchmark Apr 11, 2014
config makefile: event counters are now a separate flags Aug 28, 2013
masstree @ 1150824 update masstree. seems to give a small (1-2%) speedup on tpcc Mar 23, 2015
new-benchmarks Call btree_choice in a couple places Oct 5, 2013
record Add a Masstree btree variant. make M=1 to get it. Oct 5, 2013
scripts WIP: checkpoint for porting persistence system Jul 22, 2013
third-party/lz4 lz4 context is also allocated on numa memory when possible Jul 31, 2013
.gitignore makefile: use auto dependencies from GCC Aug 19, 2013
.gitmodules Add a Masstree btree variant. make M=1 to get it. Oct 5, 2013
AUTHORS AUTHORS: ack eddie Sep 11, 2013
BUILD checkpoint: start of a numa aware memory allocator Apr 10, 2013
LICENSE LICENSE + AUTHORS Aug 12, 2013
Makefile Add a bidding benchmark Apr 11, 2014
README.md Update README.md Aug 6, 2014
allocator.cc #include nits Oct 6, 2013
allocator.h allocator: allow MADV_HUGEPAGE to be disabled Sep 8, 2013
amd64.h add nop_pause() to spinloops, for better perf Feb 8, 2013
base_txn_btree.h txn_btree: equip with rsearch helpers Oct 23, 2013
btree.cc mbtree: add a few test cases for reverse scans Oct 23, 2013
btree.h mbtree: add a few test cases for reverse scans Oct 23, 2013
btree_choice.h Add a Masstree btree variant. make M=1 to get it. Oct 5, 2013
btree_impl.h update masstree. seems to give a small (1-2%) speedup on tpcc Mar 23, 2015
circbuf.h WIP: still does not work, making progress though Jul 22, 2013
core.cc WIP: try to do a better job of assigning core ids to bench workers Aug 1, 2013
core.h counters: fix cache alignment issues Sep 12, 2013
counter.cc counters: fix cache alignment issues Sep 12, 2013
counter.h counters: fix cache alignment issues Sep 12, 2013
fileutils.h fix silly bugs Aug 13, 2013
imstring.h cleanup: get rid of util::noncopyable Aug 17, 2013
lockguard.h tuple: more fine-grained "dirty" bits for tuples Mar 19, 2013
log2.hh log2: import from xv6 Mar 25, 2013
macros.h Fix NDEBUG ALWAYS_ASSERT. Sep 8, 2013
marked_ptr.h holy shit this finally compiles Apr 30, 2013
masstree_btree.h update masstree. seems to give a small (1-2%) speedup on tpcc Mar 23, 2015
memory.cc print stacktrace on large allocation memory Mar 10, 2013
ndb_type_traits.h new-benchmark now compiles - still not tested May 6, 2013
ownership_checker.h replace pthread_* functions with std::* variants Oct 11, 2013
persist_test.cc remove unused Aug 13, 2013
prefetch.h #include nits Oct 6, 2013
pxqueue.h WIP: transition proto2 gc out of background Aug 7, 2013
rcu.cc Need one more guard, in RCU itself Oct 23, 2013
rcu.h Introduce rcu::dealloc_rcu. Oct 5, 2013
scopedperf.hh scopedperf: make buffer size larger + some class decl macros Mar 20, 2013
small_unordered_map.h counters: fix cache alignment issues Sep 12, 2013
small_vector.h ndb_type_traits: try to avoid calling destructors in containers Mar 25, 2013
spinbarrier.h cleanup: get rid of util::noncopyable Aug 17, 2013
spinlock.h spinlocks: add exp backoff mechanism guarded by macro Aug 28, 2013
static_unordered_map.h ndb_type_traits: try to avoid calling destructors in containers Mar 25, 2013
static_vector.h #include nits Oct 6, 2013
stats_client.cc increase sampling rate Aug 15, 2013
stats_common.h separate stats_client + some refactoring Aug 13, 2013
stats_server.cc fix silly bugs Aug 13, 2013
stats_server.h separate stats_client + some refactoring Aug 13, 2013
str_arena.h tpcc: port changes over to new-benchmark Aug 24, 2013
test.cc Actually test Masstree scans. Oct 13, 2013
thread.cc ndb_thread: port to use std::thread + cleanup Oct 11, 2013
thread.h ndb_thread: port to use std::thread + cleanup Oct 11, 2013
ticker.cc WIP: redo the RCU subsystem Jul 23, 2013
ticker.h ticker.h: fix careless sleeping bug in tickerloop Mar 11, 2014
tuple.cc replace pthread_* functions with std::* variants Oct 11, 2013
tuple.h replace pthread_* functions with std::* variants Oct 11, 2013
txn.cc cleanup: remove unused key_range_t Aug 20, 2013
txn.h Internal: Change consistent_snapshot_tid() interface Oct 22, 2013
txn_btree.cc txn_btree: add another test case for rsearch Oct 23, 2013
txn_btree.h txn_btree: add some rsearch test case Oct 23, 2013
txn_impl.h Internal: Change consistent_snapshot_tid() interface Oct 22, 2013
txn_proto2_impl.cc percore_lazy: fix up implementation Aug 21, 2013
txn_proto2_impl.h Fix derp in commit protocol Oct 23, 2013
typed_txn_btree.h Add a Masstree btree variant. make M=1 to get it. Oct 5, 2013
util.h Make tree print output tolerable Oct 13, 2013
varint.cc checkpoint: some work on read/write cursors Apr 19, 2013
varint.h record: failsafe serializer variants Apr 19, 2013
varkey.h Actually compile Oct 5, 2013

README.md

Silo

This project contains the prototype of the database system described in

Speedy Transactions in Multicore In-Memory Databases 
Stephen Tu, Wenting Zheng, Eddie Kohler, Barbara Liskov, Samuel Madden 
SOSP 2013. 
http://people.csail.mit.edu/stephentu/papers/silo.pdf

This code is an ongoing work in progress.

Build

There are several options to build. MODE is an important variable governing the type of build. The default is MODE=perf, see the Makefile for more options. DEBUG=1 triggers a debug build (off by default). CHECK_INVARIANTS=1 enables invariant checking. There are two targets: the default target which builds the test suite, and dbtest which builds the benchmark suite. Examples:

MODE=perf DEBUG=1 CHECK_INVARIANTS=1 make -j
MODE=perf make -j dbtest

Each different combination of MODE, DEBUG, and CHECK_INVARIANTS triggers a unique output directory; for example, the first command above builds to out-perf.debug.check.masstree.

Silo now uses Masstree by default as the default index tree. To use the old tree, set MASSTREE=0.

Running

To run the tests, simply invoke <outdir>/test with no arguments. To run the benchmark suite, invoke <outdir>/benchmarks/dbtest. For now, look in benchmarks/dbtest.cc for documentation on the command line arguments. An example invocation for TPC-C is:

<outdir>/benchmarks/dbtest \
    --verbose \
    --bench tpcc \
    --num-threads 28 \
    --scale-factor 28 \
    --runtime 30 \
    --numa-memory 112G 

Benchmarks

To reproduce the graphs from the paper:

$ cd benchmarks
$ python runner.py /unused-dir <results-file-prefix>

If you set DRYRUN=True in runner.py, then you get to see all the commands that would be issued by the benchmark script.