Skip to content
RocksDB/LevelDB inspired key-value database in Go
Branch: master
Clone or download
petermattis Merge pull request #61 from petermattis/pmattis/arena-size-overflow
internal/arenaskl: protect against overflow in Arena allocation
Latest commit 47f9647 Mar 15, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bloom sstable: remove support for block-based bloom filters Jan 4, 2019
cache Add vanity imports Nov 28, 2018
cmd/pebble db: fix default AbbreviatedKey implementation Mar 7, 2019
db
internal internal/arenaskl: protect against overflow in Arena allocation Mar 12, 2019
ptable Add vanity imports Nov 28, 2018
sstable
storage Fix flakiness in TestManualCompaction and TestEventListener Dec 4, 2018
testdata Fix a buglet where flushable creation batch would crash on an empty key Dec 16, 2018
vendor Vendor dependencies Jan 4, 2019
.travis.yml
LICENSE
Makefile
README.md
batch.go *: rename InlineKey to AbbreviatedKey Feb 16, 2019
batch_test.go
bench.txt
commit.go
commit_test.go Move record to internal/record Nov 10, 2018
compaction.go Elide range tombstones when possible during flushing / compaction Dec 16, 2018
compaction_iter.go Elide range tombstones when possible during flushing / compaction Dec 16, 2018
compaction_iter_test.go
compaction_picker.go
compaction_picker_test.go
compaction_test.go
controller.go Move rate to internal/rate Nov 2, 2018
controller_test.go
data_test.go
db.go
db_test.go
error_iter.go
event_listener_test.go Remove inadvertent dependency on cockroach Dec 4, 2018
filenames.go
filenames_test.go
flush_test.go
get_iter.go
get_iter_test.go Add Comparer.Equal Dec 20, 2018
go.mod
go.sum Add support for prefix bloom filters Feb 18, 2019
ingest.go Return iterator validity from First/Last/Seek{GE,LT} Dec 9, 2018
ingest_test.go
internal.go
iterator.go Add Comparer.Equal Dec 20, 2018
iterator_test.go Fix NPEs during iterator benchmarks Jan 5, 2019
level_iter.go Return iterator validity from First/Last/Seek{GE,LT} Dec 9, 2018
level_iter_test.go Fix a bug in setting table boundaries in the presence of range tombst… Dec 3, 2018
mem_table.go
mem_table_test.go
merging_iter.go
merging_iter_heap.go Add mergingIterItem.value Dec 20, 2018
merging_iter_test.go
open.go
open_test.go
race_off.go Detect leaked iterators Nov 16, 2018
race_on.go Detect leaked iterators Nov 16, 2018
range_del_test.go
rate_limited_file.go Add missing copyright notices Aug 12, 2018
snapshot.go
snapshot_test.go Replace Iterator interface with dbIter implementation Dec 9, 2018
table_cache.go Rename sstable.Iter to sstable.Iterator (for naming consistency) Dec 9, 2018
table_cache_test.go Open both point and range-del iterators in a single tableCache call Nov 29, 2018
version.go Fix a bug in setting table boundaries in the presence of range tombst… Dec 3, 2018
version_edit.go
version_edit_test.go
version_set.go
version_test.go

README.md

DO NOT USE: Pebble is an incomplete work-in-progress.

Pebble Build Status

Pebble is a LevelDB/RocksDB inspired key-value store focused on performance and internal usage by CockroachDB. Pebble inherits the RocksDB file formats and a few extensions such as range deletion tombstones, table-level bloom filters, and updates to the MANIFEST format.

Pebble intentionally does not aspire to include every feature in RocksDB and is specifically targetting the use case and feature set needed by CockroachDB:

  • Block-based tables
  • Indexed batches
  • [TODO] Iterator options (prefix, lower/upper bound, table filter)
  • Level-based compaction
  • Manual compaction
  • Merge operator
  • [TODO] Prefix bloom filters
  • Range deletion tombstones
  • Reverse iteration
  • Snapshots
  • SSTable ingestion
  • Table-level bloom filters

RocksDB has a large number of features that are not implemented in Pebble:

  • Backups and checkpoints
  • Column families
  • Delete files in range
  • FIFO compaction style
  • Forward iterator / tailing iterator
  • Hash table format
  • Memtable bloom filter
  • Persistent cache
  • Pin iterator key / value
  • Plain table format
  • Single delete
  • SSTable ingest-behind
  • Sub-compactions
  • Transactions
  • Universal compaction style

Pebble may silently corrupt data or behave incorrectly if used with a RocksDB database that uses a feature Pebble doesn't support. Caveat emptor!

Advantages

Pebble offers several improvements over RocksDB:

  • Faster reverse iteration via backwards links in the memtable's skiplist.
  • Faster commit pipeline that achieves better concurrency.
  • Seamless merged iteration of indexed batches. The mutations in the batch conceptually occupy another memtable level.
  • Smaller, more approachable code base.
  • [TODO] Pacing of user writes vs flushes vs compactions.

Pedigree

Pebble is based on the incomplete Go version of LevelDB:

https://github.com/golang/leveldb

The Go version of LevelDB is based on the C++ original:

https://github.com/google/leveldb

Optimizations and inspiration were drawn from RocksDB:

https://github.com/facebook/rocksdb

You can’t perform that action at this time.