Skip to content
RocksDB/LevelDB inspired key-value database in Go
Go Other
  1. Go 99.9%
  2. Other 0.1%
Branch: master
Clone or download
petermattis Merge pull request #397 from cockroachdb/pmattis/db-tool
tool: automatically infer comparer/merger for db commands
Latest commit 309120b Nov 15, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bloom Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
cmd/pebble db: extend the metrics returned by DB.Metrics Oct 11, 2019
docs More code comments about compactions and range tombstones. Oct 30, 2019
internal internal/batchskl: add in-order insertion fast-path Nov 4, 2019
sstable sstable: retain blockIter buffers across sync.Pool reuse Nov 1, 2019
testdata change TableIngestInfo.String to a single-line format Nov 15, 2019
tool tool: automatically infer comparer/merger for db commands Nov 15, 2019
vendor Update vendored modules Sep 13, 2019
vfs Add vfs.FS.ReuseForWrite method for reusing WAL files. Oct 30, 2019
.travis.yml use `go get -d` so we don't install sub-packages and deps Nov 7, 2019
LICENSE Initial check-in. Aug 23, 2011
Makefile explicitly disable go module support for {test*,stress*,generate} Oct 20, 2019
README.md add DB.Checkpoint Sep 28, 2019
batch.go dynamically grow the memtable size Nov 11, 2019
batch_test.go db: extend the metrics returned by DB.Metrics Oct 11, 2019
cache.go move pebble/cache to pebble/internal/cache Sep 27, 2019
checkpoint.go add DB.Checkpoint Sep 28, 2019
checkpoint_test.go add DB.Checkpoint Sep 28, 2019
cleaner.go db: provide facility for retaining obsolete WALs, MANIFESTs, and ssta… Oct 4, 2019
cleaner_test.go db: provide facility for retaining obsolete WALs, MANIFESTs, and ssta… Oct 4, 2019
commit.go remove batchStorage Nov 4, 2019
commit_test.go dynamically grow the memtable size Nov 11, 2019
compaction.go disable zeroing of seqnums when flushing Nov 12, 2019
compaction_iter.go allow accumulation of merge operands before merge Oct 29, 2019
compaction_iter_test.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
compaction_picker.go add missing compaction.expandInputs calls Oct 17, 2019
compaction_picker_test.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
compaction_test.go fix compaction.allowZeroSeqNum when flushing Nov 7, 2019
comparer.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
data_test.go dynamically grow the memtable size Nov 11, 2019
db.go only grow the memtable size if we filled it over 50% Nov 13, 2019
db_test.go only grow the memtable size if we filled it over 50% Nov 13, 2019
error_iter.go Add support for dynamic bounds May 25, 2019
error_test.go Add vfs.FS.ReuseForWrite method for reusing WAL files. Oct 30, 2019
event.go change TableIngestInfo.String to a single-line format Nov 15, 2019
event_listener_test.go only grow the memtable size if we filled it over 50% Nov 13, 2019
filenames.go Run tests on Windows and OS X during CI Sep 20, 2019
flush_test.go Introduce Formatter function for formatting user keys Sep 8, 2019
get_iter.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
get_iter_test.go dynamically grow the memtable size Nov 11, 2019
go.mod Update vendored modules Sep 13, 2019
go.sum Update vendored modules Sep 13, 2019
ingest.go ingest: Keep paths and fileMetas in sync when sorting Oct 22, 2019
ingest_test.go dynamically grow the memtable size Nov 11, 2019
internal.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
internal_test.go Remove InternalIterator.{Key,Value,Valid} Sep 15, 2019
iterator.go fix Iterator.SeekPrefixGE semantics Nov 11, 2019
iterator_test.go Merge pull request #279 from cockroachdb/pmattis/merging-iter-valid Sep 18, 2019
level_iter.go levelIter bug fix: Handle the case where index is valid, err is nil Nov 13, 2019
level_iter_test.go sstable: rename {,Table}Options to {Reader,Writer}Options Oct 10, 2019
log_recycler.go Fix metrics WAL file tracking Jul 18, 2019
log_recycler_test.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
logger.go make DefaultLogger a variable instead of a type Oct 10, 2019
mem_table.go only grow the memtable size if we filled it over 50% Nov 13, 2019
mem_table_test.go dynamically grow the memtable size Nov 11, 2019
merger.go allow accumulation of merge operands before merge Oct 29, 2019
merging_iter.go Comments on why certain loops in mergingIter will make forward progress. Nov 9, 2019
merging_iter_heap.go Replace db package with internal/base May 23, 2019
merging_iter_test.go Comments on why certain loops in mergingIter will make forward progress. Nov 9, 2019
metrics.go db: extend the metrics returned by DB.Metrics Oct 11, 2019
metrics_test.go db: extend the metrics returned by DB.Metrics Oct 11, 2019
open.go only grow the memtable size if we filled it over 50% Nov 13, 2019
open_test.go dynamically grow the memtable size Nov 11, 2019
options.go tool: automatically infer comparer/merger for db commands Nov 15, 2019
options_test.go move base.Options to pebble.Options Oct 10, 2019
pacer.go db: extend the metrics returned by DB.Metrics Oct 11, 2019
pacer_test.go db: extend the metrics returned by DB.Metrics Oct 11, 2019
race_off.go Detect leaked iterators Nov 16, 2018
race_on.go Detect leaked iterators Nov 16, 2018
range_del_test.go *: display symbolic key kind, instead of numeric Oct 30, 2019
read_state.go reserve space for memtables from the cache Oct 14, 2019
read_state_test.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
snapshot.go Provide guard rails for various invalid operations Jun 2, 2019
snapshot_test.go Fix imports: petermattis/pebble -> cockroachdb/pebble Sep 6, 2019
syncing_fs.go add DB.Checkpoint Sep 28, 2019
table_cache.go rename make{Reader,Writer}Options to Options.Make{Reader,Writer}Options Oct 13, 2019
table_cache_test.go sstable: rename {,Table}Options to {Reader,Writer}Options Oct 10, 2019
version_set.go close the manifest on DB.Close() Nov 4, 2019

README.md

DO NOT USE: Pebble is an incomplete work-in-progress.

Pebble Build Status

Pebble is a LevelDB/RocksDB inspired key-value store focused on performance and internal usage by CockroachDB. Pebble inherits the RocksDB file formats and a few extensions such as range deletion tombstones, table-level bloom filters, and updates to the MANIFEST format.

Pebble intentionally does not aspire to include every feature in RocksDB and is specifically targetting the use case and feature set needed by CockroachDB:

  • Block-based tables
  • Checkpoints
  • Indexed batches
  • Iterator options (lower/upper bound, table filter)
  • Level-based compaction
  • Manual compaction
  • Merge operator
  • Prefix bloom filters
  • Prefix iteration
  • Range deletion tombstones
  • Reverse iteration
  • SSTable ingestion
  • Single delete
  • Snapshots
  • Table-level bloom filters

RocksDB has a large number of features that are not implemented in Pebble:

  • Backups
  • Column families
  • Delete files in range
  • FIFO compaction style
  • Forward iterator / tailing iterator
  • Hash table format
  • Memtable bloom filter
  • Persistent cache
  • Pin iterator key / value
  • Plain table format
  • SSTable ingest-behind
  • Sub-compactions
  • Transactions
  • Universal compaction style

Pebble may silently corrupt data or behave incorrectly if used with a RocksDB database that uses a feature Pebble doesn't support. Caveat emptor!

Advantages

Pebble offers several improvements over RocksDB:

  • Faster reverse iteration via backwards links in the memtable's skiplist.
  • Faster commit pipeline that achieves better concurrency.
  • Seamless merged iteration of indexed batches. The mutations in the batch conceptually occupy another memtable level.
  • Smaller, more approachable code base.
  • Pacing of flushes vs compactions: Pebble smooths latency spikes caused by flushes and compactions by only flushing/compacting as fast as necessary to keep up with user writes.

See the Pebble vs RocksDB: Implementation Differences doc for more details on implementation differences.

Pedigree

Pebble is based on the incomplete Go version of LevelDB:

https://github.com/golang/leveldb

The Go version of LevelDB is based on the C++ original:

https://github.com/google/leveldb

Optimizations and inspiration were drawn from RocksDB:

https://github.com/facebook/rocksdb

You can’t perform that action at this time.