Go Embeddable Store
Go is a compiled language with a number of high-performance data stores that can be compiled with Go into a single binary. These databases are called "in-process" or "embedded".
By default, most apps slap MongoDB or MySQL/MariaDB into an application for storing state. Since we left the .net/node/php/ruby/python world we can have our database bundled right into our application. For smaller projects, which do not need a globally distributed / sharded database, this is a great devops win.
Embedded databases are propbably not for you if:
- building a serverless app (AWS lambda / Zeit Now)
- dealing with >1 Million HTTP requests a day
- have highly concurrent request patterns
These benchmarks and tests are off for a variety of reasons including differing levels of durability/sync and feature-scope. I recommend people default to bbolt for a general store.
Embeddable Go Databases
boltdb - https://github.com/boltdb/bolt
Solid, well-tested, development is locked. Each transaction has a consistent view of the data as it existed when the transaction started. Slowest of the bunch. Largest file size. Slowest (due to consistent-data views).
- Blast - full text search and indexing server cluster via Raft
- Storm - ORM-ish layer for database access
bbolt - https://github.com/etcd-io/bbolt
Fork of boltdb to add new features.
- BoltHold - simple querying and indexing layer
- Storm - query-builder, indexes, and struct storage
- Buckets - simple access layer
pogreb - https://github.com/akrylysov/pogreb
Optimized store for random lookups. No support for range/prefix scans. Slow in this benchmark. More...
goleveldb - https://github.com/syndtr/goleveldb
Well tested. Smallest storage size.
- ledisdb - redis-like abstraction over it
badgerdb - https://github.com/dgraph-io/badger
Fastest engine for random lookups and inserts. Graph engine dgraph built upon it. More...
keydb - https://github.com/robaho/keydb
Not tested.
More Complex Engines
tiedot - https://github.com/HouzuoGuo/tiedot/
Tiedot is a document store, it really can't be compared to these other databases correctly. Included only for loose reference. Super-fast write speeds. Huge storage requirements.
SQLite
Not tested. Multiple versions exist, most wrap the C code.
database/sql driver
no database/sql driver
Comparison between bvinc/go-sqlite-lite & crawshaw/sqlite.
Non-Go Embeddable Databases (Cgo)
- RocksDB
- LMDB
- hyperleveldb
Sample Results
Number of keys: 5000
Minimum key size: 32, maximum key size: 64
Minimum value size: 128, maximum value size: 1024
Concurrency: 3
Running tiedot benchmark...
Put: 0.022 sec, 226300 ops/sec
Get: 0.001 sec, 3552460 ops/sec
Put + Get time: 0.024 sec
File size: 512.00MB
Running pogreb benchmark...
Put: 0.142 sec, 35293 ops/sec
Get: 0.002 sec, 2489791 ops/sec
Put + Get time: 0.144 sec
File size: 4.25MB
Running goleveldb benchmark...
Put: 0.035 sec, 144902 ops/sec
Get: 0.009 sec, 568586 ops/sec
Put + Get time: 0.043 sec
File size: 3.05MB
Running bolt benchmark...
Put: 15.868 sec, 315 ops/sec
Get: 0.004 sec, 1136989 ops/sec
Put + Get time: 15.873 sec
File size: 8.00MB
Running bbolt benchmark...
Put: 14.710 sec, 339 ops/sec
Get: 0.006 sec, 874240 ops/sec
Put + Get time: 14.716 sec
File size: 8.00MB
Articles
- https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/
- https://petewarden.com/2010/10/01/how-i-ended-up-using-s3-as-my-database/
- https://hackernoon.com/what-i-learnt-from-building-3-high-traffic-web-applications-on-an-embedded-key-value-store-68d47249774f
Indexing data
- https://github.com/blevesearch/bleve
- http://roaringbitmap.org/
- https://www.pilosa.com/docs/latest/data-model/
- https://github.com/Xeoncross/keyset
Other benchmarks
- badgerdb, goleveldb, boltdb benchmark by zchee.
- badgerdb, goleveldb, boltdb, rocksdb benchmark by dgraph. (High-Quality)
- goleveldb, boltdb, pogreb write/put benchmark by mattn.