Data partitioning/Column families

[marvin-j97]

Column families/partitions are the missing piece to having a great storage engine. Each column family is its own partition inside the LSM, but all share the same journal, enabling atomic writes. So each column family is physically stored separately from the others. Compaction then looks at each partition instead, never mixing tables of different partitions. It's like a big meta-LSM tree, instead of using multiple trees, which don't have atomic semantics, unless putting another super structure on top. That super structure will be the new LSM tree which contains partitions basically. Compaction may even be set differently for each column family.

When creating no column family, everything gets put into the "default" column family.
Creating column families is pretty simple, you add a new memtable. Dropping a column family is simple. Delete its memtable and all the segments inside the partition. The journal needs to be handled accordingly, because flushing one column family doesn't necessarily mean the others are flushed too. And if the column family was deleted, the journal should not flush parts to the partition at all.

There needs to be new semantics for writing to a column family:

• insert(col_family, key, value)
• remove(col_family, key)
• batch() ... needs col_family support

https://github.com/facebook/rocksdb/wiki/Column-Families

Example use cases

• One partition could be an index in a relational database for example, so writing a row into "default" also atomically inserts/updates rows inside "index-1" (https://github.com/facebook/mysql-5.6/wiki/MyRocks-data-dictionary-format)
• Dropping the index just drops the column family "index-1", which is a O(1) operation
• another example would be locality groups in Bigtable and column families in RocksDB and Cassandra

[marvin-j97]

Architecture first pass