RocksDB concepts not in Badger

[mperham]

Hi, I build Faktory, a background job server built in Go which uses RocksDB today. It's painful to deal with CGo and Badger looks awesome, probably the most realistic Go-native replacement. However there are a few features I use that Badger does not offer today.

Column families allow RocksDB to separate different types of data into different buckets within the same DB. This makes it easy to iterate only over items within that type, similar to tables in an RDBMS. https://github.com/facebook/rocksdb/wiki/Column-Families

Merge operator - RocksDB allows merging of values so you can fire multiple writes at a key without having to read the current value first. We use it for incrementing statistics counters. https://github.com/facebook/rocksdb/wiki/Merge-Operator

Do you have any plans to add similar features to Badger?

[manishrjain]

The concept of col families can be simulated by adding prefixes to keys. Let us know if that wouldn't solve your use case.

The merge operator looks useful, and needs a bit of thought. I wonder if the generic merge operator class is actually useful, or an unnecessary complication. Would a simple AddCounter() like operation work better. Badger's Gets are fast, so the biggest benefit I see here is that txns adding to a counter won't abort if run concurrently.

[manishrjain]

On a second thought, you could easily create a channel where can take in the deltas for counters, have a goroutine which goes over these deltas and merges them, at the client side. And this goroutine can periodically via a ticker, write

ch := make(chan int, 100)
ch <- 1 // Say on a page view

func reduce(ch chan int) {
tick := time.NewTicker(5*time.Second)
var total int
for {
  select {
    case delta := <-ch:
      total += delta
    case <-tick.Ch:
      if total == 0 { continue }
      // Run a transaction in Badger to read counter val,
      // then add total to it and write it back.
      // You can use callback in txn.Commit() to avoid
      // waiting for the write to disk to happen.
     total = 0
  }
}
}

This would be a lot more efficient than having Badger do it because every write to Badger must be persisted to disk before ack. So, every Add in Badger would have to hit the disk, which you ideally don't want to do for handling stats counters.

[mperham]

Sure, I realize that column families can be simulated but that just forces everyone to build their own abstraction.

That counter implementation isn't transaction safe, right?

The whole point of asking for these features is so that you, as the storage engine developer, can give certain guarantees that I can't implement at the app layer, e.g. persisting counter increments to the WAL for consistency.

[manishrjain]

I realize that column families can be simulated but that just forces everyone to build their own abstraction.

If Badger could provide any more functionality on top which would differentiate among col families, then it makes sense for us to bake it into the API. Given that's not the case, providing a col families concept would only be confusing -- people would expect the storage to be different for different col families; but that won't be the case. Therefore, there's little benefit in Badger providing this abstraction.

@deepakjois do you have any thoughts on providing a Merge like functionality? A durable counter sounds potentially useful -- we could generate a new (key) version per entry, and then merge them in the background to provide the sum, purging lower versions in the process. Similar to what we do in Dgraph.

[mperham]

@manishrjain I can think of lots of possible column family functionality that would be painful or impossible to implement at a higher-layer:

• per-CF memory, cache, persistence tunings, e.g. in-memory-only CFs.
• Size(), Clear(), Drop() atomic operations per-CF.

I don't pretend to know if the complexity introduced is worth it but I appreciate you considering it; I think it's a natural evolution to go from one big global keyspace to smaller named keyspaces.

[manishrjain]

The current design isn't capable of providing specific tunings to different potential CFs; I can't think of an easy way to do this, without changing the underlying design of the system. So, this isn't a priority.

Merge, OTOH, is something we can easily bake in.

[svperfecta]

FWIW - We use namespaced Sets, and it works pretty well.

The pattern we followed was to wrap Badger up in a higher level strongly typed API which exposed method like SetJob for specific types we work with. I think this is a better pattern in go given the lack of Generics. That said, I do think it would be pretty easy to expose this and its going to be a typical use case. BoltDB does this pretty nicely IIRC.

@mperham - Another option for you would be to reflect on the type and use that as some sort of namespace.

[svperfecta]

@manishrjain On the merge issue - We're dealing with this now too.

[mperham]

I'd definitely love to see merge support. I'm trying to spike a Badger version of Faktory now.

[deepakjois]

@mperham We are working on merge support in #360. It should be merged very soon now.

[deepakjois]

Merge support is now added. Please try it out. Closing this for now. Please re-open if you have any other concerns related to this.

[komuw]

you guys, do move fast. sweet. 👍