add badger db support

[joeblew99]

https://github.com/dgraph-io/badger

I am proposing that this DB be supported by bleve.
Badger on this outperforms rocksdb, which is outstanding.

here is a blog post with the benchmarks
https://open.dgraph.io/post/badger/

My understanding is that they will be adding snapshotting also.

[mschoch]

If someone from the community wants to add support it that would be great.

Internally we're focusing on support for moss in the bleve 1.x line. And in bleve 2.x we'll most likely be moving to a custom format and not using generic k/v store in the default index implementation.

[joeblew99]

ok.. well good to know the roadmap.
i dont have time to do it. Will leave open ?

[mschoch]

Sure, let's leave it open.

[akhenakh]

I'm planning on giving it a shot.

Anyone else working on it?

[bt]

@akhenakh That would be awesome, please! :)

[akhenakh]

WIP https://github.com/akhenakh/bleve/tree/badger

[bt]

Hey @akhenakh, great work so far!

I'm having a look at it and it's looking great. Was hoping I could help out.

I'm just taking a look at Badger and given that they don't have any plans to support snapshots (hypermodeinc/badger#39), what's the plan here?

I believe they talked a bit about dumping the database concurrently, but the database won't be frozen in time.

[akhenakh]

Yeah the snapshot is not going to happen soon.
I'm using the KV interface outside of Bleve, so it won't be an issue for me when used in specific places.

But not sure how it would impact Bleve, any clue @bt ?

[mschoch]

Pretty much all of the search side of Bleve requires a snapshot to work correctly. The reason is that multiple iterators are created, and they have to be guaranteed to the see the same underlying data. If they don't, you'll get inconsistent search results.

A simple example would be a phrase search for "fat cat". If version 1 of a document has "fat man" and the new version of a document has "old cat", without snapshots its possible that a search for the phrase "fat cat" is successful, even though neither the old nor the new version of the document ever contained that phrase.

You might think that atomic batches prevent this, but without snapshots, its possible for the first iterator (looking for "fat") sees the old version of the document, and second iterator (looking for "cat") sees the new version of the document.

[akhenakh]

thanks for the clarification @mschoch
Let's talk to the dgraph guys to see what they have in mind.

[bt]

Hey @mschoch, just as a clarification just to make sure I'm getting this correct:

If I was to execute a search, and whilst the search is running, the store is updated with a new document AND that document matches the search phrase, the search shouldn't see the new document, because the search would only be searching the snapshot that is made at the time the search was first executed. Am I correct?

In this case, even with the workaround that was suggested in the linked issue above would not work, as if the iterator moves as new data is indexed, it would pick up the new data as the search is run.

If that's true, then I guess the only way would be to implement snapshotting into Badger, if that's part of their plans.