Badger instead of RocksDB

[Scapal]

Have you considered using Badger K/V store instead of RocksDB in order to keep CGO dependencies minimal ?

https://github.com/dgraph-io/badger

[ghost]

BLast uses bleve to do its magic.

I agree that removing cgo is nice and badgers perf and scaling is much easier to get working with raft IMHO.

here is the issue relating to using badger.
blevesearch/bleve#591

It looks possible now. It was not possible before.

[mosuka]

Hi @Scapal

I agree with you to keep minimize CGO dependencies.
Currently, Blast focuses on server implementation for using Bleve on distributed environment.
I will support Badger when Bleve merges Badger's code.

Thanks.

[ghost]

@mosuka The blocker for using Badger seems to have been lifted

see: blevesearch/bleve#591
at the bottom @manishrjain changed Badger to support transactions with multiples readers !

EDIT. It looks like Scorch is the only way forward to use bleve. KV ( BoltDB) is no longer supported.
I am not sue yet how you did your Raft implementation yet, but is it dependent on boltdb ?

[mosuka]

@gedw99 Blast has two storages. One is Bleve's index. The other one is KVS for storing raw data before indexing.
The latter currently uses BoltDB, but it is not affected by Bleve.
In the future I would like to migrate to Badger.

[ghost]

Hay all, there is a new badger bleve repo here: https://github.com/alash3al/bbadger

Its called "bbadger" !!

It would be great if everyone interested in this can help with examples, PR's and general dog fooding.
Its also very easy to use.

@mosuka
This should address the need for badger that you mentioned.

@Scapal This removes the CGO aspects. I agree with you about the CGO as its a huge slowdown for golang to use CGO

[mosuka]

Hi @Scapal @gedw99 ,

Fixed to support Bagder.

For example, build Badger to be available for index storage as follows:

$ make BUILD_TAGS=badger build

Then, you can start Blast as follows:

$ ./bin/blast-index start --node-id=index1 --data-dir=/tmp/blast/index1 --bind-addr=:6060 --grpc-addr=:5050 --http-addr=:8080 --index-storage-type=badger

Please clone the master branch.

[mosuka]

Sorry, it still have a bug.
Please wait.

[mosuka]

Fixed some Bader plugin bugs. But this feature is considered experimental.