Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use WriteBatch for Put operation in badger DB. #2

Merged
merged 2 commits into from Jun 26, 2019

Conversation

martinmr
Copy link

This reduces the running time by a lot. In my machine (with 1M keys and
two go routines) the Put running time went from ~20s to ~4s.

This reduces the running time by a lot. In my machine (with 1M keys and
two go routines) the Put running time went from ~20s to ~4s.
@recoilme
Copy link
Owner

Hello, @martinmr !
But other databases may batch writing operations too. I think use batching only for badger - will not fair..
May you please:

  • add batching for goleveldb and bbolt?
  • implement batching as an command line option for the test?

@martinmr
Copy link
Author

I have added batching for goleveldb and bbolt.

With respect to the flag, I don't think it's needed. If batching is available, it should be the only default. At least with badger, using a transaction per put is not the way serial transactions would be made in a real-use case so using that procedure for benchmarks would result in inaccurate results. I assume it's the same for other dbs.

@recoilme
Copy link
Owner

Thank you for adding batching, looks good!
I merge it now but test a little later. But i think code like this:

start batch
insert millions key/value
write batch (write insertions on disk, with fsync())

Far from the real-world use case. Because without fsync all writes may be lost (they are stored in disk-cache before fsync, but i think you know it). On the other hand, batching very useful when you write some logs or data by portions or import database from a backup for example.

So, i think flag for batching is needed

@recoilme recoilme merged commit e876a83 into recoilme:master Jun 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants