Skip to content

[Optimization] Tunes badger for raftwal #4472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 9, 2020
Merged

Conversation

harshil-goel
Copy link
Contributor

@harshil-goel harshil-goel commented Dec 26, 2019

We observed that when running dgraph at scale, things started to get slow. The throughput in diskless mode went from 8k to less than 1k within minutes for continous mutations. We noticed that most(80%) of time was being taken while PreFetch() in badger for raft-wal. This PR fixes that. Now the throughput stabalizes around 7k expect for when compaction happens.


This change is Reviewable

@harshil-goel harshil-goel requested a review from a team December 26, 2019 16:45
Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: Please add a big comment. Also, prefix the title with [Optimization].

Reviewed 1 of 1 files at r1.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @harshil-goel)


raftwal/storage.go, line 526 at r1 (raw file):

		iopt := badger.DefaultIteratorOptions
		iopt.PrefetchValues = false

Add a big comment here that: multiple things: One that in LSM tree mode, which is how WAL is opened, there's no need to prefetch values, because the values are colocated with the keys. And secondly, if we set this to false, we no longer see the spikes in mutation throughput exhibited by Dgraph.

Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Just one thing. From the description I gather this was affecting diskless mode. What about the normal mode? From the code it looks like it's also being affected by this change. If that's the case it's a good idea to mention it in the commit description and/or comment.

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @harshil-goel)

@harshil-goel harshil-goel changed the title Tunes badger for raftwal [Optimization] Tunes badger for raftwal Dec 31, 2019
@harshil-goel harshil-goel merged commit a053e83 into master Jan 9, 2020
@harshil-goel harshil-goel deleted the harshil-goel/raft-opt branch January 9, 2020 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants