-
Notifications
You must be signed in to change notification settings - Fork 504
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compaction multi-threading #754
Conversation
Ha! you deleted compactor bookmarks. This is my new favorite PR. |
Sorry then to say they were actually just moved into iterator_multiblock.go. Although there was a slight change to retain currentError. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comments included.
I'm finding the logic of the iterators and channels to be a bit confusing to follow. I think it's mostly b/c you directly translated the gross bookmark logic into a goroutine. The error path (actual error vs EOF) is I think the part I find most confusing.
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
…, separate context Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
4821abf
to
18c06b0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice!
Nice! Couldn't give this a thorough review but the decoupling of reads and writes using a buffered message queue looks great |
What this PR does:
Compaction is single-threaded and solidly limited to using a single core. This PR introduces some basic multi-threading by moving iteration to a separate goroutine, which improves performance.
Benchmark was added and shows ~12% improvement:
Testing in a real env is better:
![image](https://user-images.githubusercontent.com/13524475/121548760-4b1acb00-c9db-11eb-98fc-38c955420b49.png)
![image](https://user-images.githubusercontent.com/13524475/121549061-85846800-c9db-11eb-8aa5-38a34095fbc2.png)
objects and bytes written up ~20%
Some notes:
Iterator
interface doesn't expose the ability return slices fromNext()
. Will explore this in a later PR unless there is interest in trying it now.Which issue(s) this PR fixes:
n/a
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]