[Backport] Opportunistic Striping#2495
Merged
eolivelli merged 1 commit intoapache:branch-4.11from Nov 24, 2020
Merged
Conversation
If you have a very small cluster, 3 bookies, you must use **2-2-2** as replication parameters in order to guarantee durability and tolerate the failure of at least one bookie. With this situation you cannot leverage the striping feature of BookKeeper, when **ensembleSize > writeQuorumSize**. With striping you can distribute your data among all of the bookies, and use better the available disk space. You can also exploit the ability to write to/read from more disks (bookies) in parallel, achieving better overall performances. With this change we introduce a new client option "opportunistic striping", that basically tells to the BK client to use as much bookies as configured in the ensembleSize parameter in the best case scenario, and to use less and less bookies, up to the configured writeQuorumSize. - New ClientConfiguration option "opportunisticStriping" - in case of opportunisticStriping the client is allowed to use less bookies than the requested ensemble size, up to writeQuorumSize - add test cases Reviewers: Nicolò Boschi <boschi1997@gmail.com>, Anup Ghatage <ghatage@apache.org> This closes apache#2493 from eolivelli/fix/bp-43-impl-auto-striping
eolivelli
approved these changes
Nov 24, 2020
Contributor
eolivelli
left a comment
There was a problem hiding this comment.
LGTM
thank you for backporting my patch !
Contributor
Author
|
@eolivelli doesn't GH Actions run for previous branches? |
Contributor
|
@Ghatage maybe you are right. |
Contributor
Author
|
We might have to make changes to the github workflow yml files. I'll check up if we can do this in some centralized place so that we can toggle it for only the branches / releases which are not EOL. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If you have a very small cluster, 3 bookies, you must use 2-2-2 as replication parameters in order to guarantee durability and tolerate the failure of at least one bookie.
With this situation you cannot leverage the striping feature of BookKeeper, when ensembleSize > writeQuorumSize.
With striping you can distribute your data among all of the bookies, and use better the available disk space.
You can also exploit the ability to write to/read from more disks (bookies) in parallel, achieving better overall performances.
With this change we introduce a new client option "opportunistic striping", that basically tells to the BK client to use as much bookies as configured in the ensembleSize parameter in the best case scenario, and to use less and less bookies, up to the configured writeQuorumSize.
Reviewers: Nicolò Boschi boschi1997@gmail.com, Anup Ghatage ghatage@apache.org
This closes #2493 from eolivelli/fix/bp-43-impl-auto-striping
Descriptions of the changes in this PR:
Motivation
(Explain: why you're making that change, what is the problem you're trying to solve)
Changes
(Describe: what changes you have made)
Master Issue: #