Skip to content

Comments

[Backport] Opportunistic Striping#2495

Merged
eolivelli merged 1 commit intoapache:branch-4.11from
Ghatage:branch-4.11
Nov 24, 2020
Merged

[Backport] Opportunistic Striping#2495
eolivelli merged 1 commit intoapache:branch-4.11from
Ghatage:branch-4.11

Conversation

@Ghatage
Copy link
Contributor

@Ghatage Ghatage commented Nov 24, 2020

If you have a very small cluster, 3 bookies, you must use 2-2-2 as replication parameters in order to guarantee durability and tolerate the failure of at least one bookie.
With this situation you cannot leverage the striping feature of BookKeeper, when ensembleSize > writeQuorumSize.

With striping you can distribute your data among all of the bookies, and use better the available disk space.
You can also exploit the ability to write to/read from more disks (bookies) in parallel, achieving better overall performances.

With this change we introduce a new client option "opportunistic striping", that basically tells to the BK client to use as much bookies as configured in the ensembleSize parameter in the best case scenario, and to use less and less bookies, up to the configured writeQuorumSize.

  • New ClientConfiguration option "opportunisticStriping"
  • in case of opportunisticStriping the client is allowed to use less bookies than the requested ensemble size, up to writeQuorumSize
  • add test cases

Reviewers: Nicolò Boschi boschi1997@gmail.com, Anup Ghatage ghatage@apache.org

This closes #2493 from eolivelli/fix/bp-43-impl-auto-striping

Descriptions of the changes in this PR:

Motivation

(Explain: why you're making that change, what is the problem you're trying to solve)

Changes

(Describe: what changes you have made)

Master Issue: #


In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
checks for pull requests. A pull request can only be merged when it passes precommit checks.


Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

If this PR is a BookKeeper Proposal (BP):

  • Make sure the PR title is formatted like:
    <BP-#>: Description of bookkeeper proposal
    e.g. BP-1: 64 bits ledger is support
  • Attach the master issue link in the description of this PR.
  • Attach the google doc link if the BP is written in Google Doc.

Otherwise:

  • Make sure the PR title is formatted like:
    <Issue #>: Description of pull request
    e.g. Issue 123: Description ...
  • Make sure tests pass via mvn clean apache-rat:check install spotbugs:check.
  • Replace <Issue #> in the title with the actual Issue number.

If you have a very small cluster, 3 bookies, you must use **2-2-2** as replication parameters in order to guarantee durability and tolerate the failure of at least one bookie.
With this situation you cannot leverage the striping feature of BookKeeper, when **ensembleSize > writeQuorumSize**.

With striping you can distribute your data among all of the bookies, and use better the available disk space.
You can also exploit the ability to write to/read from more disks (bookies) in parallel, achieving better overall performances.

With this change we introduce a new client option "opportunistic striping", that basically tells to the BK client to use as much bookies as configured in the ensembleSize parameter in the best case scenario, and to use less and less bookies, up to the configured writeQuorumSize.

- New ClientConfiguration option "opportunisticStriping"
- in case of opportunisticStriping the client is allowed to use less bookies than the requested ensemble size, up to writeQuorumSize
- add test cases

Reviewers: Nicolò Boschi <boschi1997@gmail.com>, Anup Ghatage <ghatage@apache.org>

This closes apache#2493 from eolivelli/fix/bp-43-impl-auto-striping
@Ghatage Ghatage requested a review from eolivelli November 24, 2020 00:39
@Ghatage Ghatage self-assigned this Nov 24, 2020
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
thank you for backporting my patch !

@Ghatage
Copy link
Contributor Author

Ghatage commented Nov 24, 2020

@eolivelli doesn't GH Actions run for previous branches?

@eolivelli
Copy link
Contributor

@Ghatage maybe you are right.
We have to investigate

@Ghatage
Copy link
Contributor Author

Ghatage commented Nov 24, 2020

It seems so.

We might have to make changes to the github workflow yml files. I'll check up if we can do this in some centralized place so that we can toggle it for only the branches / releases which are not EOL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants