Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 8.0]: Introduce experimental SST partitioning #12630

Merged
merged 3 commits into from
May 3, 2023
Merged

[Backport 8.0]: Introduce experimental SST partitioning #12630

merged 3 commits into from
May 3, 2023

Conversation

Zelldon
Copy link
Member

@Zelldon Zelldon commented May 2, 2023

Description

Backports #12483

Related issues

closes #12033

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

  • The changes are backwards compatibility with previous versions
  • If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

  • There are unit/integration tests that verify all acceptance criterias of the issue
  • New tests are written to ensure backwards compatibility with further versions
  • The behavior is tested manually
  • The change has been verified by a QA run
  • The impact of the changes is verified by a benchmark

Documentation:

  • The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
  • If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Other teams:
If the change impacts another team an issue has been created for this team, explaining what they need to do to support this change.

Please refer to our review guidelines.

Zelldon and others added 3 commits May 2, 2023 16:01
@Zelldon
feat: introduce SST partitioning
2f1a400 
SST partitioning based on column family prefix (virtual column family)
allows to split up key ranges in separate SST files, which should
improve compaction and makes propagation of SST files less write
amplifying.

It will cause to create more files in runtime and snapshot as it will
create more SST files. At least for each column family we use at
runtime.

Benchmarks have shown impressive results.
@Zelldon
feat: introduce experimental config for sst partitioning
ac03e3f
@Zelldon @megglos
docs: improve config description
cd66e83 
Co-authored-by: Sebastian Bathke <sebastian.bathke@camunda.com>
@Zelldon
Copy link
Member Author

Zelldon commented May 3, 2023

Only smaller merge conflict had to be resolved.

remcowesterhoud
remcowesterhoud approved these changes May 3, 2023
View reviewed changes
Copy link
Contributor

@remcowesterhoud remcowesterhoud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bors merge

@zeebe-bors-camunda
Copy link
Contributor

Build succeeded:

@zeebe-bors-camunda zeebe-bors-camunda bot merged commit fd2076a into stable/8.0 May 3, 2023
25 checks passed
@zeebe-bors-camunda zeebe-bors-camunda bot deleted the zell-backport-80-12483 branch May 3, 2023 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@remcowesterhoud remcowesterhoud remcowesterhoud approved these changes

@megglos megglos Awaiting requested review from megglos

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Zelldon @remcowesterhoud