Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[xCluster] Batch size computation in BatchedWriteImplementation::ProcessRecord is inefficient #16237

Closed
hari90 opened this issue Feb 25, 2023 · 0 comments
Closed

[xCluster] Batch size computation in BatchedWriteImplementation::ProcessRecord is inefficient #16237

hari90 opened this issue Feb 25, 2023 · 0 comments
Assignees
@hari90
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@hari90
Copy link
Contributor

hari90 commented Feb 25, 2023 •
edited by jira bot
Loading

Jira Link: DB-5659

Description

queue.back()->ByteSizeLong() for every record takes o(n^2)
@hari90 hari90 added kind/enhancement This is an enhancement of an existing feature area/docdb YugabyteDB core features labels Feb 25, 2023
@hari90 hari90 self-assigned this Feb 25, 2023
@yugabyte-ci yugabyte-ci added the priority/medium Medium priority issue label Feb 25, 2023
@hari90 hari90 changed the title [DocDB] Batch size computation in BatchedWriteImplementation::ProcessRecord is inefficient [xCluster] Batch size computation in BatchedWriteImplementation::ProcessRecord is inefficient Feb 25, 2023
@hari90 hari90 mentioned this issue Feb 25, 2023
Closed
@karan-yb karan-yb mentioned this issue Apr 6, 2023
Closed
1 task
karan-yb added a commit that referenced this issue Apr 7, 2023
@karan-yb
[#16237] xCluster: deprecate flags cdc_max_apply_batch_num_records an…
1ab0019 
…d cdc_max_apply_batch_size_bytes

Summary: This change removes the sub-batching performed at XCluster consumer level since splitting a single producer side batch into multiple consumer side batches can cause overwrite of intents and lead to rocksdb corruption. See this commit 64a7b3a for more details.

Test Plan:
./yb_build.sh --cxx-test xcluster-test --gtest_filter XCluster*TestInsertDeleteWorkloadWithRestart*
Jenkins

Reviewers: slingam, mlillibridge, hsunder, rahuldesirazu

Reviewed By: hsunder, rahuldesirazu

Subscribers: bogdan, rthallam

Differential Revision: https://phabricator.dev.yugabyte.com/D24185
premkumr pushed a commit to premkumr/yugabyte-db that referenced this issue Apr 10, 2023
@karan-yb @premkumr
[yugabyte#16237] xCluster: deprecate flags cdc_max_apply_batch_num_re…
957ca34 
…cords and cdc_max_apply_batch_size_bytes

Summary: This change removes the sub-batching performed at XCluster consumer level since splitting a single producer side batch into multiple consumer side batches can cause overwrite of intents and lead to rocksdb corruption. See this commit 64a7b3a for more details.

Test Plan:
./yb_build.sh --cxx-test xcluster-test --gtest_filter XCluster*TestInsertDeleteWorkloadWithRestart*
Jenkins

Reviewers: slingam, mlillibridge, hsunder, rahuldesirazu

Reviewed By: hsunder, rahuldesirazu

Subscribers: bogdan, rthallam

Differential Revision: https://phabricator.dev.yugabyte.com/D24185
@hari90 hari90 closed this as completed May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hari90 hari90

Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hari90 @yugabyte-ci