Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support split write request to batches for small wal logs #754

Merged
merged 6 commits into from
Mar 24, 2023

Conversation

ShiKaiWi
Copy link
Member

Which issue does this PR close?

Closes #

Rationale for this change

Currently, one large write request may lead to one large WAL log, which is not friendly for some wal implementations.

What changes are included in this PR?

  • Support split write table request to multiple batches.

Are there any user-facing changes?

None.

How does this change test

New unit tests and existing integraion tests.

@ShiKaiWi ShiKaiWi marked this pull request as draft March 22, 2023 10:51
@ShiKaiWi ShiKaiWi marked this pull request as ready for review March 22, 2023 13:03
@codecov-commenter
Copy link

codecov-commenter commented Mar 22, 2023

Codecov Report

Merging #754 (2dca0bb) into main (cba54b3) will increase coverage by 0.06%.
The diff coverage is 54.40%.

❗ Current head 2dca0bb differs from pull request most recent head dabfa4e. Consider uploading reports for the commit dabfa4e to get more accurate results

@@            Coverage Diff             @@
##             main     #754      +/-   ##
==========================================
+ Coverage   68.24%   68.30%   +0.06%     
==========================================
  Files         296      296              
  Lines       46011    46306     +295     
==========================================
+ Hits        31399    31630     +231     
- Misses      14612    14676      +64     
Impacted Files Coverage Δ
analytic_engine/src/instance/mod.rs 83.33% <ø> (ø)
catalog_impls/src/table_based.rs 83.58% <ø> (ø)
server/src/handlers/influxdb.rs 67.02% <0.00%> (ø)
server/src/handlers/prom.rs 68.44% <0.00%> (ø)
server/src/lib.rs 100.00% <ø> (ø)
server/src/proxy/grpc/prom_query.rs 53.82% <0.00%> (-0.28%) ⬇️
server/src/proxy/grpc/route.rs 0.00% <0.00%> (ø)
server/src/proxy/grpc/sql_query.rs 0.00% <0.00%> (ø)
server/src/proxy/grpc/write.rs 34.76% <0.00%> (-2.06%) ⬇️
src/setup.rs 0.00% <ø> (ø)
... and 9 more

... and 4 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShiKaiWi ShiKaiWi added this pull request to the merge queue Mar 24, 2023
Merged via the queue into apache:main with commit 13a81c2 Mar 24, 2023
chunshao90 pushed a commit to chunshao90/ceresdb that referenced this pull request May 15, 2023
…he#754)

* feat: support split write request to batches for small wal logs

* test: add unit test for the split write requests

* fix: rename the config option name

* chore: add some comments

* chore: explain that `max_bytes_per_write_batch` will break atomicity

* refactor: do maybe_split_write_request
@ShiKaiWi ShiKaiWi deleted the feat-split-write-requests branch May 29, 2023 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants