Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sst_importer: add wire extra bytes into the packed size #14312

Merged
merged 7 commits into from Mar 2, 2023

Conversation

YuJuncen
Copy link
Contributor

@YuJuncen YuJuncen commented Mar 1, 2023

What is changed and how it works?

Issue Number: Close #14313

What's Changed:
This PR added some extra bytes for each modification, so the unpacked_size would be nearer to the real raft entry size.

Check List

Tests

  • Manual test (add detailed scripts or steps below)

I have tested it in the internal cluster, if you are a PingCAP employee, you can check it at here. There aren't suitable screenshots to be published, sorry for that.

And I guess this can also be checked by any table with average row size less than 70B (see the comments), but for now I don't have suitable cluster for running such tests...

Release note

None

(This is a fix over the master branch.)

Signed-off-by: hillium <yujuncen@pingcap.com>
Signed-off-by: hillium <yujuncen@pingcap.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 1, 2023

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • tabokie
  • tonyxuqqi

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

src/import/sst_service.rs Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the status/LGT1 Status: PR - There is already 1 approval label Mar 1, 2023
@ti-chi-bot ti-chi-bot added status/LGT2 Status: PR - There are already 2 approvals and removed status/LGT1 Status: PR - There is already 1 approval labels Mar 2, 2023
@tabokie
Copy link
Member

tabokie commented Mar 2, 2023

Better add a test.

Signed-off-by: hillium <yujuncen@pingcap.com>
@BusyJay
Copy link
Member

BusyJay commented Mar 2, 2023

Maybe we should let the engine to tell the size of modification. In raftkv2, writes are encoded using more simple codec and can batch up more modifications than v1.

@YuJuncen
Copy link
Contributor Author

YuJuncen commented Mar 2, 2023

@BusyJay I agree the idea of letting the engine provide some way to calculate "real size" of some write, and I think before we figuring out some ideal interface definition, perhaps we can merge this for solving the problem we have encountered now?

@BusyJay
Copy link
Member

BusyJay commented Mar 2, 2023

LGTM

@tabokie
Copy link
Member

tabokie commented Mar 2, 2023

/merge

@ti-chi-bot
Copy link
Member

@tabokie: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 7385494

@ti-chi-bot ti-chi-bot added the status/can-merge Status: Can merge to base branch label Mar 2, 2023
@ti-chi-bot ti-chi-bot merged commit 4f2430d into tikv:master Mar 2, 2023
@ti-chi-bot ti-chi-bot added this to the Pool milestone Mar 2, 2023
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Mar 2, 2023
close tikv#14313

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #14328.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.6: #14329.

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Mar 2, 2023
close tikv#14313

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pitr restore data fail for "failed: raft entry is too large, region 170, entry size 8616655"
5 participants