log-backup: support encrypt backing up files #15084

YuJuncen · 2023-07-06T13:42:55Z

What is changed and how it works?

Issue Number: Ref #15083

What's Changed:
This PR added encryption support to the tempfiles.
Also, it implemented AsyncWrite for EncrypterWriter and DecrypterWriter.

This PR doesn't pass the Encryptor to Endpoint, hence it won't impact the behavior of TiKV for now. The connection between them will be established in next PR.

Check List

Tests

Unit test
Integration test

Release note

Now, temporary files generated by log backup will be encrypted when encryption is enabled.

Signed-off-by: hillium <yujuncen@pingcap.com>

…r-cahce-in-mem

Signed-off-by: hillium <yujuncen@pingcap.com>

…r-cahce-in-mem

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot · 2023-07-06T13:42:57Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3pointer
Leavrth

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: hillium <yujuncen@pingcap.com>

3pointer

rest LGTM

components/encryption/src/io.rs

3pointer · 2024-04-28T10:59:18Z

components/encryption/src/io.rs

@@ -514,6 +612,15 @@ impl CrypterCore {
    }

    pub fn do_crypter(&mut self, buf: &[u8]) -> IoResult<&[u8]> {
+        assert_eq!(


maybe debug_assert_eq!

If this happens in a production cluster, the data will be corrupted. I think panic here instead will be better than writing wrong data.

Signed-off-by: Yu Juncen <yu745514916@live.com>

YuJuncen · 2024-05-06T05:40:46Z

/merge

ti-chi-bot · 2024-05-06T05:40:48Z

@YuJuncen: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

YuJuncen · 2024-05-06T05:41:30Z

/merge cancel

YuJuncen · 2024-05-06T06:00:22Z

/merge

ti-chi-bot · 2024-05-06T06:00:25Z

@YuJuncen: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-05-06T06:00:27Z

This pull request has been accepted and is ready to merge.

Commit hash: 86592ad

overvenus · 2024-05-07T07:37:23Z

components/encryption/src/io.rs

+            concat!(
+                "unreachable: try to override the encrypted content when there is pending async writing.",
+                "(canceled future? concurrency call to `write`?)"
+            )


Suggested change

concat!(

"unreachable: try to override the encrypted content when there is pending async writing.",

"(canceled future? concurrency call to `write`?)"

)

"unreachable: try to override the encrypted content when there is \

pending async writing. (canceled future? concurrency call to \

`write`?)"

YuJuncen and others added 28 commits April 24, 2023 10:10

basic change (not compile)

e68f359

Signed-off-by: hillium <yujuncen@pingcap.com>

tombstone for async version

986fd8e

Signed-off-by: hillium <yujuncen@pingcap.com>

impl for AsyncRead and AsyncWrite

8eabc67

Signed-off-by: hillium <yujuncen@pingcap.com>

basic async version

fe62b33

Signed-off-by: hillium <yujuncen@pingcap.com>

allow it being deleted

f4b1505

Signed-off-by: hillium <yujuncen@pingcap.com>

added a test case; make clippy happy

5f57703

Signed-off-by: hillium <yujuncen@pingcap.com>

make format

937f115

Signed-off-by: hillium <yujuncen@pingcap.com>

remove hard max

15a6201

Signed-off-by: hillium <yujuncen@pingcap.com>

fix tests

2d6ea0d

Signed-off-by: hillium <yujuncen@pingcap.com>

make clippy happy

61384ff

Signed-off-by: hillium <yujuncen@pingcap.com>

make format

2dfc335

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'master' into pitr-cahce-in-mem

f0f9cba

added more metrics

9a71163

Signed-off-by: hillium <yujuncen@pingcap.com>

change minimal size from 8M to 1M, added some metrics

8bffb01

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'pitr-cahce-in-mem' of github.com:YuJuncen/tikv into pit…

696820d

…r-cahce-in-mem

set the I/O type for tokio runtime (even may miss of statistics)

04cf567

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'master' into pitr-cahce-in-mem

4b57d55

address comments

60662c6

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'pitr-cahce-in-mem' of github.com:YuJuncen/tikv into pit…

1bdfb24

…r-cahce-in-mem

call done after making sure the flush success

7d9e81f

Signed-off-by: hillium <yujuncen@pingcap.com>

fixed a bug that may cause file not synced

3d41a59

Signed-off-by: hillium <yujuncen@pingcap.com>

added a "closed" state for file

2ce967f

Signed-off-by: hillium <yujuncen@pingcap.com>

version one: random key

96407b7

Signed-off-by: hillium <yujuncen@pingcap.com>

experiment

89972db

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'master' of https://github.com/tikv/tikv into enc-files

cc874de

Signed-off-by: hillium <yujuncen@pingcap.com>

fix build

365e2ad

Signed-off-by: hillium <yujuncen@pingcap.com>

make writting safer

4c11ff9

Signed-off-by: hillium <yujuncen@pingcap.com>

added more comments

3927400

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot bot added the release-note label Jul 6, 2023

YuJuncen requested review from 3pointer and Leavrth April 2, 2024 04:14

Leavrth approved these changes Apr 9, 2024

View reviewed changes

ti-chi-bot bot added the status/LGT1 Status: PR - There is already 1 approval label Apr 9, 2024

YuJuncen added 3 commits April 9, 2024 18:14

fix a misleading comment

f974f57

Signed-off-by: hillium <yujuncen@pingcap.com>

remove another misleading comment

5f19998

Signed-off-by: hillium <yujuncen@pingcap.com>

fix status inconsisteny when encountered errors

bc2fa86

Signed-off-by: hillium <yujuncen@pingcap.com>

3pointer reviewed Apr 28, 2024

View reviewed changes

YuJuncen added 2 commits April 29, 2024 11:23

Merge branch 'master' of https://github.com/tikv/tikv into enc-files

a88dbcb

address comment

a90fdcd

Signed-off-by: Yu Juncen <yu745514916@live.com>

YuJuncen requested a review from 3pointer April 29, 2024 03:40

3pointer approved these changes Apr 29, 2024

View reviewed changes

ti-chi-bot bot added status/LGT2 Status: PR - There are already 2 approvals and removed status/LGT1 Status: PR - There is already 1 approval labels Apr 29, 2024

ti-chi-bot bot added the status/can-merge Status: Can merge to base branch label May 6, 2024

Merge branch 'master' into enc-files

86592ad

ti-chi-bot bot removed the status/can-merge Status: Can merge to base branch label May 6, 2024

ti-chi-bot bot added the status/can-merge Status: Can merge to base branch label May 6, 2024

Merge branch 'master' into enc-files

a2a48dd

overvenus self-requested a review May 6, 2024 06:46

Merge branch 'master' into enc-files

e1697a9

overvenus approved these changes May 7, 2024

View reviewed changes

ti-chi-bot bot merged commit 608fc17 into tikv:master May 7, 2024
7 checks passed

ti-chi-bot bot added this to the Pool milestone May 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log-backup: support encrypt backing up files #15084

log-backup: support encrypt backing up files #15084

YuJuncen commented Jul 6, 2023 •

edited

ti-chi-bot bot commented Jul 6, 2023 •

edited

3pointer left a comment

3pointer Apr 28, 2024

YuJuncen Apr 29, 2024

YuJuncen commented May 6, 2024

ti-chi-bot bot commented May 6, 2024

YuJuncen commented May 6, 2024

YuJuncen commented May 6, 2024

ti-chi-bot bot commented May 6, 2024

ti-chi-bot bot commented May 6, 2024

overvenus May 7, 2024

log-backup: support encrypt backing up files #15084

log-backup: support encrypt backing up files #15084

Conversation

YuJuncen commented Jul 6, 2023 • edited

What is changed and how it works?

Check List

Release note

ti-chi-bot bot commented Jul 6, 2023 • edited

3pointer left a comment

Choose a reason for hiding this comment

3pointer Apr 28, 2024

Choose a reason for hiding this comment

YuJuncen Apr 29, 2024

Choose a reason for hiding this comment

YuJuncen commented May 6, 2024

ti-chi-bot bot commented May 6, 2024

YuJuncen commented May 6, 2024

YuJuncen commented May 6, 2024

ti-chi-bot bot commented May 6, 2024

ti-chi-bot bot commented May 6, 2024

overvenus May 7, 2024

Choose a reason for hiding this comment

YuJuncen commented Jul 6, 2023 •

edited

ti-chi-bot bot commented Jul 6, 2023 •

edited