Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: Merge the feature branch 3pointer/br-stream into master #12427

Closed
wants to merge 124 commits into from
Closed

*: Merge the feature branch 3pointer/br-stream into master #12427

wants to merge 124 commits into from

Conversation

YuJuncen
Copy link
Contributor

@YuJuncen YuJuncen commented Apr 25, 2022 •
edited

What is changed and how it works?

Issue Number: close #12429
Also ref pingcap/tidb#29501.

What's Changed:

Added a new component `backup-stream` for TiKV, which allows TiKV listen the change from raftstore and collect them to external storage.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
    (TBD)
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM

Release note 

Added the support for log backup.

YuJuncen and others added 30 commits December 8, 2021 12:51
@YuJuncen @kennytm
br-stream: the metadata client (tikv#11507)
b3d082f 
ref pingcap/tidb#30098

* stream-meta: initial commit

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: refactor metadata

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: basic store abstraction

Signed-off-by: Yu Juncen <yu745514916@live.com>

* stream-meta: add etcd store

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: basic framework for testing

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: some tests

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: added some test cases

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: make ci happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: move store related stuffs into store mod

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: add some metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: add txn into the store interface

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br-stream: address some comments

Signed-off-by: Yu Juncen <yu745514916@live.com>

* Apply suggestions from code review

Co-authored-by: kennytm <kennytm@gmail.com>
Signed-off-by: Yu Juncen <yu745514916@live.com>

* br-stream: make clippy happy and ref tidb#30098

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br-stream: make clippy happy again

Signed-off-by: yujuncen <yujuncen@pingcap.com>

Co-authored-by: kennytm <kennytm@gmail.com>
@3pointer
BackupStream: implement the basic endpoint and observer logic (tikv#1…
8fabbb3 
…1611)

close pingcap/tidb#30446

* br-stream: implement endpoint & observer basic logic

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* make format

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix test

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix test

Signed-off-by: 3pointer <luancheng@pingcap.com>

* try fix test

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix test

Signed-off-by: 3pointer <luancheng@pingcap.com>

* try fix

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix test

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>
@joccau
merge master to br-steam
b938def 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@3pointer
br: implement router for backup stream (tikv#11619)
c6e3574 
* br: implement router for backup stream

Signed-off-by: 3pointer <luancheng@pingcap.com>

* make clippy

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>
@joccau
Merge remote-tracking branch 'remotes/upstream/br-stream' into merge-…
b0336ba 
…master
@joccau
Merge remote-tracking branch 'remotes/upstream/master' into merge-master
065e0b2 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Debug stream log to test flow
36ea463 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Debug stream log to test flow
5019ea5 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Make format
3fa62bc 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Merge remote-tracking branch 'origin/debug-br-stream' into debug-br-s…
9498c24 
…tream

Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Deal the comments in pr, ref tikv#11679
87b8d7e 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@YuJuncen @3pointer @kennytm
br/stream: Flush files (tikv#11652)
0437c08 
* br: implement router for backup stream

Signed-off-by: 3pointer <luancheng@pingcap.com>

* make clippy

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* br/stream: basic routers

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br-stream: basic local temporary file logic

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: added local file buffers

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: added some comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: remove replace

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix license

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix the wrap_key

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: make max-temp-file-size configable

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: added assertions for test

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: remove meta data key

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: revert cargo.toml

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: revert export.rs

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix lint

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: update kvproto

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Update components/br-stream/src/router.rs

Co-authored-by: kennytm <kennytm@gmail.com>
Signed-off-by: yujuncen <yujuncen@pingcap.com>

* router: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: remove 'z' prefix

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: (maybe) fix test

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: (maybe) fix tests

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix unstable tests

Signed-off-by: yujuncen <yujuncen@pingcap.com>

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: kennytm <kennytm@gmail.com>
@joccau
Merge branch 'br-stream' of https://github.com/tikv/tikv into debug-b…
fae2004 
…r-stream

Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Test and debug from backup stream log
35f614a 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Test and debug from backup stream log
ff504f2 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Make format
9fa0088 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Add log-prefix backup stream
077c7c2 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
modify debug log format
f7660ed 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@YuJuncen
br/stream: finer locking
f7d6907 
Signed-off-by: yujuncen <yujuncen@pingcap.com>
@YuJuncen
br-stream: added some log
5375ada 
Signed-off-by: yujuncen <yujuncen@pingcap.com>
@YuJuncen
br-stream: added some log
e5f5f21 
Signed-off-by: Yu Juncen <yu745514916@live.com>
@YuJuncen
br/stream: fix multi flush
4b21784 
Signed-off-by: yujuncen <yujuncen@pingcap.com>
@3pointer
Merge pull request #3 from joccau/debug-br-stream
e5b7283 
br: backup stream: Modify log print format
@kennytm
br-stream: don't clone the key & value in encode_event()
220a7f7 
Signed-off-by: kennytm <kennytm@gmail.com>
@YuJuncen
move the copyright to righy place
9f3547f 
Signed-off-by: yujuncen <yujuncen@pingcap.com>
@YuJuncen
Merge pull request #2 from YuJuncen/optimize
affb4bc 
br-stream: reduce the lock of tables; added some metrics
@kennytm
Merge branch 'br-stream' into less-cloning
288ca7b
@joccau
br stream flush
504ca3e 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@3pointer
Merge pull request #4 from kennytm/less-cloning
8cae6df 
br-stream: don't clone the key & value in encode_event()
@joccau
Add tmp code
a079130 
Signed-off-by: joccau <zak.zhao@pingcap.com>
YuJuncen and others added 18 commits March 9, 2022 11:18
@YuJuncen
use local thread pool for downloading (#30)
52c4903
@YuJuncen
br-stream: added store error to the store (#32)
9245791 
* br-stream: added store error to the store

* revert kvproto

* Update Cargo.lock
@YuJuncen
Scan on Leader Change (#31)
456db6b 
* initial try

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* refined

Signed-off-by: Yu Juncen <yu745514916@live.com>

* fix end key being empty

Signed-off-by: Yu Juncen <yu745514916@live.com>

* fix tokio panic

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* add metrics for flushing

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* concurrent flush file

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* refined some metrics and logs

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* move download out of gRPC thread

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* Revert "move download out of gRPC thread"

This reverts commit ed66b0d.
@3pointer
refine br-stream to backup-stream (#34)
d17d3db 
* refine br-stream to backup-stream
@3pointer
merge to master
4f47f87 
Signed-off-by: 3pointer <luancheng@pingcap.com>
@YuJuncen
Allow local storage support directory and partition the log files by …
179cf9e 
…table (#33)

* move the files to some dirs

Signed-off-by: Yu Juncen <yu745514916@live.com>

* set failpoints when enabled

Signed-off-by: Yu Juncen <yu745514916@live.com>

* udpate the file name to {date}-{min_ts}-{uuid}

Signed-off-by: Yu Juncen <yu745514916@live.com>

* don't make test needs failpoints enabled

Signed-off-by: Yu Juncen <yu745514916@live.com>

* ignore already exists

Signed-off-by: Yu Juncen <yu745514916@live.com>

* create parent dirs for restoring

Signed-off-by: Yu Juncen <yu745514916@live.com>

* added version spec

Signed-off-by: Yu Juncen <yu745514916@live.com>

* run cargo check

Signed-off-by: Yu Juncen <yu745514916@live.com>
@joccau
br: support checkSum during stream restore dml kv-events (#35)
0f084d3 
* support checksum in file level for br-stream

Signed-off-by: joccau <zak.zhao@pingcap.com>

* update kvproto package and maintain testcase.

Signed-off-by: joccau <zak.zhao@pingcap.com>
@YuJuncen
eliminate the block call in ticker (#36)
cd8c7ae 
* eliminate the block call in ticker

* revert some change

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@YuJuncen
added integration test framework (#22)
5daf570 
* added integration test framework

Signed-off-by: Yu Juncen <yu745514916@live.com>

* added more cases

Signed-off-by: Yu Juncen <yu745514916@live.com>

* fix tests

Signed-off-by: Yu Juncen <yu745514916@live.com>

* added async commit testing

Signed-off-by: Yu Juncen <yu745514916@live.com>

* fix build

Signed-off-by: Yu Juncen <yu745514916@live.com>
@YuJuncen
Fix size leakage and build (#37)
0556434 
* use a vector to contain flushing files

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix size leakage

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix build

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@YuJuncen
fix the upload part of S3 storage (#39)
a3e9302 
* fix the upload part of S3 storage

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* tuned the test case

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* remove unnesseary doctests

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix typo

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@YuJuncen @3pointer
Use min ts of mem lock (#38)
b506cf5 
* check the min_ts and advance the resolved ts then

Signed-off-by: Yu Juncen <yu745514916@live.com>

* check min_ts when updating checkpoint ts

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix build

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* always sync with fs

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* Update components/backup-stream/src/endpoint.rs

Co-authored-by: 3pointer <qdlc2010@gmail.com>

Co-authored-by: 3pointer <qdlc2010@gmail.com>
@YuJuncen
set uuid for header to prevent raftstore merging (#40)
3a3b1f1 
* set uuid for header to prevent raftstore merging

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* move the uuid to `make_request`

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@YuJuncen
Adapt error code for endpoint::Error and implement the contextual e…
01ad3eb 
…rror (#41)

* adapt error code for endpoint::Error

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix typo

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@joccau
br: support pause/resume stream task (#42)
5c290bb 
* support pause/resume task

Signed-off-by: joccau <zak.zhao@pingcap.com>
@YuJuncen
report error to PD server (#43)
1b69362 
* adapt error code for endpoint::Error

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix typo

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* added basic (unformed) error uploading

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix build, remove dead code

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* do pause

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@3pointer
backup-stream: fix some bugs in log backup (#45)
a8bd783 
* backup-stream: fix some bugs in log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>

* implemenet clear rpc && make clippy

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix

Signed-off-by: 3pointer <luancheng@pingcap.com>

* remove Mutex

* fix lost the key not in ts range

Signed-off-by: 3pointer <luancheng@pingcap.com>
@YuJuncen
Refactor resolver (#44)
96a196f 
* refactor resolver

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* run cargo fix

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* move back some test cases

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix register range too late

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* make region subscription hold all region metadata

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix build

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* reduce the frequency of reregistering

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* make more debug info

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* remove destory lock range

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* move subscription track to sperated file; fix incremental scan when resuming

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix some tests

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* added some metrics, and revert some changes

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* fix typo

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* added retry for getting snapshot

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot
Copy link
Member

@YuJuncen: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot added contribution Type: PR - From contributors size/XXL labels Apr 25, 2022
@3pointer
backup-stream: update kvproto to master (#48)
1a02de0 
Signed-off-by: 3pointer <luancheng@pingcap.com>
@YuJuncen
Added a new observer type 'pitr' (#46)
72d6f67 
* added a new observer type 'pitr'

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* make `ChangeObserver` a sum type

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

* remove replica code

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
@YuJuncen
Copy link
Contributor Author

Make another branch to be used as base.

@YuJuncen YuJuncen closed this Apr 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
contribution Type: PR - From contributors needs-rebase release-note size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support Capturing Data Change and Save Them to External Storages
5 participants
@YuJuncen @ti-chi-bot @3pointer @joccau @kennytm