Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Development Tasks for Log-based Increment Backup, directly on TiKV #29501

Closed
32 of 41 tasks
3pointer opened this issue Nov 5, 2021 · 0 comments · Fixed by #36031, #36197 or #37546
Closed
32 of 41 tasks

Development Tasks for Log-based Increment Backup, directly on TiKV #29501

3pointer opened this issue Nov 5, 2021 · 0 comments · Fixed by #36031, #36197 or #37546
Labels
component/br This issue is related to BR of TiDB. type/feature-request This is a feature requests on the product

Comments

@3pointer
Copy link
Contributor

3pointer commented Nov 5, 2021 •
edited

Description

We want to have a log-based streaming backup system which

  • can scale up with the cluster
  • native, does not require an external processor, to reduce the number of “moving parts”
  • light-weight, continuous backup in should not affect latency
  • low-level, keeping the details of the original cluster as much as possible, reducing backup and restore complexity
  • support point-in-time recovery

This issue tracks our progress.

Milestone

  • Backup the DML data only to s3 with 10G data size and scheduler paused. due date 12/16/21
  • Restore above DML data from s3.
  • Implement local(nfs) storage. due date 12/16/21
  • Support backup & restore DDL. #34621
  • Support backup only DML data with random scheduler enabled.
  • Support TB level workload (TBD: determine incremental flow in workload).
  • Support 10k tables workload (TBD: determine incremental flow in workload).
  • Support TB level workload with random TiKV crash.
  • Support 10k tables workload with random TiKV crash.

Sub Tasks

Log File Structure

Support Log-based Backup

Support Log-based Restore

Support Checksum

  • Support generate checksum data during Backup.
  • Support valid checksum data during Restore.

Trivia
@3pointer 3pointer added the type/feature-request This is a feature requests on the product label Nov 5, 2021
@kennytm kennytm added the component/br This issue is related to BR of TiDB. label Nov 5, 2021
@3pointer 3pointer changed the title [Draft] Development Task for Log-based Increment Backup, directly on TiKV [Draft] Development Tasks for Log-based Increment Backup, directly on TiKV Nov 5, 2021
@3pointer 3pointer changed the title [Draft] Development Tasks for Log-based Increment Backup, directly on TiKV Development Tasks for Log-based Increment Backup, directly on TiKV Nov 22, 2021
This was referenced Dec 16, 2021
Open
Closed
@YuJuncen YuJuncen mentioned this issue Jan 21, 2022
Merged
4 tasks
@joccau joccau mentioned this issue Mar 2, 2022
Closed
@3pointer 3pointer mentioned this issue Mar 7, 2022
Closed
This was referenced Mar 24, 2022
Closed
Closed
@joccau joccau mentioned this issue Apr 12, 2022
Closed
This was referenced Apr 25, 2022
Closed
Closed
Merged
@joccau joccau mentioned this issue May 6, 2022
Merged
12 tasks
ti-chi-bot added a commit to tikv/tikv that referenced this issue May 7, 2022
@YuJuncen @3pointer @ti-chi-bot
backup-stream: Merge the feature branch 3pointer/br-stream into mas…
deb6145 
…ter (#12430)

close #12429, ref pingcap/tidb#29501

Added a new component `backup-stream` for TiKV, which allows TiKV listen the change from raftstore and collect them to external storage.

Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
@joccau joccau mentioned this issue May 9, 2022
Closed
@3pointer 3pointer mentioned this issue May 11, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue May 12, 2022
@joccau
br: support PiTR feature (#34409)
c6fe032 
ref #29501
@joccau joccau mentioned this issue May 13, 2022
Closed
@joccau joccau mentioned this issue May 26, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue May 27, 2022
@joccau
log backup: hidden the log backup and restore point (#34991)
5837686 
ref #29501
@ti-srebot ti-srebot mentioned this issue May 27, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jun 2, 2022
@YuJuncen
backup: add design doc of stream backup (#31888)
7c65326 
ref #29501
@3pointer 3pointer mentioned this issue Jun 2, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jun 3, 2022
@3pointer
docs: rename log backup design document (#35139)
173dd00 
ref #29501
ti-chi-bot pushed a commit that referenced this issue Jul 11, 2022
@joccau
backup-log: support the new feature PiTR- backup/restore log at the t…
354cb53 
…ikv endpoint (#36031)

close #29501
@3pointer 3pointer mentioned this issue Jul 12, 2022
Merged
12 tasks
@joccau joccau mentioned this issue Jul 13, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jul 13, 2022
@joccau
log-backup: get can restored global-checkpoint-ts when support v3 che…
1885ebf 
…ckpoint advance (#36197)

close #29501
ti-chi-bot pushed a commit that referenced this issue Jul 15, 2022
@3pointer
log-backup: add the switch for log backup (#36115)
f47978c 
ref #29501
@joccau joccau mentioned this issue Jul 17, 2022
Merged
12 tasks
@YuJuncen YuJuncen mentioned this issue Jul 18, 2022
Merged
4 tasks
ti-chi-bot pushed a commit that referenced this issue Jul 18, 2022
@YuJuncen
log-backup: added more metrics and hint; fixed a bug may cause inf lo…
9859512 
…op (#36228)

ref #29501
@3pointer 3pointer mentioned this issue Jul 20, 2022
Merged
1 task
ti-chi-bot pushed a commit that referenced this issue Jul 20, 2022
@YuJuncen
log-backup: removed log-backup related configs (#36315)
e2ab431 
ref #29501
@joccau joccau mentioned this issue Jul 28, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jul 29, 2022
@joccau
br: enhance the output infomation (#36676)
7a47538 
ref #29501
@ti-srebot ti-srebot mentioned this issue Jul 29, 2022
Merged
12 tasks
joccau added a commit to ti-srebot/tidb that referenced this issue Jul 29, 2022
@joccau
br: enhance the output infomation (pingcap#36676)
830d3f1 
ref pingcap#29501
ti-chi-bot pushed a commit that referenced this issue Jul 29, 2022
@ti-srebot
br: enhance the output infomation (#36676) (#36706)
edab299 
ref #29501
@3pointer 3pointer reopened this Jul 29, 2022
@joccau joccau mentioned this issue Jul 29, 2022
Merged
9 tasks
@joccau joccau mentioned this issue Sep 8, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Sep 15, 2022
@joccau
log-backup: refuse PiTR if the full-snapshot and stream-log do not co…
fae88ae 
…me from the same cluster (#37546)

close #29501
@joccau joccau mentioned this issue Sep 16, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Sep 19, 2022
@joccau
log-backup: update method for telemetry to check usage of pitr feature (
413a091 
#37896)

ref #29501
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/br This issue is related to BR of TiDB. type/feature-request This is a feature requests on the product
Projects
None yet
Milestone
No milestone
2 participants
@kennytm @3pointer