From 1569f3a18f05e4577d04eda39fc39e7c46b462b4 Mon Sep 17 00:00:00 2001 From: hillium Date: Tue, 4 Mar 2025 13:50:41 +0800 Subject: [PATCH 1/4] refined br-log-arch --- br/br-log-architecture.md | 58 +++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index c75f3593dc75c..a50b340e3620c 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -108,18 +108,24 @@ Log backup generates the following types of files: . ├── v1 │   ├── backupmeta -│   │   ├── {min_restored_ts}-{uuid}.meta -│   │   ├── {checkpoint}-{uuid}.meta +│   │   ├── ... +│   │   └── {resolved_ts}-{uuid}.meta │   ├── global_checkpoint -│   │   ├── {store_id}.ts -│   ├── {date} -│   │   ├── {hour} -│   │   │   ├── {store_id} -│   │   │   │   ├── {min_ts}-{uuid}.log -│   │   │   │   ├── {min_ts}-{uuid}.log -├── v1_stream_truncate_safepoint.txt +│   │   └── {store_id}.ts +│   └── {date} +│      └── {hour} +│         └── {store_id} +│            ├── ... +│            └── {min_ts}-{uuid}.log +└── v1_stream_truncate_safepoint.txt ``` +Among them: + +- `backupmeta` contains the metadata of the backup. The `resolved_ts` in the file name indicates the progress of the backup: data before this TSO has been fully backed up. However, please note that this data only reflects the progress of some shards, so this number is not very meaningful to users. +- `global_checkpoint` stores the global progress of the backup: it records the latest point in time that can be restored to using `br restore point`. +- The `{date}/{hour}` directory stores the backup data for the corresponding date. Please note that when cleaning up storage, you should use `br log truncate` instead of manually deleting the data, because the metadata points to this data though deleted manually, and manually deleting it will cause problems. + The following is an example: ``` @@ -129,24 +135,24 @@ The following is an example: │   │   ├── ... │   │   ├── 435213818858112001-e2569bda-a75a-4411-88de-f469b49d6256.meta │   │   ├── 435214043785779202-1780f291-3b8a-455e-a31d-8a1302c43ead.meta -│   │   ├── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta +│   │   └── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta │   ├── global_checkpoint │   │   ├── 1.ts │   │   ├── 2.ts -│   │   ├── 3.ts -│   ├── 20220811 -│   │   ├── 03 -│   │   │   ├── 1 -│   │   │   │   ├── ... -│   │   │   │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log -│   │   │   │   ├── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log -│   │   │   ├── 2 -│   │   │   │   ├── ... -│   │   │   │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log -│   │   │   │   ├── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log -│   │   │   ├── 3 -│   │   │   │   ├── ... -│   │   │   │   ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log -│   │   │   │   ├── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log -├── v1_stream_truncate_safepoint.txt +│   │   └── 3.ts +│   └── 20220811 +│      └── 03 +│         ├── 1 +│         │   ├── ... +│         │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log +│         │   └── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log +│         ├── 2 +│         │   ├── ... +│         │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log +│         │   └── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log +│         └── 3 +│            ├── ... +│            ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log +│            └── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log +└── v1_stream_truncate_safepoint.txt ``` From 8e8c0e2a3947992a5eec4ef00eb908e579609ab5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=B1=B1=E5=B2=9A?= <36239017+YuJuncen@users.noreply.github.com> Date: Tue, 4 Mar 2025 17:17:42 +0800 Subject: [PATCH 2/4] Apply suggestions from code review Co-authored-by: Lilian Lee --- br/br-log-architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index a50b340e3620c..9f79f183c2cba 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -120,11 +120,11 @@ Log backup generates the following types of files: └── v1_stream_truncate_safepoint.txt ``` -Among them: +Explanation of the backup file directory structure: -- `backupmeta` contains the metadata of the backup. The `resolved_ts` in the file name indicates the progress of the backup: data before this TSO has been fully backed up. However, please note that this data only reflects the progress of some shards, so this number is not very meaningful to users. -- `global_checkpoint` stores the global progress of the backup: it records the latest point in time that can be restored to using `br restore point`. -- The `{date}/{hour}` directory stores the backup data for the corresponding date. Please note that when cleaning up storage, you should use `br log truncate` instead of manually deleting the data, because the metadata points to this data though deleted manually, and manually deleting it will cause problems. +- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards, so its significance is limited. +- `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. +- `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore. The following is an example: From d645b70ad0ee7a7e0362c49c8e27ba751c5fb772 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 4 Mar 2025 17:52:19 +0800 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: BornChanger <97348524+BornChanger@users.noreply.github.com> --- br/br-log-architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index 9f79f183c2cba..08bc0466c1d89 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -122,7 +122,7 @@ Log backup generates the following types of files: Explanation of the backup file directory structure: -- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards, so its significance is limited. +- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. - `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. - `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore. From 93b6524c3a48f6c9c50894f4d7b82ae4118675e5 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 4 Mar 2025 18:16:19 +0800 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Grace Cai --- br/br-log-architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index 08bc0466c1d89..fedd2c98fc305 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -122,7 +122,7 @@ Log backup generates the following types of files: Explanation of the backup file directory structure: -- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. +- `backupmeta`: stores backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. - `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. - `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore.