Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

diff-from backups restore caught in loop #154

Closed
therealnb opened this issue Feb 22, 2021 · 2 comments
Closed

diff-from backups restore caught in loop #154

therealnb opened this issue Feb 22, 2021 · 2 comments

Comments

@therealnb
Copy link

I have a table where I have backed up and pushed to s3 using the diff-from flag. As a test I deleted the 'first' few backups and tried to restore from what was left.

clickhouse@ch-event01-1.sv5:~$ clickhouse-backup list | grep event_alert
- '2021_02_20_00_00_ch-event01.event_alert_local.tar.gz'	2.42 MiB	(created at 20-02-2021 09:22:12)
- '2021_02_20_04_00_ch-event01.event_alert_local.tar.gz'	1.98 MiB	(created at 20-02-2021 09:24:15)
- '2021_02_20_08_00_ch-event01.event_alert_local.tar.gz'	428.49 MiB	(created at 20-02-2021 09:25:40)
- '2021_02_20_12_00_ch-event01.event_alert_local.tar.gz'	170.00 MiB	(created at 21-02-2021 09:00:14)
- '2021_02_20_16_00_ch-event01.event_alert_local.tar.gz'	203.57 MiB	(created at 21-02-2021 09:01:46)
- '2021_02_20_20_00_ch-event01.event_alert_local.tar.gz'	272.38 MiB	(created at 21-02-2021 09:03:00)
- '2021_02_21_00_00_ch-event01.event_alert_local.tar.gz'	172.66 MiB	(created at 21-02-2021 09:04:36)
- '2021_02_21_04_00_ch-event01.event_alert_local.tar.gz'	114.17 MiB	(created at 21-02-2021 09:05:52)
- '2021_02_21_08_00_ch-event01.event_alert_local.tar.gz'	907.23 MiB	(created at 21-02-2021 09:07:20)
- '2021_02_21_12_00_ch-event01.event_alert_local.tar.gz'	174.80 MiB	(created at 21-02-2021 12:31:01)
- '2021_02_21_16_00_ch-event01.event_alert_local.tar.gz'	73.53 MiB	(created at 22-02-2021 09:00:13)
- '2021_02_21_20_00_ch-event01.event_alert_local.tar.gz'	245.51 MiB	(created at 22-02-2021 09:01:42)
- '2021_02_22_00_00_ch-event01.event_alert_local.tar.gz'	2.30 MiB	(created at 22-02-2021 09:03:04)
- '2021_02_22_04_00_ch-event01.event_alert_local.tar.gz'	177.13 MiB	(created at 22-02-2021 09:04:42)
- '2021_02_22_08_00_ch-event01.event_alert_local.tar.gz'	2.02 MiB	(created at 22-02-2021 09:05:59)
clickhouse@ch-event01-1.sv5:~$ clickhouse-backup download 2021_02_22_08_00_ch-event01.event_alert_local
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:08 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:08 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:09 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:09 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:10 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:10 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:10 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:11 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:11 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:12 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:12 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:12 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:13 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:13 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:13 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:14 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:14 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:15 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:15 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:15 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:16 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:16 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:17 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:17 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:17 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:18 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:18 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:19 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:19 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:04:19 Backup '2021_02_22_08_00_ch-event01.event_alert_local' required '2021_02_22_08_00_ch-event01.event_alert_local'. Downloading.
 2.02 MiB / 2.02 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%^C
clickhouse@ch-event01-1.sv5:~$ clickhouse-backup list | grep event_alert
- '2021_02_22_08_00_ch-event01.event_alert_local'	(created at 22-02-2021 12:04:08)
- '2021_02_20_00_00_ch-event01.event_alert_local.tar.gz'	2.42 MiB	(created at 20-02-2021 09:22:12)
- '2021_02_20_04_00_ch-event01.event_alert_local.tar.gz'	1.98 MiB	(created at 20-02-2021 09:24:15)
- '2021_02_20_08_00_ch-event01.event_alert_local.tar.gz'	428.49 MiB	(created at 20-02-2021 09:25:40)
- '2021_02_20_12_00_ch-event01.event_alert_local.tar.gz'	170.00 MiB	(created at 21-02-2021 09:00:14)
- '2021_02_20_16_00_ch-event01.event_alert_local.tar.gz'	203.57 MiB	(created at 21-02-2021 09:01:46)
- '2021_02_20_20_00_ch-event01.event_alert_local.tar.gz'	272.38 MiB	(created at 21-02-2021 09:03:00)
- '2021_02_21_00_00_ch-event01.event_alert_local.tar.gz'	172.66 MiB	(created at 21-02-2021 09:04:36)
- '2021_02_21_04_00_ch-event01.event_alert_local.tar.gz'	114.17 MiB	(created at 21-02-2021 09:05:52)
- '2021_02_21_08_00_ch-event01.event_alert_local.tar.gz'	907.23 MiB	(created at 21-02-2021 09:07:20)
- '2021_02_21_12_00_ch-event01.event_alert_local.tar.gz'	174.80 MiB	(created at 21-02-2021 12:31:01)
- '2021_02_21_16_00_ch-event01.event_alert_local.tar.gz'	73.53 MiB	(created at 22-02-2021 09:00:13)
- '2021_02_21_20_00_ch-event01.event_alert_local.tar.gz'	245.51 MiB	(created at 22-02-2021 09:01:42)
- '2021_02_22_00_00_ch-event01.event_alert_local.tar.gz'	2.30 MiB	(created at 22-02-2021 09:03:04)
- '2021_02_22_04_00_ch-event01.event_alert_local.tar.gz'	177.13 MiB	(created at 22-02-2021 09:04:42)
- '2021_02_22_08_00_ch-event01.event_alert_local.tar.gz'	2.02 MiB	(created at 22-02-2021 09:05:59)
clickhouse@ch-event01-1.sv5:~$ clickhouse-backup download 2021_02_20_04_00_ch-event01.event_alert_local
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:06 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:06 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:06 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:07 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:07 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:08 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:08 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:08 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:09 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:09 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:09 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:10 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:10 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:11 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:11 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:11 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:12 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:12 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:12 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:13 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:13 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:14 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:14 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:14 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:15 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:15 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:16 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:16 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%2021/02/22 12:05:16 Backup '2021_02_20_04_00_ch-event01.event_alert_local' required '2021_02_20_04_00_ch-event01.event_alert_local'. Downloading.
 1.98 MiB / 1.98 MiB [====================================================================================================================================================================================================================================================================================================================] 100.00%^C

The restore gets stuck in an endless loop.

Maybe this isn't a valid use case, but either way clickhouse-backup (v0.6.4) should not loop like this. At a minimum it should report an error and exit.

My preferred behaviour is that it would just download the parts it can (reporting that the chain of backups is broken). This would mean that I can keep doing partial backups, no need for full backup at any point. I can then roll the older backups away on s3 when needed.
@therealnb
Copy link
Author

Please ignore this for now - I think there is a bug in my script, making a diff-from self post.

@therealnb
Copy link
Author

Although, if someone is mad enough to make an upload diff from itself, the code should probably exit or something, not loop.

AlexAkulov added a commit that referenced this issue Feb 25, 2021
@AlexAkulov
fix #154
86268de
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@therealnb