-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to clean remote broken (can't stat metadata.json) #924
Comments
could share
means you have key with the same name as s3->path from config
and check |
This is how config looks like.
Command just dropped root folder for clickhouse backups:
Executed backup list looks ok
But created "clickhouse" directory on AWS S3 side again and broken back again:
|
Are you sure you just run clickhouse-backup list remote twice? and dind't do anything? |
I did created "clickhouse" root directory on AWS S3 side manually, myself. After directory is back, broken is back as well. |
Why did you create it and how? S3 doesn't contains "directories" try to remove again s3:
path: clickhouse instead of |
Ok, I've got it. Seems I misunderstood the concept of "path" parameter from documentation. I created that "clickhouse" directory using S3 browser tool. I believe it could be done on AWS side using gui as well. Everything looks good right now. Thank you for so fast response and support.
|
Hi, I'm using latest version of clickhouse-backup tool and it's really great and powerful and makes life easier, but at the moment I've some minor issue and unable to clean up some broken metadata.
Backup does not already exist.
[root@dba01 tmp]# LOG_LEVEL=debug clickhouse-backup list remote 2024/05/15 17:01:47.595248 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse 2024/05/15 17:01:47.597664 info clickhouse connection success: tcp://localhost:9000 logger=clickhouse 2024/05/15 17:01:47.597691 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse 2024/05/15 17:01:47.603832 info SELECT macro, substitution FROM system.macros logger=clickhouse 2024/05/15 17:01:47.606388 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse 2024/05/15 17:01:47.612704 info SELECT macro, substitution FROM system.macros logger=clickhouse 2024/05/15 17:01:47.740470 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3 2024/05/15 17:01:47.801144 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3 ??? 20/03/2024 10:36:21 remote broken (can't stat metadata.json) 2024/05/15 17:01:47.801570 info clickhouse connection closed logger=clickhouse
Executing clean remote
[root@dba01 tmp]# clickhouse-backup clean_remote_broken 2024/05/15 16:58:22.798382 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse 2024/05/15 16:58:22.800333 info clickhouse connection success: tcp://localhost:9000 logger=clickhouse 2024/05/15 16:58:22.800404 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse 2024/05/15 16:58:22.804907 info SELECT macro, substitution FROM system.macros logger=clickhouse 2024/05/15 16:58:22.806342 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse 2024/05/15 16:58:22.812055 info SELECT macro, substitution FROM system.macros logger=clickhouse 2024/05/15 16:58:22.979528 info clickhouse connection closed logger=clickhouse 2024/05/15 16:58:22.979612 info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse 2024/05/15 16:58:22.981613 info clickhouse connection success: tcp://localhost:9000 logger=clickhouse 2024/05/15 16:58:22.981679 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse 2024/05/15 16:58:22.987974 info SELECT macro, substitution FROM system.macros logger=clickhouse 2024/05/15 16:58:22.990067 info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse 2024/05/15 16:58:22.994326 info SELECT macro, substitution FROM system.macros logger=clickhouse 2024/05/15 16:58:23.126664 info SELECT value FROM
system.
build_optionswhere name='VERSION_INTEGER' logger=clickhouse 2024/05/15 16:58:23.129801 info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='object_storage_type') AS is_object_storage_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies') logger=clickhouse 2024/05/15 16:58:23.139090 info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse 2024/05/15 16:58:23.223063 info done backup= duration=243ms location=remote logger=RemoveBackupRemote operation=delete 2024/05/15 16:58:23.223254 info clickhouse connection closed logger=clickhouse
But it's not working.
Deletion of /tmp/.clickhouse-backup-metadata.cache.S3 is not working as well. This file appears again with next backup list command run.
Please suggest how to fix this issue.
The text was updated successfully, but these errors were encountered: