GCS storage error: can't restore object_disk

[IMMORTALxJO]

Hi,
I have a clickhouse cluster installation ( 1 shard and 3 replicas ), deployed in a kubernetes via clickhouse-operator with configured gcs storage. And I have an error during the restore of the table with the gcp storage backend:

2023/09/25 20:31:20.216828 error can't restore object_disk server-side copy data parts 'backup_test.covid19_gcp': /var/lib/clickhouse/disks/gcs/backup/2023-09-25T20-27-44/shadow/backup_test/covid19_gcp/gcs/all_0_6_1/frozen_metadata.txt: invalid object_dist.Metadata: &object_disk.Metadata{Version:0x2, StorageObjectCount:-1, TotalSize:-1, StorageObjects:[]object_disk.StorageObject(nil), RefCount:-1, ReadOnly:false, InlineData:"", Path:"/var/lib/clickhouse/disks/gcs/backup/2023-09-25T20-27-44/shadow/backup_test/covid19_gcp/gcs/all_0_6_1/frozen_metadata.txt"}

full log is here

Could you help me to fix the issue and find the root cause please?

Steps to reproduce:

1. Prepare the cluster and create table with gcs storage, ex. backup_test.covid19_gcp
2. Create remote backup clickhouse-backup create_remote --tables 'backup_test.covid19_gcp'
3. Restore the backup, ex. clickhouse-backup restore_remote --tables 'backup_test.covid19_gcp' 2023-09-25T20-27-44

Versions:

• clickhouse 23.7.4.5
• clickhouse-backup 2.4.1

Clickhouse storage config example from the first replica ( endpoint is different for each replica chi-test-test-0-{0,1,2}-0):

<clickhouse>
    <storage_configuration>
        <disks>
            <gcs>
                <support_batch_delete>false</support_batch_delete>
                <type>s3</type>
                <endpoint>https://storage.googleapis.com/<GOOGLE_STORAGE_BUCKET_FOR_DATA>/test/chi-test-test-0-0-0/</endpoint>
                <access_key_id from_env="GCP_ACCESS_KEY_ID"></access_key_id>
                <secret_access_key from_env="GCP_SECRET_ACCESS_KEY"></secret_access_key>
                <metadata_path>/var/lib/clickhouse/disks/gcs/</metadata_path>
            </gcs>
            <gcs_cache>
                <type>cache</type>
                <disk>gcs</disk>
                <path>/var/lib/clickhouse/disks/gcs_cache/</path>
                <max_size>30Gi</max_size>
            </gcs_cache>
        </disks>
        <policies>
            <default>
                <storage_policy>gcs</storage_policy>
            </default>
            <gcs_main>
                <volumes>
                    <main>
                        <disk>gcs</disk>
                    </main>
                </volumes>
            </gcs_main>
        </policies>
    </storage_configuration>
</clickhouse>

Test table with gcs_main storage:

CREATE DATABASE backup_test ON CLUSTER 'test';

CREATE TABLE backup_test.covid19_gcp ON CLUSTER 'test' (
    date Date,
    location_key LowCardinality(String),
    new_confirmed Int32,
    new_deceased Int32,
    new_recovered Int32,
    new_tested Int32,
    cumulative_confirmed Int32,
    cumulative_deceased Int32,
    cumulative_recovered Int32,
    cumulative_tested Int32
)
ENGINE = ReplicatedMergeTree
ORDER BY (location_key, date)
SETTINGS storage_policy='gcs_main';

INSERT INTO backup_test.covid19_gcp
   SELECT *
   FROM
      url(
        'https://storage.googleapis.com/covid19-open-data/v3/epidemiology.csv',
        CSVWithNames,
        'date Date,
        location_key LowCardinality(String),
        new_confirmed Int32,
        new_deceased Int32,
        new_recovered Int32,
        new_tested Int32,
        cumulative_confirmed Int32,
        cumulative_deceased Int32,
        cumulative_recovered Int32,
        cumulative_tested Int32'
    );

Backup config clickhouse-backup print-config :

general:
    remote_storage: gcs
    max_file_size: 0
    disable_progress_bar: true
    backups_to_keep_local: -1
    backups_to_keep_remote: 7
    log_level: debug
    allow_empty_backups: true
    download_concurrency: 3
    upload_concurrency: 3
    use_resumable_state: true
    restore_schema_on_cluster: test
    upload_by_part: true
    download_by_part: true
    restore_database_mapping: {}
    retries_on_failure: 3
    retries_pause: 30s
    watch_interval: 1h
    full_interval: 24h
    watch_backup_name_template: shard{shard}-{type}-{time:20060102150405}
    sharded_operation_mode: ""
    retriesduration: 30s
    watchduration: 1h0m0s
    fullduration: 24h0m0s
clickhouse:
    username: default
    password: ""
    host: localhost
    port: 9000
    disk_mapping: {}
    skip_tables:
        - system.*
        - INFORMATION_SCHEMA.*
        - information_schema.*
        - _temporary_and_external_tables.*
    skip_table_engines: []
    timeout: 5m
    freeze_by_part: false
    freeze_by_part_where: ""
    use_embedded_backup_restore: false
    embedded_backup_disk: ""
    backup_mutations: true
    restore_as_attach: false
    check_parts_columns: true
    secure: false
    skip_verify: false
    sync_replicated_tables: false
    log_sql_queries: true
    config_dir: /etc/clickhouse-server/
    restart_command: exec:systemctl restart clickhouse-server
    ignore_not_exists_error_during_freeze: true
    check_replicas_before_attach: true
    tls_key: ""
    tls_cert: ""
    tls_ca: ""
    debug: false
s3:
    access_key: ""
    secret_key: ""
    bucket: ""
    endpoint: ""
    region: us-east-1
    acl: private
    assume_role_arn: ""
    force_path_style: false
    path: ""
    object_disk_path: ""
    disable_ssl: false
    compression_level: 1
    compression_format: tar
    sse: ""
    sse_kms_key_id: ""
    sse_customer_algorithm: ""
    sse_customer_key: ""
    sse_customer_key_md5: ""
    sse_kms_encryption_context: ""
    disable_cert_verification: false
    use_custom_storage_class: false
    storage_class: STANDARD
    custom_storage_class_map: {}
    concurrency: 4
    part_size: 0
    max_parts_count: 5000
    allow_multipart_download: false
    object_labels: {}
    debug: false
gcs:
    credentials_file: ""
    credentials_json: ""
    credentials_json_encoded: <GOOGLE_STORAGE_CREDS>
    bucket: <GOOGLE_STORAGE_BUCKET_FOR_BACKUPS>
    path: cluster-test/shard-{shard}
    object_disk_path: cluster-test/s3
    compression_level: 1
    compression_format: tar
    debug: false
    endpoint: ""
    storage_class: STANDARD
    object_labels:
        cluster: test
    custom_storage_class_map: {}
    client_pool_size: 500
cos:
    url: ""
    timeout: 2m
    secret_id: ""
    secret_key: ""
    path: ""
    compression_format: tar
    compression_level: 1
    debug: false
api:
    listen: 0.0.0.0:7171
    enable_metrics: true
    enable_pprof: false
    username: ""
    password: ""
    secure: false
    certificate_file: ""
    private_key_file: ""
    ca_cert_file: ""
    ca_key_file: ""
    create_integration_tables: true
    integration_tables_host: ""
    allow_parallel: false
    complete_resumable_after_restart: true
ftp:
    address: ""
    timeout: 2m
    username: ""
    password: ""
    tls: false
    skip_tls_verify: false
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 4
    debug: false
sftp:
    address: ""
    port: 22
    username: ""
    password: ""
    key: ""
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 4
    debug: false
azblob:
    endpoint_schema: https
    endpoint_suffix: core.windows.net
    account_name: ""
    account_key: ""
    sas: ""
    use_managed_identity: false
    container: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    sse_key: ""
    buffer_size: 0
    buffer_count: 3
    max_parts_count: 5000
    timeout: 15m
custom:
    upload_command: ""
    download_command: ""
    list_command: ""
    delete_command: ""
    command_timeout: 4h
    commandtimeoutduration: 4h0m0s

[Slach]

thanks for detailed report

could you share
content of /var/lib/clickhouse/disks/gcs/backup/2023-09-25T20-27-44/shadow/backup_test/covid19_gcp/gcs/all_0_6_1/frozen_metadata.txt?

[IMMORTALxJO]

@Slach sure, here it is:

chi-test-test-0-0-0:/$ cat /var/lib/clickhouse/disks/gcs/backup/2023-09-25T20-27-44/shadow/backup_test/covid19_gcp/gcs/all_0_6_1/frozen_metadata.txt
2
service%2Dtest%2D0%2D0
default
82161fbc-4ad9-480d-879b-73b6b3159ede