debug disk_name_4 disk have no parts operation=HardlinkBackupPartsToStorage

[zouwenjie00]

Hi：
I want to backup database in clickhouse cluster A ,and restore it on other clickhouse cluster B, But the clickhouse cluster B table have no data.
when run the command is "./build/linux/amd64/clickhouse-backup restore_remote ck-bak-20230907 -c config-metric-platform-46.yml -s -d --rm"
there have many this message in different table:
2023/09/08 02:54:30.724549 info done backup=ck-bak-20230907 operation=restore table=metric_platform.ods_rdm_tester_efficiency
2023/09/08 02:54:30.724653 debug disk_name_2 disk have no parts operation=HardlinkBackupPartsToStorage
2023/09/08 02:54:30.724693 debug disk_name_4 disk have no parts operation=HardlinkBackupPartsToStorage
2023/09/08 02:54:30.724722 debug disk_name_1 disk have no parts operation=HardlinkBackupPartsToStorage
2023/09/08 02:54:30.724752 debug disk_name_3 disk have no parts operation=HardlinkBackupPartsToStorage

[Slach]

Could you clarify? Is tables restored finally?

Could you share:
SELECT * FROM system.disks FORMAT Vertical
from source and destination clickhouse-server?

[zouwenjie00]

the metadata had restored，but the data not restored；

Cluster A disk is：

Cluster B disk is：

[Slach]

Use clickhouse-client in the future and please don't share information as pictures

So, you have 2 disk in source server and 5 disk in destination server?

Could you share:
SELECT * FROM system.storage_policies from both servers?

Could you please clarify and answer to all my qustions?
how did you run clickhouse-backup?
Is your source clickhouse-server run as standalone or inside docker container?
Did you run clickhouse-backup create outside docker on host?
Is your destination clickhouse-server run as standalone on a separate machine?
How did you move data between the source and destination server?

Could you share a result of the following command from the destination server?
ls -la /data/ssd01/ck1/data/clickhouse/backup/ck-bak-20230907/shadow/*/*/

[zouwenjie00]

Use clickhouse-client in the future and please don't share information as pictures

So, you have 2 disk in source server and 5 disk in destination server?

Could you share: SELECT * FROM system.storage_policies from both servers?

Could you please clarify and answer to all my qustions? how did you run clickhouse-backup? Is your source clickhouse-server run as standalone or inside docker container? Did you run clickhouse-backup create outside docker on host? Is your destination clickhouse-server run as standalone on a separate machine? How did you move data between the source and destination server?

Could you share a result of the following command from the destination server? ls -la /data/ssd01/ck1/data/clickhouse/backup/ck-bak-20230907/shadow/*/*/

1. Yes, I have 2 disk in source server and 5 disk in destination server

2. Could you share:
   SELECT * FROM system.storage_policies from both servers?
   Cluster A:

policy_name            volume_name                disks
default                     default                           default
policy_name_1         volume_name_0            policy_name_1

Cluster B:

policy_name            volume_name                disks
default                     default                           default
policy_name_1         volume_name_0            ['disk_name_1','disk_name_2','disk_name_3','disk_name_4']

3.I run clickhouse-backup in the clusterA Server
first i create and upload to s3:
./build/linux/amd64/clickhouse-backup create_remote ck-bak-20230907 -c config-metric-platform.yml
last i restore the command on cluster A
./build/linux/amd64/clickhouse-backup restore_remote ck-bak-20230907 -c config-metric-platform-46.yml
the config of config-metric-platform.yml is :

general:
  remote_storage: s3
  #remote_storage: none
  backups_to_keep_local: 1
  backups_to_keep_remote: 0
  max_file_size: 1099511627776
  disable_progress_bar: true
  log_level: debug
  allow_empty_backups: false
  download_concurrency: 1
  upload_concurrency: 1
  use_resumable_state: true
  restore_schema_on_cluster: ""
  upload_by_part: true
  download_by_part: true
  restore_database_mapping: {}
clickhouse:
  username: default
  password: "XXX"
  host: 10.18.106.46
  port: 19000
  timeout: 5m
  freeze_by_part: false
  secure: false
  skip_verify: false
  sync_replicated_tables: true
  log_sql_queries: true
  #data_path: "/data/clickhouse-backup"
  #disk_mapping: {"default": "/data/clickhouse-backup"}
  skip_tables:
  - system.*
  - default.*
  - information_schema.*
  - finance_d_ksha.*
  - dm_goa_new.*
  - INFORMATION_SCHEMA.*

s3:
  access_key: "xxxxxx"                   # S3_ACCESS_KEY，<AWS访问密钥> 
  secret_key: "xxxxxx"                   # S3_SECRET_KEY 
  bucket: "datasafe"                       # S3_BUCKET，<存储桶BUCKET名称>                  #  
  endpoint: "http://oss-cn-uat.com"
  force_path_style: true          # S3_FORCE_PATH_STYLE 
  path: "backup"                         # S3_PATH ， <备份路径> 
  debug: true                     # S3_DEBUG
  disable_ssl: false               # S3_DISABLE_SSL 
  part_size: 536870912
  compression_level: 1             # S3_COMPRESSION_LEVEL 
  compression_format: tar          # S3_COMPRESSION_FORMAT 

config-metric-platform-46.yml is :

general:
  remote_storage: s3
  #remote_storage: none
  backups_to_keep_local: 1
  backups_to_keep_remote: 0
  max_file_size: 1099511627776
  disable_progress_bar: true
  log_level: debug
  allow_empty_backups: false
  download_concurrency: 1
  upload_concurrency: 1
  use_resumable_state: true
  restore_schema_on_cluster: ""
  upload_by_part: true
  download_by_part: true
  restore_database_mapping: {}
clickhouse:
  username: default
  password: "XXXXXXX"
  host: 10.18.159.229
  port: 8010
  timeout: 5m
  freeze_by_part: false
  secure: false
  skip_verify: false
  sync_replicated_tables: true
  log_sql_queries: true
 
  skip_tables:
  - system.*
  - default.*
  - information_schema.*
  - finance_d_ksha.*
  - dm_goa_new.*
  - INFORMATION_SCHEMA.*
s3:
  access_key: "xxxxxx"                   # S3_ACCESS_KEY，<AWS访问密钥> 
  secret_key: "xxxxxx"                   # S3_SECRET_KEY 
  bucket: "datasafe"                       # S3_BUCKET，<存储桶BUCKET名称>                  # S3_ENDPOINT   
  endpoint: "http://oss-cn-uat.com"
  force_path_style: true          # S3_FORCE_PATH_STYLE 
  path: "backup"                         # S3_PATH ， <备份路径> 
  debug: true                     # S3_DEBUG
  disable_ssl: false               # S3_DISABLE_SSL 
  part_size: 536870912
  compression_level: 1             # S3_COMPRESSION_LEVEL 
  compression_format: tar          # S3_COMPRESSION_FORMAT 

4.Is your source clickhouse-server run as standalone or inside docker container?

the source clickhouse-server and destination clickhouse-server run as standalone

5. the source clickhouse-server and destination clickhouse-server run as standalone , two shards two replicas on physical machine

6.How did you move data between the source and destination server?

I want to backup to s3 ,and restored from s3,i dont konw it is can be come true ? so i ask you for help me.

[Slach]

Yes, I have 2 disk in source server and 5 disk in destination server

clickhouse-backup can't balance disks and during download if disk not present in destination server then default disk will use
look details
#561

Based on shared system.storage_policies, during download all data should be stored in default disk

As a workaround
try to add to config-metric-platform-46.yml

clickhouse:
  disk_mapping:
    policy_name_1: /path/to/disk_name_1

and add <load_balancing>least_used</load_balancing> to your <storage_configuration><policies><policy_name_1><volumes><volume_name_0>...

close issue as duplicated with #561