Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macro 'uuid' and empty arguments of ReplicatedMergeTree are supported only for ON CLUSTER queries with Atomic database engine #552

Closed
bigdata-spec opened this issue Oct 26, 2022 · 8 comments · Fixed by #585
Assignees
Milestone

Comments

@bigdata-spec
Copy link

[root@localhost backup]# clickhouse-backup restore 2022-10-26T09-02-16 --table default.t_order_rep
2022/10/26 17:20:47 error can't create table `default`.`t_order_rep`: code: 62, message: Macro 'uuid' and empty arguments of ReplicatedMergeTree are supported only for ON CLUSTER queries with Atomic database engine after 1 times, please check your schema depencncies

create table t_order_rep on cluster 'ch_cluster' (
    id UInt32,
    sku_id String,
    total_amount Decimal(16,2),
    create_time  Datetime
 ) engine =ReplicatedMergeTree
   partition by toYYYYMMDD(create_time)
   primary key (id)
   order by (id,sku_id);
   
   
   
 insert into t_order_rep values
(101,'sku_001',1000.00,'2020-06-01 12:00:00'),
(102,'sku_002',2000.00,'2020-06-01 12:00:00'),
(103,'sku_004',2500.00,'2020-06-01 12:00:00'),
(104,'sku_002',2000.00,'2020-06-01 12:00:00'),
(105,'sku_003',600.00,'2020-06-02 12:00:00');
@Slach
Copy link
Collaborator

Slach commented Oct 26, 2022

engine =ReplicatedMergeTree without parameters, means use default_replica_path default_replica_name
https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings/#default_replica_path

add following lines to your config.yaml

general:
  restore_schema_on_cluster: ch_cluster

or
create /etc/clickhouse-server/config.d/default_replica_path.xml

<clickhouse>
<default_replica_path>/clickhouse/tables/{shard}/{database}/{table}</default_replica_path>
</clickhouse>

@bigdata-spec
Copy link
Author

ch_cluster

Thank you very much,I want to backup one by one,so restore_schema_on_cluster: ch_cluster no need,
and follower your advice ,try set
general:
restore_schema_on_cluster: ch_cluster

[root@SP03691 backup]# clickhouse-backup restore 2022-10-26T13-44-16 --table zone_dynamic_ods.cluster_dynamic__message_detail
2022/10/26 21:57:01.956278  info clickhouse connection prepared: tcp://xxxx:9003 run ping logger=clickhouse
2022/10/26 21:57:01.958898  info clickhouse connection open: tcp://xxx:9003 logger=clickhouse
2022/10/26 21:57:01.958927  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2022/10/26 21:57:01.960921  info SELECT * FROM system.disks; logger=clickhouse
2022/10/26 21:57:01.965671  info CREATE DATABASE IF NOT EXISTS `default`  ON CLUSTER 'ch_cluster' ENGINE = Atomic with args [[]] logger=clickhouse
2022/10/26 21:57:02.081422  info CREATE DATABASE IF NOT EXISTS `zone_dynamic_ods`  ON CLUSTER 'ch_cluster' ENGINE = Atomic with args [[]] logger=clickhouse
2022/10/26 21:57:02.189516  info SELECT engine FROM system.databases WHERE name = 'zone_dynamic_ods' logger=clickhouse
2022/10/26 21:57:02.193617  info DROP TABLE IF EXISTS `zone_dynamic_ods`.`cluster_dynamic_ods_kafka_message_detail` ON CLUSTER 'ch_cluster'  NO DELAY logger=clickhouse
2022/10/26 21:57:06.860010  info CREATE DATABASE IF NOT EXISTS `zone_dynamic_ods` ON CLUSTER 'ch_cluster' logger=clickhouse
2022/10/26 21:57:06.966446  info CREATE TABLE zone_dynamic_ods.cluster_dynamic_ods__message_detail UUID '0f9f216d-510b-4a3e-b7f9-84998e33a417' (`vin` String , `col` String  `collect_type` String COMMENT '采集方式', `dt` String COMMENT '日期', `hh` String COMMENT '小时', `message_receive_time` DateTime64(3) COMMENT 'Kafka消息接收时间', `etl_update_time` DateTime64(3) COMMENT 'ETL处理时间', `update_time` DateTime DEFAULT now() COMMENT '更新时间') ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') PARTITION BY toYYYYMMDD(fromUnixTimestamp64Milli(ct)) ORDER BY (uuid, collect_type) TTL toDateTime(fromUnixTimestamp64Milli(ct)) + toIntervalDay(7) SETTINGS index_granularity = 8192 logger=clickhouse
2022/10/26 21:57:06.985987  info clickhouse connection closed logger=clickhouse
2022/10/26 21:57:06.986021 error can't create table `zone_dynamic_ods`.`cluster_dynamic_ods_message_detail`: code: 62, message: Macro 'uuid' and empty arguments of ReplicatedMergeTree are supported only for ON CLUSTER queries with Atomic database engine after 1 times, please check your schema dependencies

info DROP TABLE IF EXISTS zone_dynamic_ods.cluster_dynamic_ods_kafka_message_detail ON CLUSTER 'ch_cluster' NO DELAY logger=clickhouse why drop table ?
error can't create table zone_dynamic_ods.cluster_dynamic_ods_message_detail: code: 62, message: Macro 'uuid' and empty arguments of ReplicatedMergeTree are supported only for ON CLUSTER queries with Atomic database engine after 1 times, please check your schema dependencies

@Slach
Copy link
Collaborator

Slach commented Oct 26, 2022

drop database if exists, to allow create table because you can't create table if it already present on server

looks strange, CREATE TABLE
should look like

CREATE TABLE zone_dynamic_ods.cluster_dynamic_ods__message_detail ON CLUSTER 'ch-cluster'  UUID '0f9f216d-510b-4a3e-b7f9-84998e33a417' (`vin` String , `col` String  `collect_type` String COMMENT '采集方式', `dt` String COMMENT '日期', `hh` String COMMENT '小时', `message_receive_time` DateTime64(3) COMMENT 'Kafka消息接收时间', `etl_update_time` DateTime64(3) COMMENT 'ETL处理时间', `update_time` DateTime DEFAULT now() COMMENT '更新时间') ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') PARTITION BY toYYYYMMDD(fromUnixTimestamp64Milli(ct)) ORDER BY (uuid, collect_type) TTL toDateTime(fromUnixTimestamp64Milli(ct)) + toIntervalDay(7) SETTINGS index_granularity = 8192

need time to reproduce localy

@Slach Slach self-assigned this Oct 26, 2022
@Slach
Copy link
Collaborator

Slach commented Oct 26, 2022

could you share
cat /var/lib/clickhouse/backup/2022-10-26T09-02-16/metadta/zone_dynamic_ods/cluster_dynamic_ods__message_detail.json ?

@bigdata-spec
Copy link
Author

cat /var/lib/clickhouse/backup/2022-10-26T09-02-16/metadta/zone_dynamic_ods/cluster_dynamic_ods__message_detail.json

{
 "table": "cluster_dynamic_ods_kafka_message_detail",
 "database": "zone_dynamic_ods",
 "parts": {
  "default": [
   {
    "name": "20221022_0_240_4"
   },
   {
    "name": "20221022_241_257_2"
   },
   {
    "name": "20221022_258_274_2"
   },
   {
    "name": "20221022_275_290_2"
   },
   {
    "name": "20221022_291_291_0"
   },
   {
    "name": "20221024_0_3009_5"
   },
   {
    "name": "20221024_3010_3580_4"
   },
   {
    "name": "20221024_3581_4032_4"
   },
   {
    "name": "20221024_4033_4502_4"
   },
   {
    "name": "20221024_4503_5314_5"
   },
   {
    "name": "20221024_5315_6062_5"
   },
   {
    "name": "20221024_6063_6802_11"
   },
   {
    "name": "20221024_6803_6832_3"
   },
   {
    "name": "20221024_6833_6875_4"
   },
   {
    "name": "20221024_6876_6889_2"
   },
   {
    "name": "20221024_6890_6893_1"
   },
   {
    "name": "20221024_6894_6894_0"
   },
   {
    "name": "20221025_0_5520_6"
   },
   {
    "name": "20221025_11369_13744_5"
   },
   {
    "name": "20221025_13745_14198_4"
   },
   {
    "name": "20221025_14199_15363_5"
   },
   {
    "name": "20221025_15364_16077_6"
   },
   {
    "name": "20221025_16078_16148_3"
   },
   {
    "name": "20221025_16149_16363_5"
   },
   {
    "name": "20221025_16364_16387_3"
   },
   {
    "name": "20221025_16388_16407_4"
   },
   {
    "name": "20221025_16408_16408_0"
   },
   {
    "name": "20221025_16409_16409_0"
   },
   {
    "name": "20221025_5521_8643_6"
   },
   {
    "name": "20221025_8644_11368_5"
   },
   {
    "name": "20221026_0_6594_6"
   },
   {
    "name": "20221026_6595_9197_5"
   },
   {
    "name": "20221026_9198_9329_3"
   },
   {
    "name": "20221026_9330_9434_3"
   },
   {
    "name": "20221026_9435_9457_2"
   },
   {
    "name": "20221026_9458_9480_2"
   },
   {
    "name": "20221026_9481_9502_2"
   },
   {
    "name": "20221026_9503_9503_0"
   },
   {
    "name": "20221026_9504_9504_0"
   }
  ]
 },
 "query": "CREATE TABLE zone_dynamic_ods.cluster_dynamic_ods_kafka_message_detail UUID '0f9f216d-510b-4a3e-b7f9-84998e33a417' (`vin` String , `col` String , `ct` Int64 , `r` UInt32 , `uuid` String COMMENT 'uuid', `ext` String ,  `message_receive_time` DateTime64(3) COMMENT 'Kafka消息接收时间', `etl_update_time` DateTime64(3) COMMENT 'ETL处理时间', `update_time` DateTime DEFAULT now() COMMENT '更新时间') ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') PARTITION BY toYYYYMMDD(fromUnixTimestamp64Milli(ct)) ORDER BY (uuid, collect_type) TTL toDateTime(fromUnixTimestamp64Milli(ct)) + toIntervalDay(7) SETTINGS index_granularity = 8192",
 "size": {
  "default": 37378656761
 },
 "total_bytes": 37378596960,
 "metadata_only": false
}

clickhouse version 22.8.1
backup version 2.1.1

@Slach Slach added this to the 2.2.0 milestone Oct 26, 2022
@Slach
Copy link
Collaborator

Slach commented Oct 27, 2022

looks weird

CREATE TABLE zone_dynamic_ods.cluster_dynamic_ods_kafka_message_detail 
UUID '0f9f216d-510b-4a3e-b7f9-84998e33a417' 
(`vin` String , `col` String , `ct` Int64 , `r` UInt32 , `uuid` String COMMENT 'uuid', `ext` String ,  `message_receive_time` DateTime64(3) COMMENT 'Kafka消息接收时间', `etl_update_time` DateTime64(3) COMMENT 'ETL处理时间', `update_time` DateTime DEFAULT now() COMMENT '更新时间') 
ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') 
PARTITION BY toYYYYMMDD(fromUnixTimestamp64Milli(ct)) 
ORDER BY (uuid, collect_type) 
TTL toDateTime(fromUnixTimestamp64Milli(ct)) + toIntervalDay(7) SETTINGS index_granularity = 8192

according to
#466

{uuid} shall rewrite in your case
in following code fragment
https://github.com/AlexAkulov/clickhouse-backup/blob/master/pkg/backup/restore.go#L373-L380

@Slach
Copy link
Collaborator

Slach commented Oct 27, 2022

really weird
https://regex101.com/r/7Fu4q7/1

could you remove
"restore_schema_on_cluster: ch-cluster"
from your config.yaml

and share result of following command

LOG_LEVEL=debug clickhouse-backup restore --rm 2022-10-26T09-02-16 --table default.t_order_rep

@Slach
Copy link
Collaborator

Slach commented Oct 27, 2022

https://regex101.com/r/Ka9yps/1
found why ON CLUSTER didn't apply to CREATE TABLE statement when you use this option

but still strange why {uuid} is not replaced without restore_schema_on_cluster

@Slach Slach modified the milestones: 2.2.0, 2.1.3 Oct 27, 2022
@Slach Slach changed the title Macro 'uuid' and empty arguments of ReplicatedMergeTree are supported only for ON CLUSTER queries with Atomic database engine after 1 times Macro 'uuid' and empty arguments of ReplicatedMergeTree are supported only for ON CLUSTER queries with Atomic database engine Oct 27, 2022
@Slach Slach mentioned this issue Dec 19, 2022
@Slach Slach closed this as completed in 3db1c33 Dec 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants