ReplicatedMergeTree replica template is hardcoding database and table names after upgrade from 20.7.2.30 to 21.1.2

[alvarotuso]

Describe the unexpected behaviour

On previous ClickHouse versions (earlier than 20.7.2.30) we've created tables using ReplicatedMergeTree with the following replica template:

CREATE TABLE mydb.mytable ON CLUSTER '{cluster}' (
    `my_field` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/replicated/{database}/{table}', '{replica}')
ORDER BY (my_field)
SETTINGS index_granularity = 8192

Retrieving this info back produces the expected result:

SHOW CREATE TABLE mydb.mytable 

CREATE TABLE mydb.mytable 
(
    `my_field` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/replicated/{database}/{table}', '{replica}')
ORDER BY my_field
SETTINGS index_granularity = 8192

The important portion being that the database and table parameters in the replica path remains in template form: '/clickhouse/{installation}/{cluster}/tables/replicated/{database}/{table}'

However, in 21.1.2, the replica path template has a hardcoded table name after creation:

SHOW CREATE TABLE mydb.mytable 

CREATE TABLE mydb.mytable 
(
    `my_field` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/replicated/mydb/mytable', '{replica}')
ORDER BY my_field
SETTINGS index_granularity = 8192

This causes several problems. For example if we try to create a new table using the original one as a base with an AS clause, the new table will try to replicate to the same paths and fail:

CREATE TABLE mydb.myothertable AS mydb.mytable

Received exception from server (version 21.1.2):
Code: 253. DB::Exception: Received from localhost:9000. DB::Exception: There was an error on [mypod:9000]: Code: 253, e.displayText() = DB::Exception: Replica /clickhouse/myinstallation/cluster/tables/replicated/mydb/mytable/replicas/replica-name already exists. (version 21.1.2.15 (official build)).

How to reproduce

• Which ClickHouse server version to use: 21.1.2
• Step by step queries:

CREATE DATABASE mydb ON CLUSTER '{cluster}';

CREATE TABLE mydb.mytable ON CLUSTER '{cluster}' (
    `my_field` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/replicated/{database}/{table}', '{replica}')
ORDER BY (my_field)
SETTINGS index_granularity = 8192;

CREATE TABLE mydb.myothertable AS mydb.mytable;

SQL Error [253]: ClickHouse exception, code: 253, host: localhost, port: 18125; Code: 253, e.displayText() = DB::Exception: There was an error on [mypod:9000]: Code: 253, e.displayText() = DB::Exception: Replica /clickhouse/myinstallation/cluster/tables/replicated/mydb/mytable/replicas/replica-name already exists. (version 21.1.2.15 (official build)) (version 21.1.2.15 (official build))

Expected behavior
The database and table name remain in template form in the table definition, which allows cloning and renaming tables without issues

Error message and/or stacktrace
This is how the problem shows up:

CREATE TABLE mydb.myothertable AS mydb.mytable

Received exception from server (version 21.1.2):
Code: 253. DB::Exception: Received from localhost:9000. DB::Exception: There was an error on [mypod:9000]: Code: 253, e.displayText() = DB::Exception: Replica /clickhouse/myinstallation/cluster/tables/replicated/mydb/mytable/replicas/replica-name already exists. (version 21.1.2.15 (official build)).

Additional context

• We believe this problem might have started right after 20.7.2.30
• Weirdly, tables created before the update still work correctly, so it looks like the code that stores the definition is replacing the database and table name in the string

[UnamedRus]

However, in 21.1.2, the replica path template has a hardcoded table name after creation:

This was done on purpose, because when people tried to RENAME those tables it was breaking replication because {database} and {table} macroses have been updated with new table name.
#6917
#15348

[alvarotuso]

Hey @UnamedRus thanks for the reply
Does that mean that when renaming, the replicas are kept in the old path? I wonder if renaming the path would have worked as well

Anyways, thanks for the explanation! we'll look for a workaround

[UnamedRus]

Does that mean that when renaming, the replicas are kept in the old path? I wonder if renaming the path would have worked as well

Yes. (in new versions of clickhouse)

[den-crane]

@alvarotuso it's impossible to rename key in ZK. ZK does not have such feature.