error can't freeze table: code: 32

[bunkiedc]

HI,
I'm getting this error, code 32 and I can't find what it's trying to tell me.
I'm running a 3 node cluster. It seems that when I run the command on two of the nodes via clickhouse-client, it works, but the third
gives the error. I restarted the cluster and the problem migrated to a different node (only one of the three). The previous
table also completes fine. This is running in docker environment, and the file it mentioned, increment.txt, looks to be
fine and is accessible by all nodes. This behavior just started a few days ago, and it has been fine for months.

ClickHouse client version 22.2.2.1.
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 22.5.1 revision 54455.

2023/01/21 07:53:45.904707 info SELECT value FROM system.build_options where name='VERSION_INTEGER'
2023/01/21 07:53:45.909975 info SELECT * FROM system.disks;
2023/01/21 07:53:45.919022 info ALTER TABLE health_metrics.dave_home_inventory FREEZE WITH NAME '4af0db556b814b20bdc5dd0ccb66534d';
2023/01/21 07:53:45.941649 info done backup=2023-01-21T07:53:45 operation=create table=health_metrics.dave_home_inventory
2023/01/21 07:53:45.941734 info ALTER TABLE health_metrics.dave_home_messages FREEZE WITH NAME 'd552ee568710410988a6e027cdad69a9';
2023/01/21 07:53:46.276057 info done backup=2023-01-21T07:53:45 operation=create table=health_metrics.dave_home_messages
2023/01/21 07:53:46.276681 info ALTER TABLE health_metrics.dave_home_metrics FREEZE WITH NAME '6343e1c83e514609b879943750f891ea';
2023/01/21 07:53:46.282449 error can't freeze table: code: 32, message: File /var/lib/clickhouse/shadow/increment.txt is empty. You must fill it manually with appropriate value. backup=2023-01-21T07:53:45 operation=create table=health_metrics.dave_home_metrics
2023/01/21 07:53:46.306290 info done backup=2023-01-21T07:53:45 duration=24ms location=local operation=delete
2023/01/21 07:53:46.309997 info SELECT value FROM system.build_options where name='VERSION_INTEGER'
2023/01/21 07:53:46.317066 info SELECT * FROM system.disks;
2023/01/21 07:53:46.326192 info Clean /var/lib/clickhouse/shadow
2023/01/21 07:53:46.327658 error can't freeze table: code: 32, message: File /var/lib/clickhouse/shadow/increment.txt is empty. You must fill it manually with appropriate value.

When I do it from the clickhouse-client, this is returned:
clickhouse.1 :) ALTER TABLE health_metrics.dave_home_messages FREEZE WITH NAME '86462f9b7c4e40f0ae7bfa041f8e9c4a';

ALTER TABLE health_metrics.dave_home_messages
FREEZE WITH NAME '86462f9b7c4e40f0ae7bfa041f8e9c4a'

Query id: 302792b0-1aae-438d-9b96-5a01196fdd53

0 rows in set. Elapsed: 0.004 sec.

Received exception from server (version 22.5.1):
Code: 32. DB::Exception: Received from localhost:9000. DB::ParsingException. DB::ParsingException: File /var/lib/clickhouse/shadow/increment.txt is empty. You must fill it manually with appropriate value.. (ATTEMPT_TO_READ_AFTER_EOF)

[bunkiedc]

Well, I looked at the file from the container, and indeed it was empty. I put the number that was in the file on one of the other node and now it's running, but it isn't creating the shadow directory any longer.

Oh, where can I find the definitions of these error codes for future issues?

[Slach]

Looks weird, and looks like bug in clickhouse-server not in clickhouse-backup

Could you provide more context?
Which host os do you use to run clickhouse-server container?
Could you upgrade your clickhouse-server to 22.3 or 22.8?

[bunkiedc]

well, I just ran it again, 3 times, and it's now creating the shadow directory. The way it's implemented, it can run on any of the 3 nodes as a one-shot container, so I was worried it was symptomatic of the initial issue I posted here. I'll keep an eye on it over the next few days to be sure it's really is happy! Thanks for responding so quickly, and thanks for the app!

[bunkiedc]

I failed again last night, so I dug into this morning. What's going on is this... My clickhouse cluser is using docker, and the nodes are replicated as clickhouse.1, .2 and .3, and all serve at port 9000. The docker service is doing a round robin for each access to port 9000, so there are times when the backup is running on one of the nodes that has clickhouse, but it may be that the actual backup command runs on a different clickhouse node. I validated this by getting a failure, then looking at all the nodes to see if the shadow was there, I fail on the copy from the shadow to my replicated storage backup space. So I guess I'm going to have to look for the shadow copy on all the nodes or perhaps there's another way of the backup knowing which clickhouse node the backup is running against? Does the backup keep a record of which clickhouse node it's talking so that it could provide that back to the script?

[Slach]

You need to run ALTER TABLE ... FREEZE ... only one server
so, you need to connect to one docker container directly instead of use docker service with round robin