Fix possible "The local set of parts of X doesn't look like the set of parts in ZooKeeper" error

[azat]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix possible "The local set of parts of X doesn't look like the set of parts in ZooKeeper" error (if DROP fails during removing znodes from zookeeper)

Detailed description / Documentation draft:
If during removing replica_path from zookeeper, some error occurred
(zookeeper goes away), then it may not remove everything from zookeeper.

And on DETACH/ATTACH (or server restart, like stress tests does in the
analysis from this comment 1), it will trigger an error:

The local set of parts of table test_1.alter_table_4 doesn't look like the set of parts in ZooKeeper:

Fix this, by removing "metadata" at first, and only after this
everything else, this will avoid this error, since on ATTACH such table
will be marked as read-only.

Cc: @alesapin

[alesapin]

Fix looks reasonable, but I don't understand why tests fail.

[azat]

Fix looks reasonable, but I don't understand why tests fail.

Forget to remove remote_replica_path itself.

[azat]

Functional stateless tests (ubsan) — Timeout, fail: 0, passed: 0

2021-10-30 00:12:21 clickhouse-test: error: unrecognized arguments: --check-zookeeper-session

CIGithubActions / FunctionalStatelessTestDebug (pull_request) Failing after 59m — FunctionalStatelessTestDebug

Traceback (most recent call last):
  File "functional_test_check.py", line 281, in <module>
    state, description, test_results, additional_logs = process_results(result_path, server_log_path)
  File "functional_test_check.py", line 179, in process_results
    with open(status_path, 'r', encoding='utf-8') as status_file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/actions-runner/_work/_temp/stateless_debug/result_path/check_status.tsv'

[alesapin]

No failures related to changes.

[tavplubix]

Fix this, by removing "metadata" at first, and only after this
everything else, this will avoid this error, since on ATTACH such table
will be marked as read-only.

It will not be marked as readonly, metadata node will be re-created and table will try to start up:

ClickHouse/src/Storages/StorageReplicatedMergeTree.cpp

Lines 440 to 452 in 4398a35

	/// In old tables this node may missing or be empty
	String replica_metadata;
	const bool replica_metadata_exists = current_zookeeper->tryGet(replica_path + "/metadata", replica_metadata);
	if (!replica_metadata_exists || replica_metadata.empty())
	{
	/// We have to check shared node granularity before we create ours.
	other_replicas_fixed_granularity = checkFixedGranularityInZookeeper();
	ReplicatedMergeTreeTableMetadata current_metadata(*this, metadata_snapshot);
	current_zookeeper->createOrUpdate(replica_path + "/metadata", current_metadata.toString(),
	zkutil::CreateMode::Persistent);