zookeeper::connect failed to handle DNS failures

[dqminh]

We had a case where some nodes in zookeeper cluster were pulled along with their DNS names, so some DNS names declared in zookeeper configuration becomes invalid. This seems to cause the clickhouse tables in some nodes to be readonly.

Our clickhouse version is 1.1.54388

What we see in the logs is that nodes failed to handle DNS errors

<Error> some_tables (ReplicatedMergeTreeRestartingThread): void DB::ReplicatedMergeTreeRestartingThread::run(): Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = No address found: node1.zookeeper, e.what() = No address found
....
<Error> executeQuery: Code: 242, e.displayText() = DB::Exception: Table is in readonly mode, Stack trace:
....
<Error> ConfigReloader: Error loading config from `/etc/clickhouse-server/config.xml': Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Exception: Failed to preprocess config '/etc/clickhouse-server/config.xml': No address found: node1.zookeeper, e.what() = Exception

We do have multiple zookeeper servers declared in config.xml. Any idea why it bailed after failing to query only 1 name ?

[alexey-milovidov]

Yes, in case of DNS errors, it fails too early and no failover is performed.

[alexey-milovidov]

This will help: 17b8e20