reconfig support for CH Keeper

[myrrc]

Resolves: #45886
Depends on: eBay/NuRaft#432, ClickHouse/NuRaft#63
Related: #50251

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support ZooKeeper reconfig command for CH Keeper with incremental reconfiguration which can be enabled via keeper_server.enable_reconfiguration setting.
Support adding servers, removing servers, and changing server priorities.

Documentation entry for user-facing changes

Docs PR: ClickHouse/clickhouse-docs#1164

• Documentation is written (mandatory for new features)

[robot-ch-test-poll]

This is an automated comment for commit 59ad2d9 with description of existing statuses. It's updated for the latest CI running
The full report is available here
The overall status of the commit is 🟡 pending

Check name	Description	Status
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	🟡 pending
ClickHouse build check	Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process	🟢 success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	🟢 success
Docs Check	Builds and tests the documentation	🟡 pending
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	🟢 success
Mergeable Check	Checks if all other necessary checks are successful	🟢 success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	🟢 success
Push to Dockerhub	The check for building and pushing the CI related docker images to docker hub	🟢 success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	🟢 success
Style Check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	🟢 success

[clickhouse-ci]

This is an automatic comment. The PR descriptions does not match the template.

Please, edit it accordingly.

The error is: Category 'Backward incompatible' is not valid

[alesapin]

@antonio2368 , please also take a look when it will be finished.

[antonio2368]

We have a conflict with #50091 which will add an HTTP endpoint for cluster management.

I'm okay with reconfig compatibility from ZK clients but I would make the code much simpler by pushing the reconfig requests directly to update configuration queue and not let it go through RAFT - same what we do in #50091

The problem is that NuRaft already handles internally the cluster configuration and it would be hell to keep those things in sync with the way you implemented it.

I would put this PR on hold until #50091 is merged and then we can keep only part for adding ReconfigRequest as an alternative (more limited) version for HTTP endpoint.

[myrrc]

Things that need to be addressed:

• Backoff interval while trying to apply configuration update in dispatcher thread. It's currently set to 50ms which may be too slow (I've encountered situations where replica fails to push configuration update in seconds which would be a lot of unnecessary network traffic; this situation is usually encountered while removing a leader from large clusters).
• Setting /keeper/config node on server start when there are no reconfigurations. In Zookeeper this node is populated on first leader elections. However, at the point of leader elections we don't get configuration like in Zookeeper, so I update /keeper/config from state manager's config as a workaround:
  • Nuraft allows us to transfer a payload with every message though sending configuration with every message is obviously wrong
  • Updating from state manager is correct if any reconfiguration has been issued, because state manager would have this information (transferred from leader) at the point we query it on initialization
  • If there was no reconfiguration, /keeper/config may possibly get out of sync if e.g. a new node has incorrect configuration. However, incorrect configuration should be handled on other levels so I believe it's not of an issue as server is broken anyway.

Currently test_reconfig_remove_add is flaky. In this test we remove a leader from a 3-node cluster and simultaneously add another replica. Maybe I'd have to prohibit having joining and leaving filled if I don't find a way to fix it