Zookeeper to ClickHouse Keeper Migration.

[karthikvikki]

Is it possible to do ZK to ClickHouse Keeper Migration without bringing down ClickHouse. We tried stopping merges, distributed sends, fetches, moves, ttl merges, cleanup , replication queues, replication log etc and we stopped all writers to clickhouse. We still could see zxid keeps incrementing sometimes after 15 mins to 1 hour (approximately).

Is it safe to take a zk snapshot (with only leader zk up and running and clickhouse being operational)and push to clickhouse-keeper if zxid does not change for 15 mins continuously.

Whats the best way to achieve zero downtime migration.

[dfbustosus]

Straight from the official Keeper guide (migration section): "Seamless migration from ZooKeeper to ClickHouse Keeper isn't possible. You have to stop your ZooKeeper cluster, convert data, and start ClickHouse Keeper." And a mixed ZK/Keeper cluster is impossible too — the interserver protocols are incompatible even though the client protocol is shared. So true zero-downtime isn't a supported outcome today; what you can get is a short, well-rehearsed write-maintenance window. On your two specific questions:

Why the zxid keeps creeping despite everything stopped. Live ClickHouse servers hold ZooKeeper sessions — ephemeral znodes, heartbeats, replica registration, occasional touches of coordination paths that aren't covered by SYSTEM STOP MERGES / the replication toggles (the docs' own list of background tasks to halt is explicitly non-exhaustive: "for example:"). Session mechanics alone will tick the zxid periodically no matter how quiesced the data paths are. So "zxid stable for 15 minutes" is not an achievable or meaningful safety criterion while the CH fleet is connected — which leads to:

Is the leader-only live snapshot safe? Not as safe as it looks, for a concrete reason: ephemeral nodes baked into a snapshot taken while CH was still running become orphans in Keeper. Keeper has no knowledge of the ZK sessions that owned them, so they never get cleaned up, and things like replica-registration/leader-election ephemerals from the old world linger exactly where the new world expects fresh ones. The documented procedure avoids this by construction:

1. Stop data ingestion on all ClickHouse nodes (you already do this).
2. Stop background tasks on all nodes.
3. Stop all ZooKeeper nodes.
4. Start and stop the leader once more — this forces a consistent snapshot to disk before conversion (the docs call this optional-but-recommended; treat it as required for your case).
5. Run clickhouse-keeper-converter on the leader's data (--zookeeper-logs-dir/--zookeeper-snapshots-dir → --output-dir), or clickhouse keeper-converter if you have the full binary.
6. Copy the converted snapshot to every Keeper node before any of them starts — a node starting without the snapshot can elect itself leader with empty state.
7. Repoint ClickHouse config, start Keeper everywhere, restart ClickHouse.
8. Compare against the metrics baseline you recorded beforehand, then resume ingestion and background tasks.

Minimizing the window in practice — the part that makes this feel near-zero-downtime:

• Rehearse on a copy: copy the leader's version-2 dir aside now, run the converter against the copy, stand up your Keeper cluster in shadow mode, point a staging CH host at it, and time every step. On cutover day you execute a rehearsed script — the actual write-unavailable window is typically minutes (reads from local replicas generally keep serving while coordination is down; it's writes, DDL, and replication activity that wait).
• Pre-deploy Keeper binaries/configs everywhere, pre-stage the new <zookeeper> config in each CH server's config with a deployment switch (config reload or planned restart), and script the SYSTEM STOP ... / SYSTEM START ... bookends.
• If you run multiple ZK clusters, note the converter is strictly one-to-one — consolidating shards onto one Keeper cluster requires merging snapshots with numChildren recalculation, per the same doc.

The one thing I'd not do is the live-leader-snapshot-with-CH-running variant: even when it appears to work, the orphaned-ephemeral risk bites later in ways that are expensive to diagnose (stale leader-election nodes, replicas that think someone else holds locks).