Sometimes the number of zk-nodes suddenly increases significantly even when the bulk write QPS remains unchanged

[xlvchao]

When using ClickHouse, if there is a sudden surge of traffic, the bulk write QPS will also increase, and in this case, an increase in the number of zk-nodes is considered normal.

But sometimes the number of zk-nodes suddenly increases significantly even when the bulk write QPS remains unchanged. Has anyone encountered this situation?

I need a reasonable explanation!

[den-crane]

Replication issues/mutations issues/outages of nodes. Monitor system.replication_queue at least count(), replication lag (max_absolute_replica_delay). I saw a spike from 50 thousands znodes to 5 million because of adding a new node to the cluster.

[Algunenano]

Den's explanations are all plausible. I've also seen spikes due to adding a new replica: as it's replicating old data, new (small) parts being ingested in the meantime(.

Check replication status in all replicas and check what nodes are being created / maintained to investigate your specific circumstances.

[xlvchao]

Den's explanations are all plausible. I've also seen spikes due to adding a new replica: as it's replicating old data, new (small) parts being ingested in the meantime(.

Check replication status in all replicas and check what nodes are being created / maintained to investigate your specific circumstances.

@den-crane @Algunenano But I did not add new nodes to the new cluster. How can this be explained?

[Algunenano]

It's just a guess. Since you are not providing any data about the kind of nodes that are being created and not deleted, there is no way to provide you with anything but guesses.