How to add new column to replicated clickhouse table on production without downtime?

[IdeaHunter]

I have 3 node clickhouse production cluster with approx under 1TB of data.

According to documentation:

The ALTER query blocks all reads and writes for the table. In other words, if a long SELECT is running at the time of the ALTER query, the ALTER query will wait for it to complete. At the same time, all new queries to the same table will wait while this ALTER is running.

How clickhouse in production should be upgraded:

1. without freezing app server that would try to send data to clickhouse
2. without losing ability in future to add new node on which schema changes and data would replicate automagically

Is there way to do it in stages without causing downtime and hanging on inserts?

[den-crane]

According to documentation:

It's not quite true. Usually add column is instant and blocks nothing and waits for nothing. You can execute add column without the interruption inserts and queries.

Alter blocks the table (and inserts/select) for the very short time, ~millisecond, to change table metadata.

In some sophisticated cases alter add column can wait for the drop column, but during that time it does not hold the table lock, so it does not blocks inserts and selects.

How clickhouse in production should be upgraded:

Just execute a rolling upgrade, one by one node of clickhouse. Some selects and inserts may fail because when they are interrupted by a node shutdown.

[Cheppali-Reddy]

Hi, is this true for changing the column type ? I mean reads/writes will not be blocked when actual mutation is taking place ?

[IdeaHunter]

My cluster is now in production and i was forced to rename 2 columns, as far as i can tell, no locking was observed, not sure about mutations in general or changing type however.