feat: highly available clickhouse deployment #27
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The previous clickhouse deployment was setup to run as a single node cluster preventing us from having a highly available deployment. The database schema has been adjusted to create a replicated database so that Clickhouse will automatically replicate schema changes and data between replicas. This schema change is a breaking change so the environments will be re-created. I've also adjusted the ordering of audit logs to use the timestamp as a secondary sort column so we can maintain strict ordering of audit logs. There's still benefits in maintaining the hourly bucketing so Clickhouse can skip over entire hours of data through indexes. The apiserver has been updated to use the new ordering. The migration script will now wait for all replicas in the cluster to come up before executing migrations.
scotwells
requested review from
JoseSzycho,
cc-datum,
drewr,
ecv and
zachsmith1
Need operational visibility into Clickhouse Keeper so we can monitor the component and understand replication delay between cluster replicas.
ecv
previously approved these changes
Jan 19, 2026
Were using a replicated database now. Need to account for some replicas reporting the same metrics.
These pod disruption budgets ensure the clickhouse system is able to maintain quorum by only letting a single replica of the database and the keeper instance be offline at any given time.
Contributor
Author
|
Had an incident happen in production over the weekend after we deployed this change. To help mitigate the same issue in the future, I added resource requirements, topology spread constraints, and pod disruption budgets for the workloads to ensure they're hardened for production. I also had to replace the table engine with the ReplicatedReplacingMergeTree table engine so that data is replicated across replicas. I misunderstood the Clickhouse documentation originally and thought that using a replicated database engine was enough to configure replication of DDL schema changes and the underlying data. Also adjusted the audit log pipeline dashboard to account for replication being enabled now. |
ecv
approved these changes
Jan 20, 2026
scotwells
deleted the
feat/create-highly-available-replicated-database
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The previous clickhouse deployment was setup to run as a single node cluster preventing us from having a highly available deployment.
The database schema has been adjusted to create a replicated database so that Clickhouse will automatically replicate schema changes and data between replicas.
Important
This schema change is a breaking change so the environments will be re-created.
I've also adjusted the ordering of audit logs to use the timestamp as a secondary sort column so we can maintain strict ordering of audit logs. There's still benefits in maintaining the hourly bucketing so Clickhouse can skip over entire hours of data through indexes. The apiserver has been updated to use the new ordering.
The migration script will now wait for all replicas in the cluster to come up before executing migrations.
Relates to datum-cloud/enhancements#536