Releases: Altinity/clickhouse-operator
Releases · Altinity/clickhouse-operator
release-0.15.0
New features:
- Added 90s delay before restarting the stateful set, when ClickHouse server settings are modified. That makes sure ClickHouse server starts with an updated configuration. Before that there was a race condition between configmap updates and statefulset restart. The default timeout can be modified by 'spec.reconciling.configMapPropagationTimeout' property. See example
- Added 'troubleshooting' mode that allows pod to start even if ClickHouse server is failing. In troubleshooting mode liveness check is removed, and extra 'sleep' is added to the startup command. Controlled by 'spc.troubleshoot' property. See example
- Added a cleanup reconcilation logic that will remove k8s object that are labeled by a particular CHI but do not exist in the CHI manifest. The default behaviour can be altered with 'spec.reconciling.cleanup.unknownObjects' . See example
- ZooKeeper manifests have been modified to use 'standaloneEnabled=false' for single node setups as recommended in ZooKeeper documentation. ZooKeeper version has been bumped to 3.6.3
Bug fixes:
release-0.14.0
New features:
- Operator reconciles CHI with the actual state of objects in k8s. In previous releases it compared new CHI and old CHI, but not with k8s state.
- The current reconcile cycle can be interrupted with a new CHI update now. In previous releases user had to wait until reconcile is complete for all nodes..
- Added volumeClaimTemplate annotations. Closes #578
- clickhouse_operator user/password can be stored in a secret instead of a configuration file. Closes: #386
- Added 'excludeFromPropagationLabels' option
- Added readiness check
- LabelScope metrics are removed by default since it causes pods to restart when changing a cluster with circular replication. Closes: #666. If you need those labels, they can be turned on with 'appendScopeLabels' configuration option
- Monitoring of detached parts
- Operator ClusterRole is restricted. Closes #646
- Logging improvements
Bug fixes:
- Fixed a bug when CHI pods could get stuck in ErrImagePull status when wrong image has been used
- Fixed a bug when operator tried to apply schema to non-existing pods, and entered a lengthy retry cycle
Upgrade notes:
- Existing clusters will be restarted with the operator upgrade due to signature change
- Due to ClusterRole change the upgrade with re-applying installation manifest may fail. See #684
release-0.13.5
This is a follow-up release to 0.13.0:
Improvements:
- Readiness check has been removed in favour of custom readiness controller, liveness check has been adjusted for slower startup times
- Operator currently uses 'ready' label for graceful pods creation and modifications. That reduces possible service downtime
- More graceful re-scale and other operations
- New StatefulSet fingerprint
- Operator log has been improved and properly annotated
- Test suite has been upgraded to the recent TestFlows version
Note 1: We recommend using 0.13.5 instead of 0.13.0. We have found that implementation of liveness/readiness probes in 0.13.0 release was not optimal for healthy production operation.
Note 2: ClickHouse clusters will be restarted after operator upgrade due to pod template and labels changes.
release-0.13.0
New features:
- Added liveness (/ping) and readiness (/replicas_status) probes to ClickHouse pods. Those can be overwritten on podTemplate level
- Customizable graceful reconcile logic. Now it is possible to turn-on 'wait' behaviour that would apply configuration changes and upgrades using the following algorithm:
- Host is (optionally) excluded from ClickHouse remote_servers
- Operator (optionally) waits for exclusion to take an effect
- Host is updated
- If it is a new host, schema is created at this stage. That ensures host is not in the cluster until schema is created. Fixes #561
- Host is added back to remote_servers
- Operator (optionally) waits for inclusion to take an effect before moving to the next host.
'Optional' steps are turned-off by default and can be turned on in operator configuration:
reconcileWaitInclude: false
reconcileWaitExclude: false
or enabled for particular CHI update:
spec:
reconciling:
policy: "nowait"|"wait"
- Cluster 'stop' operation now correctly removes CHI from monitoring and deletes LoadBalancer service
- podTemplate metadata is now supported (annotations, labels). Fixes #554
- 'auto' templates are now available. If templating.policy=auto is specified for ClickHouseInstallationTemplate object, those templates are automatically applied to all ClickHouseInstallations.
- Minor changes to ClickHouse default profile settings.
Bug fixes:
- External labels, annotations and finalizers are correctly preserved for services.
- It was possible that finalizer has been inserted multiple times to CHI
** Note: existing ClickHouse clusters will be restarted after operator upgrade because of adding Liveness/Readiness probes **
release-0.12.0
This release includes a number of improvements in order to eliminate unneeded restarts and reduce service downtime:
- Pods are no longer restarted when new shards/replicas are added
- Cluster configuration (remote_servers.xml) is now updated after shards and replicas are added. That reduces a chance to get an error querying distributed tables.
- LoadBalancer node ports are no longer modified on service upgrade. That reduces possible downtime
- Service is re-created if it can not be updated for some reason (e.g. change from ClusterIP to LoadBalancer or vice versa)
- Fixed several race conditions when creating/updating a cluster
release-0.11.0
New features:
- ClickHouse settings can be configured at shard/replica level
- Custom ClickHouse configuration files may be supplied both for 'users.d' and 'config.d' sections using 'users.d/my_users.xml:' syntax.
- Operator is registered as a finalizer for CHI, so CHI deletion is more reliable now.
Bug fixes:
- Operator does not attempt to create a schema when cluster is rolled over initially anymore. Fixes #479 and may improve #478 as well
- Schema creation logic when adding new shards/replicas has been improved to address #308
- Operator restart should not cause ClickHouse pods restart anymore in most cases
Important Note: CHI can not be deleted without operator installed anymore. Also, if operator and CHI is installed in one namespace, and namespace is deleted, it may stuck forever in 'Terminating' state due to an arbitrary deletion sequence.
release-0.10.0
General:
- Kubernetes APIs used by operator have been updated to 1.15. Operator does not use features from new APIs, so it keeps working with Kubernetes 1.12 and above. Operator is tested for compatibility with different k8s versions, and we did not face any issues.
New features:
- Ability to re-scale storage, both by enlarging volume size for existing installations in volumeClaimTemplate, and by adding extra volumes. That requires pod restarts.
- Operator re-creates a StatefulSet if there are changes that can not be applied to an existing object
- Configurable namespaceNamePattern and statefulSetNamePattern (#360 by @qixiaogang), see example
- New exported metrics 'metric.ChangedSettingsHash', 'metric.LongestRunningQuery' and 'active 'label for table sizes.
- Numerous improvements to Grafana dashboard
Bug fixes:
- Set 'internal_replication' to false for single replica clusters (#422 by @yuzhichang)
- Fixed schema creation logic when adding new replica (#385), improved stability in general for different types of objects
- Service selectors did not include namespace, that could lead to a conflict in multi-namespace environment. It is fixed now.
- Operator generated ClickHouse configuration files using standard names, like 'settings.xml'. It might lead to a conflict with user configuration files. Now all generated config files are prefixed with 'chop-generated-'.
Upgrade notes: When operator is upgraded from the previous version, it re-creates statefulsets and pods to ensure they match the installation. The storage should be correctly re-attached. As a side effect it ensures there are no backwards incompatibility issues between operator versions.
release-0.9.9
Bug fixes:
- Fixed ZooKeeper installation manifests that did not mount PV properly. The bug could result in ZooKeeper data loss if ZK pods were restarted.
- Fixed metrics exporter to work across multiple ClickHouse versions -- metric descriptions are temporary removed, Prometheus does not work good if those change.
Other changes:
- Bumped prometheus/golang_client to the latest version
- Added extra logging into chi status field.
release-0.9.8
Bug fixes:
- Fixed multi-threading model for proper handling or concurrent updates
- Fixed schema propagation logic to work with database names that require quoting
Other changes:
- Disabled query_thread_log by default (only applies to new instances)
release-0.9.7
Bug fixes:
- Fixed a bug with schema management queries failing at ClickHouse 20.3.+
- Fixed a bug with logging level configuration not being respected by metrics-exporter
- Fixed schema management logic to use ClickHouse pods directly bypassing LoadBalancer service
- Fixed a bug with host being deleted twice when deleting cluster or shards/replicas
Grafana dashboard has been also improved in this release.