Repository navigation
Important upgrade notes
Check the following before you upgrade the Operator to 1.3.0. The upgrade steps are in the upgrade guide.
Leader election is enabled by default
Starting with version 1.3.0, leader election for the Operator Deployment is on by default. When you upgrade the Operator to version 1.3.0, you can disable it via the PSO_LEADER_ELECTION_ENABLED environment variable.
See Configurable leader election for the Operator.
Enable backups on asynchronous clusters
Version 1.3.0 requires backups so a failover on an asynchronous replication cluster can recover transactions from the old primary. An asynchronous cluster with backup.enabled set to false fails validation, and reconciliation stops.
The status shows an error like this:
Invalid configuration: When 'mysql.clusterType' is set to 'async', 'backup.enabled' must be true unless 'unsafeFlags.asyncFailoverWithPossibleDataLoss' is enabled. Without backups a failover cannot recover the transactions stranded on the old primary
A cluster that was valid on Operator 1.2.0 hits this check after the upgrade. Before you upgrade, do one of the following:
- Enable and configure backups, and set
backup.enabledtotrue. - Set
unsafeFlags.asyncFailoverWithPossibleDataLosstotrueto continue without backups. A failover then cannot recover transactions that remain on the old primary.
Group Replication clusters are not part of this check.
Take a new base backup when point-in-time recovery is enabled
If point-in-time recovery (PITR) is enabled, a recovery from a backup taken before the upgrade can fail with the following error:
54edf529-d2cc-11f1-aebd-668621c1d52d:1-94 --stop-datetime=2026-10-28T13:50:49]
2026-10-28T13:52:40.183741217Z 0 [Info] [Recovery] streaming binlog binlogs1/binlog.000001
2026-10-28T13:52:40.692443772Z 0 [Info] [Recovery] pitr failed: apply binlogs: mysql failed: exit status 1, stderr: ERROR 1396 (HY000) at line 1907: Operation ALTER USER failed for 'configurator'@'%'
Do one of the following:
- Take a new base backup after the upgrade. Use that backup as the starting point for later point-in-time recovery. This is the recommended option.
- Force the restore. Set
pitr.forcetotrueon the restore object. This passes--forceto the MySQL client and ignores SQL errors during binlog replay, which can hide data loss. See Ignore SQL errors during binlog replay.
Release highlights
Automatic MySQL configuration tuning
By default, the Operator tunes only two settings from the MySQL Pod's memory limit: innodb_buffer_pool_size and max_connections. Every other setting uses the same server default, unless a database administrator (DBA) sets it by hand. Without that tuning, a cluster either keeps defaults that don't align with the Pod's allocated resources, or runs on manual settings that go stale when someone resizes the Pod.
Starting with version 1.3.0, the Operator calculates a MySQL configuration from the Pod's CPU and memory, the workload you run, and the MySQL version. The calculation covers the InnoDB buffer pool and redo log, session buffers, connection limits, and replication settings for the cluster type you run. When you change the Pod's resources or the workload, the Operator calculates those settings again.
The automatic configuration tuning is enabled for new clusters deployed with Operator 1.3.0 and crVersion 1.3.0 or higher. On a cluster you upgrade, turn it on in the Custom Resource.
With MySQL configuration calculated automatically, you:
- Have MySQL settings matched to the Pod's CPU, memory, and workload
- Use the resources you already allocate to the MySQL container efficiently
- Have an updated configuration when you change
mysql.resourcesor the workload
If you define MySQL settings in spec.mysql.configuration, a ConfigMap, or a Secret, the Operator applies those settings and skips automatic tuning.
Read Automatic configuration tuning for the workflow, setup, and gotchas.
Apply dynamic MySQL configuration without restarts
Until now, every MySQL option change in the Custom Resource, ConfigMap, or a Secret triggered a rolling restart. That is correct for static variables, but many MySQL settings are dynamic and can take effect with the SET GLOBAL command. Restarts on every change dropped connections, flushed buffer pools, and made peak-hour tuning risky.
The Operator now applies dynamic variables at runtime. Update mysql.configuration in the Custom Resource and the Operator runs SET GLOBAL on every ready Pod without restarting it. If you set the value yourself with SET GLOBAL from the Pod, the Operator does not overwrite it. To keep that change after a Pod restart, add it to the Custom Resource, ConfigMap, or a Secret. If the declared value already matches, there is no restart.
Static variables, runtime-blocked changes, and removed options still trigger a rolling restart. You can tune a busy cluster and respond to an incident without a configuration restart.
Learn more in the dynamic configuration documentation.
Zero data loss failover for asynchronous clusters
Asynchronous replication copies transactions to replicas with a delay. If the primary fails, for example after an out-of-memory kill or a hardware fault, the transactions it committed but hasn't yet sent to any replica exist only in its binary log. Until now, promoting a replica in that situation meant losing those transactions.
Starting with version 1.3.0, the Operator and Orchestrator recover those transactions before promoting a new primary. The replica chosen for promotion fetches the binary logs it is missing from the failed primary and applies them. Only then does it become the new primary. The logs are read through the xtrabackup sidecar, so the failed primary's Pod must be reachable and its data volume intact, but mysqld on that Pod can be down.
You decide what happens when recovery can't finish. You set a timeout (one hour by default) and a policy for when it expires. By default, the cluster stays read-only until you act. Alternatively, you can let the Operator promote a replica and accept the possible data loss. You can also force a promotion yourself with an annotation. The cluster's status conditions and events show whether a failover is in progress or blocked.
With zero data loss failover, you gain:
- No lost transactions on primary failure — the new primary has everything the failed primary wrote to its binary log
- Predictable outcomes — the cluster waits and then follows the policy you chose instead of promoting a replica that is behind
- Visibility — status conditions and events report the state of every failover
- Declarative setup — you configure it in the cluster Custom Resource, like the rest of the cluster
This feature requires backups to be enabled for the cluster. Asynchronous replication is a tech preview, so we don't recommend it for production environments yet.
To learn more, read our documentation.
Switch replication type on a running cluster (tech preview)
Now you can switch between Group Replication and asynchronous replication on a running cluster. Change spec.mysql.clusterType, and the Operator completes the switch. You do not pause the cluster and bring it back yourself.
The Operator converts the topology in place: it tears down the old replication setup, recreates the MySQL and proxy workloads, and bootstraps the new type on the same PersistentVolumeClaims. Users, Secrets, and the HAProxy Service stay in place, so applications keep the same connection endpoint if they already use HAProxy.
Switching the replication type on a running cluster is in tech preview stage. Try it in a staging cluster before you use it in production.
Use the switch when you need to:
-
Move to Group Replication for stronger consistency, native failover, or InnoDB ClusterSet. Before you switch from asynchronous replication to Group Replication, confirm every replica is caught up and take a backup from the primary. The Operator can overwrite transactions that exist only outside
mysql-0. -
Move to asynchronous replication for lower write latency, more than nine MySQL instances, or to avoid Group Replication limits on large transactions.
If a switch from Group Replication to asynchronous replication fails during bootstrap, you cannot switch back. Make sure to make a backup before the switch. This way you can restore the cluster from that backup.
The switch causes downtime while Pods restart. It applies to the entire cluster.
Learn more in the Change replication type documentation.
Rejoin ClusterSet replicas automatically or by annotation
With this release, the Operator automates ClusterSet replica rejoin. If the replica is paused, stopped, or crashed, the Operator reboots and rejoins it. If the replica is healthy but the ClusterSet replication channel is down after a long primary outage, choose a safe window and trigger rejoin by annotating the ClusterSet Custom Resource. The Operator then creates a rejoin Job, tracks its progress, and removes the annotation. On failure, it deletes the job and emits the ClusterSetMemberRejoinFailed warning. After you fix the cause, you can retry.
You decide when to rejoin, and the Operator runs the Job. For the steps, see Rejoin a replica cluster.
Track replication lag for InnoDB ClusterSet
You can now see how far a ClusterSet replica is behind the primary in ClusterSet status with kubectl, instead of connecting to MySQL Shell. Use it to:
- Confirm a replica is catching up before you run switchover
- Keep read traffic off a site that is still behind
- Spot growing lag as it happens
Learn more in Custom resource statuses.
Size asynchronous clusters the way you use them
Asynchronous replication uses a primary and replicas. The nodes do not form a voting group. When the primary fails, Orchestrator promotes a replica, so an even number of instances is a valid high-availability setup.
You can now set mysql.size to 2, 4, 6, or any larger even count on an asynchronous cluster without unsafeFlags.mysqlSize. A single-instance cluster (mysql.size: 1) still requires the flag, because there is no replica to fail over to.
Use the replica count that fits your workload. Orchestrator still requires an odd size of 3 or greater. Set unsafeFlags.orchestratorSize to run fewer than 3 Orchestrator Pods.
Asynchronous replication remains a tech preview feature.
Learn more in Architecture and Horizontal scaling.
Improved bootstrapping behavior for async replication clusters
When a replica joins or rejoins an asynchronous cluster, MySQL copies the data with a native InnoDB clone (CLONE INSTANCE). With this release, joining works as follows:
- The
pt-heartbeatsidecar does not start during the clone. It queries MySQL and startspt-heartbeatonly after the clone is no longer in progress and thesys_operatordatabase is present. The sidecar waits until the clone finishes. A finished clone also restarts MySQL Pods. Ifpt-heartbeatexits in that short window, the script starts it again in the same container. A process that ran for a while and then exits is still treated as a failure, and the container restarts. - The clone no longer has a fixed time limit. Bootstrap polls clone progress and stops the clone only when it makes no progress for
BOOTSTRAP_CLONE_STALL_TIMEOUT(default 900 seconds). A clone that keeps moving runs until it finishes. A frozen clone, or one that cannot be observed for the whole stall window, is stopped. That fails the startup probe, kubelet restarts the container, and the clone is retried.
This behavior applies to clusters with crVersion set to 1.3.0. The Operator ignores BOOTSTRAP_CLONE_TIMEOUT for those clusters. The fixed timeout remains only for clusters with crVersion older than 1.3.0.
To turn off the stall check, set BOOTSTRAP_CLONE_STALL_TIMEOUT to 0. See Define environment variables for cluster components for guidelines.
Encrypted binlogs for point-in-time recovery
Encrypted backups and data-at-rest encryption protect the database and the base backup. They do not protect the binary logs that the Binlog Server stores in object storage for point-in-time recovery. Those files are written in plaintext, which leaves a gap if you use S3-compatible storage without server-side encryption, or if you need the full recovery path encrypted.
You can now encrypt binlogs before Percona Binlog Server uploads them. Create a Kubernetes Secret with your encryption keys, reference it in the cluster Custom Resource, and set the storage encryption options. The Operator validates the Secret, encrypts each new file on the way to the bucket, and decrypts it during restore. Files already in the bucket stay as they are and are applied as-is, so you can turn encryption on without emptying storage.
During key rotation, keep all previous keys in the Secret. Keep the Secret in the cluster if you turn binlog encryption off, so the Operator can still read existing encrypted binlogs. If a key is lost, those files cannot be recovered.
Point-in-time recovery remains a tech preview feature.
To learn more, read Binlog encryption.
Point-in-time recovery for encrypted clusters
You can now restore an encrypted cluster to a specific point in time. The Operator uses your existing Vault Secret during restore, so tablespaces and binlogs stay encrypted, and you can still roll the database forward to the GTID or time you need.
Point-in-time recovery remains a tech preview. Read more in the point-in-time recovery documentation.
Backup options for start and suspend deadlines
This release adds options for how long a backup can wait.
backup.startingDeadlineSeconds sets how long a backup can wait to start. The backup waits when the cluster is not ready, or when another backup for this cluster is still running. If the Operator does not create the Job before this time expires, the backup state becomes Error.
backup.suspendedDeadlineSeconds sets how long a suspended backup can wait to resume. The Operator suspends a backup when the cluster is unhealthy, and resumes it when the cluster is ready again. A resumed backup starts from the beginning. If the backup does not resume before this time expires, the Operator marks it Failed and deletes the suspended Job.
The default for both options is 0, which means the Operator does not enforce a deadline. Set them in the cluster Custom Resource for every backup, or on a PerconaServerMySQLBackup object for one backup. The value on the backup object takes precedence.
spec:
backup:
startingDeadlineSeconds: 900
suspendedDeadlineSeconds: 900Read Set a waiting time for a backup to start and Suspend backups on an unhealthy cluster to learn more.
Use your existing cert-manager ClusterIssuer for TLS
If you already run cert-manager with a cluster-wide issuer, you can point Percona Operator for MySQL at that ClusterIssuer resource. Percona Server for MySQL certificates are then signed under the same public key infrastructure (PKI) you use for the rest of the platform.
To use a ClusterIssuer, configure the cluster as follows:
spec:
tls:
issuerConf:
name: my-org-issuer
kind: ClusterIssuer
group: cert-manager.ioThe Operator resolves the issuer by kind, creates the leaf Certificate, and lets cert-manager sign and renew it. The Operator does not create a separate CA or overwrite your issuer.
MySQL TLS then follows the same issuer, renewal, and trust policy as your other workloads. You keep one source of truth for certificates instead of an Operator-managed CA in every database namespace.
Learn more in the cert-manager TLS documentation.
Use your own CA to verify TLS with S3-compatible storage
You can now verify TLS to S3-compatible storage with the certificate authority (CA) your organization already trusts. That matters when you use certificates from your corporate PKI, and security policy does not let you skip certificate checks.
Create a Secret with the CA and reference it in the storage configuration. Keep TLS verification enabled. The Operator uses that CA for backups and restores. You can also use the same CA for point-in-time recovery (tech preview) by referencing it in the Binlog Server storage configuration.
spec:
backup:
enabled: true
storages:
minio:
type: s3
verifyTLS: true
s3:
bucket: S3-BACKUP-BUCKET-NAME-HERE
region: us-west-2
credentialsSecret: ps-cluster1-s3-credentials
endpointUrl: https://minio-service:9000
caBundle:
name: minio-ca-bundle
key: ca.crt
pitr:
enabled: true
binlogServer:
storage:
s3:
bucket: my-binlogs
credentialsSecret: my-s3-secret
region: us-east-1
prefix: binlogs
endpointUrl: https://minio-service:9000
caBundle:
name: minio-ca-bundle
key: ca.crtWith this configuration you get:
- TLS verification that stays enabled for S3-compatible storage
- The CA your organization already trusts
- The same check for backups, restores, and point-in-time recovery
Read more in Configure TLS verification with custom certificates for S3 storage.
Rotate TLS certificates without restarting MySQL Pods
When you update the server certificate (the leaf) in the TLS Secret, the Operator reloads the updated leaf into running mysqld processes when the certificate authority (CA) stays the same. MySQL Pods stay up. Existing connections keep the previous certificate until they reconnect. New connections use the new one.
HAProxy, MySQL Router, and Orchestrator still restart, because they read certificates once at startup. Changing the CA still restarts MySQL Pods. If you replace the CA, follow the CA rotation procedure in the Update certificates documentation.
This change gives you:
- MySQL availability during routine certificate renewal
- Fewer database Pod restarts on a cert-manager schedule
- The same workflow: keep
ca.crtunchanged and replace onlytls.crtandtls.key, or let cert-manager renew the leaf
Security builds for the Operator you already run
The Operator has access to your cluster credentials and configuration. A vulnerability in the Operator image can expose that data even when your MySQL cluster is fully patched.
Security builds close that gap on a release you already run. Apply the fix without waiting for the next Operator version or retesting a full upgrade. A security build is a new Operator image that fixes a High or Critical vulnerability in your current Operator version.
Each release now comes with the following tags:
1.3.0is the original release. Use it for a standard deployment.1.3.0-1is the same release image with a numbered tag. Use it when you need a fixed digest.1.3.0-2,1.3.0-3, and later numbers are security builds of that release. Each numbered tag stays on its own image. The first security build is1.3.0-2.1.3.0-latestis the newest image for this release. The tag moves, and it points at the most recent image.
Only the Operator image is updated. Database cluster components stay on the versions from the original release. Vulnerabilities in those images are outside a security build.
To apply a security build, patch the Operator Deployment and set the image to the security tag you chose. This restarts the Operator Pod. The cluster keeps running.
Read more in the security builds documentation.
Manage application users from the Custom Resource
Previously, you could only create application users by connecting to MySQL after the cluster was ready and running SQL, or adding extra steps to your CI/CD pipeline. You can now declare users in the cluster Custom Resource and let the Operator create them, grant privileges, and rotate passwords.
For each user, set the name. Optionally, set the hosts they can connect from, the databases they can access, and the MySQL grants. Supply a password in a Secret, or omit it and the Operator generates one and stores it in the <cluster-name>-user-<username> Secret. Change the Secret to rotate the password. The Operator applies it on the next reconcile.
You can create users when you provision the cluster and update them the same way you update any other configuration.
Learn more in Manage users.
Configurable leader election for the Operator
You can now tune leader election for the Operator Deployment with environment variables. This helps when leader election fails, for example in a high-latency or resource-constrained cluster.
Leader election is enabled by default. When you upgrade, you can turn it off with PSO_LEADER_ELECTION_ENABLED environment variable if the Deployment has one replica.
- Set
PSO_LEADER_ELECTION_LEASE_DURATION,PSO_LEADER_ELECTION_RENEW_DEADLINE, andPSO_LEADER_ELECTION_RETRY_PERIODto adjust lease timing. - Set
PSO_LEADER_ELECTION_ENABLEDto turn leader election on or off. - Set
PSO_LEADER_ELECTION_LEASE_NAMEto use a custom Lease for the leader lock.
Learn more about these environment variables in the documentation.
Persistent logging
Container logs disappear when a Pod restarts, so the MySQL error log from a crash is often gone by the time you look. Now the Operator keeps that error log on the MySQL data volume and rotates it so the file does not fill the volume.
Each MySQL Pod runs two sidecars. The logs container uses Fluent Bit to collect the error logs and the logrotate container rotates those files. During a backup, Fluent Bit also tails the Percona XtraBackup log. Those files live on an emptyDir volume. The Operator does not rotate them, and they do not survive a restart of the MySQL Pod.
Persistent logging is on by default for a cluster with crVersion 1.3.0 or later. It covers MySQL Pods only. HAProxy, MySQL Router, Orchestrator, and Percona Binlog Server still write only to standard output.
Read Persistent logging and Log rotation to learn more.
Official support for DigitalOcean Kubernetes Service (DOKS)
DigitalOcean Kubernetes Service (DOKS) is now a supported platform. Every Operator release is tested on DOKS, so you can run the Operator on DigitalOcean-managed Kubernetes clusters.
To install the Operator on DOKS, read the installation guide.
Support for MySQL 9.7
This release adds support for Percona Server for MySQL 9.7. The Operator supports the major functionality of this version.
CRD changes
These are the Custom Resource changes since Operator 1.2.0.
PerconaServerMySQL
New fields:
spec.mysql.autoConfigsets automatic MySQL configuration tuning withenabled,loadType, andversion. When tuning is on, the Pod needs CPU and memory resources, andversionis required.spec.usersdeclares application users withname,hosts,dbs,grants,passwordSecretRef, andwithGrantOption. See Manage users.spec.logcollectoradds persistent logging and log rotation.logRotate.scheduledefaults to0 0 * * *.imageis required when the section is set.spec.backup.startingDeadlineSecondsandspec.backup.suspendedDeadlineSecondsset how long a backup can wait to start or to resume. The same fields exist onPerconaServerMySQLBackup. See Set a waiting time for a backup to start.spec.backup.storages.<name>.s3.caBundleandspec.backup.pitr.binlogServer.storage.s3.caBundlename the Secret that holds your CA.keydefaults toca.crt. See Configure TLS verification with custom certificates for S3 storage.spec.backup.pitr.binlogServer.storage.encryptionsets binlog encryption.cipherdefaults toAES-256-CTR.spec.backup.pitr.binlogServer.keyringSecretis required when encryption is set.keydefaults tokeyring.json.spec.orchestrator.failoversets asynchronous failover behavior.timeoutdefaults to1h.onTimeoutisAbortorForceWithPossibleDataLossand defaults toAbort. This section is valid only whenmysql.clusterTypeisasync.spec.pmm.haproxyParamspasses HAProxy parameters to the PMM client.spec.unsafeFlags.asyncFailoverWithPossibleDataLosslets an asynchronous cluster run without backups. See Important upgrade notes.status.clusterTypereports the replication type the Operator last observed.
Validation changes:
- An asynchronous cluster can use
mysql.sizeof 2 or greater, including an even count, withoutunsafeFlags.mysqlSize. Size 1 still requires that flag. - Group Replication rejects an even
mysql.sizeunlessunsafeFlags.mysqlSizeistrue. The minimum stays 3, and a size above 9 still requires the flag. - An asynchronous cluster requires
backup.enabled: trueunlessunsafeFlags.asyncFailoverWithPossibleDataLossistrue. - When you switch from asynchronous replication to Group Replication,
spec.orchestrator.enabledmust befalse. spec.tls.issuerConfis an issuer reference withname,kind, andgroup, so you can point at a cert-managerClusterIssuer. See Use an existing ClusterIssuer.spec.backup.storagesaccepts at most 100 entries.containerOptions.args.xtrabackupaccepts at most 100 arguments, and each argument can be at most 1024 characters.--defaults-filemust be the first argument and use--defaults-file=<path>.--defaults-extra-fileis rejected.
PerconaServerMySQLBackup
spec.startingDeadlineSecondsandspec.suspendedDeadlineSecondsoverride the cluster deadlines for one backup.status.sizeandstatus.uncompressedSizereport the backup size after it succeeds. See PerconaServerMySQLBackup status.
PerconaServerMySQLRestore
spec.backupSourceis its own object withdestinationandstorage. It is no longer a copy of the backup status.spec.pitr.keyringSecretsupplies the keyring Secret when you restore encrypted binlogs.status.erroredGenerationrecords the generation of a restore that ended inError.
PerconaServerMySQLClusterSet
status.clusters.<name>.replicationLagSecondsreports how far a replica cluster is behind the primary. See Per-member cluster status.
Changelog
New Features
-
K8SPS-402 - Added deadlines for backups so you can limit how long a backup waits to start or to resume. Backups for one cluster still run one at a time, and the Operator marks a backup
Errorwhen the start deadline passes orFailedwhen a suspended backup does not resume. -
K8SPS-408 - Added application user management in the Custom Resource so users, grants, and passwords are declared with the cluster. The Operator creates each user, applies the grants, and stores or rotates the password in a Secret.
-
K8SPS-500 - Added persistent logging, so the MySQL error log stays on the data volume after a Pod restarts. A
logssidecar streams that log to standard output, andlogrotaterotates the files. -
K8SPS-646 - Added an in-place switch between Group Replication and asynchronous replication on a running cluster (tech preview). Change
mysql.clusterTypeand the Operator rebuilds the topology on the same volumes, with downtime while the Pods restart. -
K8SPS-757 - Added support for a read-only root filesystem on the Percona Monitoring and Management (PMM) client sidecar. The Operator stores the PMM agent configuration on a writable volume, so the sidecar can start when the container root filesystem is read-only.
-
K8SPS-762 - Added security builds so you can patch a High or Critical vulnerability in the Operator image without upgrading to a new Operator version. Numbered tags such as
1.3.0-2stay on a fixed image, and1.3.0-latestpoints at the newest image for your current release.
- K8SPS-780 - Added support for Percona Server for MySQL 9.7 so you can run clusters on this long-term support release.
-
K8SPS-792 - Added a custom certificate authority (CA) for S3-compatible backup storage so TLS verification can stay on with your organization's certificates.
-
K8SPS-823 - Added automatic MySQL configuration tuning with
mysql.autoConfigso settings follow the Pod's CPU, memory, workload, and MySQL version. Tuning is on for a new cluster. Any value you set inmysql.configurationstill overrides the calculated setting. -
K8SPS-849 - Added binlog encryption for point-in-time recovery so binary logs are encrypted before they are uploaded. The Operator validates the key Secret, encrypts each new file, and decrypts it during restore, while files already in the bucket stay unchanged.
-
K8SPS-943 - Added official support for DigitalOcean Kubernetes Service (DOKS) so you can run the Operator on DigitalOcean-managed clusters.
Improvements
-
K8SPS-458 - Added HAProxy parameters for the PMM client so monitoring follows the proxy configured on the cluster. Set
spec.pmm.haproxyParamsto pass those parameters into the client. -
K8SPS-471 - MySQL Pods now stay up when you rotate the server certificate and the certificate authority stays the same. The Operator reloads the new leaf into running
mysqldprocesses, while HAProxy, MySQL Router, and Orchestrator still restart because they load certificates at startup. -
K8SPS-504 - Orchestrator updates now follow SmartUpdate and restart other Pods before the leader. The topology service stays available during the rollout instead of dropping when the leader restarts first.
-
K8SPS-514 - The Operator now extends Orchestrator downtime on the backup source until the backup finishes. Orchestrator no longer ends that maintenance window mid-backup and fails over the Pod that is still being backed up.
-
K8SPS-528 - You can set
mysql.sizeto an even number of two or greater on an asynchronous cluster withoutunsafeFlags.mysqlSize. A single-instance cluster still requires that flag because there is no replica to fail over to. -
K8SPS-563 - Backup status now reports
status.sizeandstatus.uncompressedSizeafter the backup succeeds. You can read the artifact size fromPerconaServerMySQLBackupinstead of checking object storage. -
K8SPS-644 - Leader election is enabled by default, and its timing is configurable with Operator environment variables. Set
PSO_LEADER_ELECTION_ENABLEDto"false"on a single-replica Deployment, or adjust the lease duration, renew deadline, and retry period when election fails on a slow cluster. -
K8SPS-652 - Disabled components no longer need an image and a size in the Custom Resource. A cluster that uses HAProxy no longer has to set MySQL Router fields, and the same applies to any other component you leave turned off.
-
K8SPS-678 - Incremental backups can use XtraBackup page tracking so only changed InnoDB pages are copied. Pass
--page-trackingin thextrabackupcontainer options on the storage or the base backup, and the Operator installs themysqlbackupcomponent that tracking needs. -
K8SPS-704 -
spec.backupSourceon a restore is now its own object withdestinationandstorage. Restore manifests no longer reuse backup status fields that a restore does not apply. -
K8SPS-726 - Point-in-time recovery now works with data-at-rest encryption. The Operator uses your existing Vault Secret during restore, so tablespaces and binary logs stay encrypted while you roll forward to a GTID or a time.
-
K8SPS-730 - The Operator now applies dynamic MySQL variables at runtime, so a change in
mysql.configurationdoes not roll the Pods. A value you already set withSET GLOBALon a Pod stays in place until you change the Custom Resource, and a matching value does not cause a restart. -
K8SPS-746 - HAProxy no longer opens a separate external check for the replica, MySQL X, and admin backends. Those backends follow the primary check, which lowers the number of connections to MySQL.
-
K8SPS-753 - ClusterSet status now includes
replicationLagSecondsfor each replica cluster. You can read that lag withkubectlinstead of connecting to MySQL Shell. -
K8SPS-768 - The Operator automatically rejoins a ClusterSet replica that was paused, stopped, or crashed. When the replica is healthy but the replication channel is down, annotate the ClusterSet and the Operator runs the rejoin Job.
- K8SPS-802 - The Operator now blocks deletion of a database cluster that still
belongs to a ClusterSet. The Operator adds a finalizer so that delete does not leave the ClusterSet broken.
-
K8SPS-848 - The Group Replication liveness probe now treats a member in
RECOVERINGas alive. Kubernetes no longer kills the MySQL container while Group Replication is still catching up. -
K8SPS-896 - A paused Group Replication cluster no longer writes a ClusterSet status skip message on every reconcile. That message is now a debug log, so the Operator log stays readable while the cluster is paused.
-
K8SPS-934 - Restores that end in the
Errorstate are no longer run again when the Operator restarts. The Operator recordsstatus.erroredGenerationand skips that restore, so a restart does not pause the cluster or apply an unexpected backup.
Bug Fixes
-
K8SPS-552 - Fixed asynchronous cluster creation on OpenShift when you set a custom Orchestrator service account. Reconciliation no longer fails with an owner-reference error on the Orchestrator Role, so the cluster can start.
-
K8SPS-649 - Fixed TLS issuer lookup so
tls.issuerConf.kind: ClusterIssueruses your cert-manager ClusterIssuer. The Operator previously looked only for an Issuer in the database namespace, so a cluster-wide issuer was ignored. -
K8SPS-763 - Fixed XtraBackup argument order so
--defaults-fileis the first option on backup and restore. XtraBackup rejects that flag in any other position, and jobs that set it incontainerOptions.args.xtrabackupnow start. -
K8SPS-782 - Fixed the HAProxy
mysql-xbackend so the primary is listed on the MySQL X Protocol port. The primary was registered on port 3306, so X Protocol clients through HAProxy did not reach it. (Thank you Woytek-Polnik for reporting this issue) -
K8SPS-784 - Fixed HAProxy routing so only a current primary that can accept writes is placed in the writable backend. A replica that was made writable by hand is rejected, and a newly elected primary stays unavailable until it finishes applying events.
-
K8SPS-786 - Fixed Operator debug logs that printed Secret contents, including passwords, while updating the binlog server configuration. Secret data is omitted from those logs, and the configuration Secret is no longer rewritten on every reconcile.
-
K8SPS-791 - Fixed restore of a backup taken on a ClusterSet replica onto a new cluster. Bootstrap clears the leftover
clusterset_replicationchannel, so creating the InnoDB Cluster no longer fails on that asynchronous channel. -
K8SPS-830 - Fixed SmartUpdate so it continues when Pods are stuck Pending on an outdated spec. The Operator deletes those Pods and proceeds with the update instead of waiting until you delete them yourself.
-
K8SPS-840 - Fixed cluster status so a new reconcile error replaces the previous one when the cluster is already in an error state.
status.conditionsnow shows the latest failure instead of an older message. -
K8SPS-846 - Fixed
admin_addressselection so MySQL binds to the Pod's routable address on IPv6 and dual-stack clusters. The entrypoint used the first address fromhostname -I, which on Amazon EKS can be a non-routable link-local address. (Thank you kzinas-adv for reporting this issue and contributing the fix) -
K8SPS-851 - The Operator no longer treats a minority Group Replication network split as a full cluster crash. Healthy Pods stay up when only some members lose connectivity. (Thank you somanchi004-code for reporting this issue)
-
K8SPS-859 - Fixed an error loop that appeared when you resized PersistentVolumeClaims after scaling the MySQL cluster up and down. The Operator no longer retries that resize and logs a missing StatefulSet error on every reconcile.
-
K8SPS-863 - Fixed reconciliation that stalled after the MySQL StatefulSet was deleted while custom configuration was set. Applying dynamic variables no longer fails the reconcile when a Pod hostname is missing, so the Operator can recreate the StatefulSet.
-
K8SPS-866 - Fixed the
pt-heartbeatsidecar crash with "Unknown database 'sys_operator'" while an asynchronous replica is cloning. The sidecar now waits until the clone finishes andsys_operatorexists before it starts. -
K8SPS-871 - Fixed point-in-time recovery failures when the binlog list exceeded the 1 MiB ConfigMap limit. The Operator stores a smaller compressed manifest, so restore still works as the binlog history grows.
Documentation updates
- Added the Recover from a full cluster crash page. It explains how the Operator detects a full cluster crash on Group Replication, how recovery runs, and what to do when automatic recovery is off or does not succeed.
- Added troubleshooting pages for marker files in the MySQL data directory. Special files in the data directory describes each file. Use them to pause a MySQL Pod, pause a point-in-time recovery, or unblock a stuck rolling update.
Supported software
The Operator was developed and tested with the following software:
- Percona Server for MySQL 9.7.2-2.1
- Percona Server for MySQL 8.4.11-11.1
- Percona Server for MySQL 8.0.46-37.1
- XtraBackup 9.7.1-rc1.1
- XtraBackup 8.4.0-7.1
- XtraBackup 8.0.35-36.1
- MySQL Router 9.7.2
- MySQL Router 8.4.11
- MySQL Router 8.0.46
- HAProxy 2.8.29-1
- Orchestrator 3.2.6-24
- Percona Toolkit 3.7.2-1
- Log collector (Fluent Bit) 5.1.4-1
- PMM Client 3.10.0
- Cert Manager 1.21.2
- Percona Binlog Server 0.4.1
Other options may also work but have not been tested.
Supported platforms
Percona Operators are designed for compatibility with all CNCF-certified Kubernetes distributions. Our release process includes targeted testing and validation on major cloud provider platforms and OpenShift, as detailed below:
- Google Kubernetes Engine (GKE) 1.34 - 1.36
- Amazon Elastic Kubernetes Service (EKS) 1.34 - 1.37
- DigitalOcean Kubernetes Service (DOKS) 1.37
- OpenShift 4.19 - 4.22
- Minikube 1.39.0 with Kubernetes v1.37.0
This list only includes the platforms on which the Percona Operators are specifically tested as part of the release process. Compatibility with other Kubernetes flavors and versions depends on the backward compatibility provided by Kubernetes itself.