Skip to content

v3.1.0

Latest
Latest

Choose a tag to compare

View all tags
@nastena1606 nastena1606 released this 09 Sep 14:16
· 3 commits to main since this release
v3.1.0
a6cb60b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Important! Operator 2.8.0 and all 2.8.x patch releases have reached end of life. They no longer receive bug fixes, security updates, or support. Upgrade to a supported Operator version to keep your clusters current and protected.

Release highlights

Transparent data encryption support with pg_tde

You can now protect PostgreSQL data at rest with pg_tde, Percona's fully open source transparent data encryption extension.

Data in tables, indexes, temporary tables, and write-ahead log (WAL) files is encrypted on disk. Without the keys, that data stays unreadable even if storage is compromised.

Enable pg_tde in the Custom Resource and configure HashiCorp Vault as the key provider. The Operator installs and configures the extension for you. After setup, you can create encrypted tables or convert existing ones. Backups and restores work as usual when the Operator can reach the encryption key.

This feature is available with PostgreSQL 17 and 18. In this
release, only HashiCorp Vault (KV v2) is supported as a key
provider. KMIP and other providers are planned for later.

See Data-at-rest encryption and Configure data-at-rest encryption with HashiCorp Vault.

Improve operational resilience and observability with persistent logging for PostgreSQL Pods

Debugging distributed systems just got easier. Percona Operator for PostgreSQL now supports persistent logging, so PostgreSQL and pgBackRest logs on the instance data volume stay available even across Pod restarts.

The Operator uses Fluent Bit to collect logs. Fluent Bit runs a logs sidecar container on each PostgreSQL instance Pod and mounts the same data volume. It collects logs and streams them to its own stdout as JSON lines. You can additionally configure Fluent Bit outputs such as S3 or Open Telemetry (OTel) envelope and have the logs forwarded there.

Learn more about persistent logging in the documentation.

Configure log rotation for persistent logs

You can now customize log rotation for persistent logs. This helps you keep the right amount of data for troubleshooting or compliance, extend the default configuration with your custom settings and schedule rotations to fit your operational windows.

You can configure log rotation in these ways:

  • Override the default configuration via the Custom Resource
  • Define additional configuration via a ConfigMap. In this case, the Operator adds your options to the default configuration
  • Set a new rotation schedule

See our documentation for step-by-step instructions for each option.

Define logical replicas declaratively (tech preview)

You can now add a read-only logical replica in the same cluster and point reporting or other heavy reads at this replica instead of the primary. The replica has its own volume and Service, so those queries do not compete with your high-availability set.

Declare the replica in the Custom Resource when you create the cluster or later during runtime. You can also define which databases receive the changes after the replica is bootstrapped.

The Operator creates the volume, copies the data, converts the physical replica to the logical one and keeps the databases you list in sync. It also creates a Service you can connect to.

Patroni does not manage nor promote it, so it stays a stable read endpoint.

Logical replication is in the tech preview stage and requires PostgreSQL 17 or later. See Deploy a logical replica.

Mount extra volumes into PostgreSQL instances

You can now mount additional Kubernetes volumes into the PostgreSQL container so that every instance in the cluster sees the same files.

This is useful when PostgreSQL needs files outside the data
directory such as for full text search.

Configure the volume and mount path under instances.extraVolumes in the Custom Resource. You can use a ConfigMap, Secret, PersistentVolumeClaim, emptyDir, or
another volume source that Kubernetes supports.

Every instance in the set sees the same mounts, so your assets stay consistent across the cluster. Prefer subPath mounts when you add files next to the existing directory, so you do not replace the contents that PostgreSQL already ships.

For steps and examples, see Mount extra volumes into PostgreSQL instances.

More control over TLS certificate management

If you manage TLS certificates manually, such as through Kubernetes Secrets synced from AWS Secrets Manager or via External Secrets, losing access to those Secrets even briefly can lead to a service outage.

By default, the Operator treats a missing TLS Secret as a signal to create new certificates, restart the database Pods, and apply the new CA. Applications that still trust your original CA may lose connectivity.

Starting with this release, set spec.tls.certManagementPolicy in the Custom Resource to control the Operator's behavior. The policies are:

  • auto (default) — If TLS Secrets are missing, the Operator creates certificates. It may use cert-manager when cert-manager is installed.
  • userProvidedOnly — You own the certificate lifecycle. The Operator does not create or replace TLS certificates if a Secret is temporarily unavailable. Applications can keep using the certificates already loaded while you restore the Secret.
  • operatorProvidedOnly — The Operator always manages TLS with its own PKI and does not use cert-manager.

For manually provided TLS, use userProvidedOnly:

spec:
  tls:
    mode: preferTLS
    certManagementPolicy: userProvidedOnly
    allowInvalidCertificates: false
  secrets:
    customTLSSecret:
      name: cluster1-cert
    customReplicationTLSSecret:
      name: replication1-cert

See TLS certificate management policy for more information.

Enable Mutual TLS (mTLS) between clients and pgBouncer

Mutual TLS (mTLS) needs pgBouncer to trust the CA that signed client certificates. Previously, you could only do that by providing TLS material in proxy.pgBouncer.customTLSSecret and you had to manage the full certificate lifecycle yourself.

Now, you can extend the pgBouncer frontend trust bundle with your external CA that signed client certificates. Create the Secret with this CA and add it to the cluster with theproxy.pgBouncer.additionalTrustedCAs option. The Operator appends your external CAs to the trust bundle and keeps rotating the pgBouncer and cluster certificates.

With this improvement, you get mTLS for clients that use your corporate or application CA, and keep Operator-managed TLS for the cluster.

See Trust additional CAs for pgBouncer client mTLS for configuration steps.

Keep backup storage healthy with auto-growable disks for pgBackRest

A full pgBackRest repository volume can interrupt backups. Starting with this release, auto-growable disks cover pgBackRest repository volumes on the repo host, not only PostgreSQL data volumes. The Operator monitors disk usage and expands the PVC automatically up to the limit you set.

Turn on the AutoGrowVolumes=true feature gate and set spec.backups.pgbackrest.repos[].volume.volumeClaimSpec.resources.limits.storage in the Custom Resource. If you already use automated storage scaling for data volumes, apply the same approach to your backup repos.

spec:
  backups:
    pgbackrest:
      repos:
      - name: repo1
        volume:
          volumeClaimSpec:
            resources:
              requests:
                storage: 1Gi
              limits:
                storage: 5Gi

See Automated scaling with auto-growable disks for the full setup.

Pause and resume pgBouncer connections

You can now pause client traffic through pgBouncer without dropping application connections. Active queries finish, then new client requests wait in a queue until you resume. Use this for a short, controlled window to restart PostgreSQL, run a planned switchover, or drain backend load.

Set proxy.pgBouncer.paused to true in the Custom Resource to pause, and back to false to resume. The Operator connects to each pgBouncer Pod
and runs the PAUSE command. Pods with active connections
remain running until they finish, then the Pod changes its state
to paused. If a pgBouncer Pod restarts while paused, a startup
probe re-applies PAUSE
so the intended state survives restarts.

Support for UBI 8, UBI 9, and UBI 10 images for Percona Distribution for PostgreSQL

You can now choose Red Hat Universal Base Image (UBI) 8, 9, or 10 for Percona Distribution clusters. That lets you migrate to Percona Operator without changing the OS inside the PostgreSQL container, so you can keep working extensions, stay aligned with your Enterprise Linux version, or move to a newer UBI when you are ready.

UBI is the operating system inside the PostgreSQL container. PostgreSQL uses OS-provided glibc and ICU libraries to sort and compare text, and each UBI major version ships different library versions. The UBI choice is part of the database runtime.

By default, images that ship with the Operator are based on UBI 9 and have no OS version in the tag, for example docker.io/percona/percona-distribution-postgresql:18.6.1-1. UBI 8 and UBI 10 images add the OS version to the tag:

docker.io/percona/percona-distribution-postgresql:18.6.1-1-ubi8
docker.io/percona/percona-distribution-postgresql:18.6.1-1-ubi8-arm64
docker.io/percona/percona-distribution-postgresql:18.6.1-1-ubi10
docker.io/percona/percona-distribution-postgresql:18.6.1-1-ubi10-arm64

Use UBI 9 as the baseline for PostgreSQL 14–18. This page lists the UBI 9 images. For UBI 8 and UBI 10, see Percona certified images.
Stay on UBI 8 when older extensions or Enterprise Linux 8 requirements still apply. Choose UBI 10 to align with RHEL 10 or OpenShift nodes that run Enterprise Linux 10, and to stay on an OS with a longer remaining support window and newer system libraries.

Important: Switching UBI major versions changes glibc and ICU. PostgreSQL reports a collation version mismatch, and indexes on text, varchar, char, and similar types can return incorrect results until you rebuild them. After you switch, identify affected indexes, run REINDEX, then run ALTER DATABASE ... REFRESH COLLATION VERSION. See the minor upgrade and major upgrade documentation.

Keep every instance in a cluster on the same UBI version. Treat a UBI change as an OS upgrade: take a backup first, and confirm that custom extensions are built for the target UBI.

Support for community PostgreSQL images

You can now run the Operator with PostgreSQL Community images built from the official PostgreSQL Global Development Group (PGDG) packages. You can also use the images you build and store in your own registry. In this way you are no longer limited to Percona Distribution images for the database runtime, so you can add community extensions and keep the image supply chain under your control.

We publish evaluation images for PostgreSQL, pgBouncer, and pgBackRest built from those PGDG packages on UBI 8 and UBI 9. You can use them to try the flow, then reproduce the same build with the Dockerfiles in percona-docker/postgresql-containers/community and push the result to a registry you control.

Point the cluster at those images with spec.image, spec.proxy.pgBouncer.image, and spec.backups.pgbackrest.image. The Operator deploys and manages the cluster the same way it does for Percona Distribution for PostgreSQL. You cannot use Percona-only features such as Transparent data encryption. You own the image lifecycle and support. You need Operator 3.1.0 or later. See PostgreSQL Community images for the published tags.

Community images are an experimental project. Try them and tell us what works, what is missing, and what you want the Operator to support. For build and deploy steps, see Deploy a cluster with community PostgreSQL images and the Percona Blog post Community Docker Images: keeping the operator open without a vendor registry lock-in by Slava Sarzhan.

Support of PostgreSQL 19 (tech preview)

With this release, the Operator supports deployment of Community PostgreSQL 19. This support is currently in the tech preview stage because this major version is not officially released yet. However, you can already deploy it and evaluate the features coming with this version. This allows you to stay on top of upcoming enhancements and gives you enough time to prepare your upgrade and migration plans before the final release lands. See Deploy the Operator with Community images for guidelines.

Official support for Rancher Kubernetes Engine (RKE2)

Rancher Kubernetes Engine (RKE2) is now an officially supported platform. Every Operator release is now tested on RKE2 to ensure that you can run it on Rancher-managed Kubernetes clusters with confidence. Check the current known limitation for using custom extensions on Rancher.

The Operator is now fully supported on ARM64 architectures

All Operator images are now available for ARM64, giving you native support on ARM based clusters with no extra setup.

Deprecation, Change, Rename and Removal

  • Removed support for PMM2. This Operator release no longer supports PMM2 as it has reached the end-of-life state. Upgrade to PMM3 as soon as possible. For how to upgrade, refer to PMM documentation.

  • The Operator version 2.8.0 and all 2.8.x patch versions have reached end of life and are no longer supported.

  • The extensions.builtin section is deprecated and will be removed after version 3.4.0. We encourage you to use extensions.<extension>.enabled. You can still use the old form during the transition. If both forms are set at the same time, extensions.builtin takes precedence.

  • pg_cron and set_user extensions have been added to the list of built-in extensions. Your existing setup via extensions.custom remains unchanged and works as expected after the upgrade.

    Upgrade the database images first and wait until Pods rejoin the cluster. Then reconfigure the extensions. Completing these steps one after the other lets the cluster finish each change in one pass.

    After the database upgrade, switch to built-in extensions as follows:

    • Remove the extension from the extensions.custom list
    • Set extensions.pg_cron.enabled or extensions.set_user.enabled to true.

    You must do these two Custom Resource changes in the same apply so they land in one reconciliation loop. Removing the extension from extensions.custom alone instructs the Operator to delete it.

  • Field descriptions were removed from the inherited CrunchyBridgeCluster CRD (upstream.pgv2.percona.com/v1beta1). The object schema and cluster behavior are unchanged. Running kubectl explain for those fields no longer shows help text.

Known limitations

Custom extensions on Rancher (RKE2)

When you enable custom extensions, the Operator restarts the cluster's Pods. On Rancher (RKE2) clusters, a former primary may fail to rejoin the cluster as the replica if it has a diverged timeline.

The affected Pod's PostgreSQL log shows the following:

LOG:  invalid xl_info in checkpoint record
PANIC:  could not locate a valid checkpoint record at <LSN>

To recover, you must manually reinitialize the replica. Before doing so, check if this replica has any transactions that are not replicated anywhere else. Then remove its data directory and let the instance perform a full copy from the primary.

Changelog

New Features

  • K8SPG-911 - Added native transparent data encryption support via pg_tde so you can encrypt data at rest from the Custom Resource.

  • K8SPG-650 - Added support for declaring additional pgBouncer users in the Custom Resource so monitoring and other integrations can keep dedicated credentials. The Operator now manages the pgBouncer user list for you instead of relying on manual Secret edits that could be overwritten.

  • K8SPG-851 - Added persistent logging for PostgreSQL and pgBackRest so logs remain available across Pod restarts. Fluent Bit collects logs on the instance data volume and can forward them as JSON or to configured outputs such as S3 or OpenTelemetry.

  • K8SPG-1114 - Added declarative logical replicas for read-only workloads so you can offload reporting queries from the primary. The Operator creates the volume, bootstraps replication, and exposes a dedicated Service that Patroni does not promote.

  • K8SPG-1056 - Added support for Community PostgreSQL images and custom registries so you are not locked to Percona-only tags. Set spec.image, proxy.pgBouncer.image, and backups.pgbackrest.image to deploy and manage community or privately built images.

  • K8SPG-952 - Added the ability to provide additional trusted CA to pgBouncer. This way you can enable client mTLS while cluster components communicate using Operator-managed PKI and the Operator continues to manage cluster TLS rotation.

  • K8SPG-949 - Added official support for Rancher Kubernetes Engine (RKE2). Every Operator release is now tested on RKE2 so you can run on Rancher-managed clusters with confidence.

Improvements

  • K8SPG-440 - Added the ability to mount extra volumes into PostgreSQL instances so every Pod can share files such as full-text search dictionaries.

  • K8SPG-691 - Extended auto-growable disks to pgBackRest repository volumes so a full backup repo no longer blocks backups. The Operator monitors repo-host disk usage and expands the PVC up to the limit you set when the AutoGrowVolumes feature gate is enabled.

  • K8SPG-870 - Added a status.size field on full, differential, and incremental PerconaPGBackup objects so you can see backup size without inspecting storage directly. This makes backup capacity planning and troubleshooting easier.

  • K8SPG-881 - Added full ARM64 support for Operator images so you can run natively on ARM-based Kubernetes clusters. No extra setup is required beyond using the ARM64 image digests published with this release.

  • K8SPG-944 - Removed PMM2 support now that PMM2 has reached end of life. Upgrade monitoring to PMM3 so cluster health checks continue to work with this Operator version.

  • K8SPG-951 - Added the support of custom CA issuer so you can plug in your own cert-manager Issuer or ClusterIssuer instead of the hardcoded self-signed CA. This lets you issue cluster TLS certificates from Vault or an existing corporate CA.

  • K8SPG-1011 - Removed the requirement to set a pgBackRest image when backups are disabled. Clusters with backups.enabled: false no longer need unused backup image configuration.

  • K8SPG-1040 - Added pg_cron and set_user to the built-in extensions list so you can enable them with extensions.<name>.enabled. Existing custom-extension installs continue to work; migrate by updating the Custom Resource in a single apply.

  • K8SPG-1045 - Added the ability to define TLS management policy for the Operator. This gives you a control whether the Operator recreates TLS Secrets when they are temporarily missing. Use userProvidedOnly to keep manually provisioned certificates stable during Secret sync outages.

  • K8SPG-1047 - Updated major-upgrade images to use the official Percona Distribution for PostgreSQL upgrade Dockerfile. This keeps upgrade containers aligned with the same build pipeline as the database images.

  • K8SPG-1051 - Added tech-preview support for Community PostgreSQL 19 so you can evaluate the next major version early. Use this to test application compatibility before PostgreSQL 19 is generally available.

  • K8SPG-1053 - Relocated the PMM agent config to a writable /tmp volume so the PMM sidecar starts when readOnlyRootFilesystem: true is enforced. Hardened environments such as OpenShift restricted-v2 can now monitor clusters without relaxing the root filesystem policy.

  • K8SPG-1073 - Stopped configuring Patroni's restore_command when backups are disabled so replicas no longer call pgBackRest against a missing stanza. WAL fetch failures in backup-less clusters fall back cleanly instead of producing archive errors.

  • K8SPG-1083 - Improved Operator telemetry to report the Kubernetes platform in use, such as EKS, GKE, AKS, or Rancher. This helps Percona understand deployment environments and improve platform-specific testing.

  • K8SPG-1085 - Improved SmartUpdate logging so rolling instance updates are visible in the Operator logs. You can track when Pods are being restarted during automated updates without extra debugging.

  • K8SPG-1115 - Added the ability to pause and resume pgBouncer connections from the Custom Resource without dropping clients. Set proxy.pgBouncer.paused to queue new requests while active queries finish during planned maintenance.

Fixed bugs

  • K8SPG-991 - Fixed the issue where outdated backup cleanup failed during minor upgrades and logged noisy errors. The Operator now handles repo-host readiness more reliably before attempting cleanup.

  • K8SPG-994 - Fixed a bug where customRootCATLSSecret kept only the first certificate from a multi-cert PEM bundle. Intermediate and root CA chains are now preserved so TLS verification works for corporate PKI setups.

  • K8SPG-1010 - Fixed an ordering issue where delete finalizers removed Secrets before the underlying PostgresCluster was gone, allowing Secrets to be recreated and left behind. Secret cleanup now runs after the cluster is fully deleted.

  • K8SPG-1012 - Fixed a bootstrap restore issue where the timeline history file was lost and after point-in-time recovery replicas could fail to rejoin via pg_rewind. History files are now archived correctly during restore bootstrap.

  • K8SPG-1049 - Fixed restore Jobs that ignored cluster tolerations and could not schedule on tainted nodes. Restore Pods now inherit the same tolerations as the backup jobs. (Thank you @Avapaa for reporting this issue)

  • K8SPG-1058 - Fixed PerconaPGUpgrade failures when the upgrade resource name was too long for generated Kubernetes object names. Long upgrade names no longer block major version upgrades. (Thank you Jakub Jaruszewski for contributing to this fix)

  • K8SPG-1113 - Fixed a bug where latestRestorableTime on older backups was overwritten with the newest restorable timestamp. Each backup now keeps its own restorable time for accurate point-in-time recovery choices.

  • K8SPG-1119 - Fixed restores from dataSource.volumes that failed validation because repoName was required. Volume-based bootstraps no longer need a pgBackRest repository name.

  • K8SPG-1138 - Fixed the issue snapshot restore-prepare job not being scheduled on tainted nodes because of missing tolerations. You can now assign tolerations to snapshot restore jobs.

Documentation updates

  • K8SPG-232 - Expanded restore documentation to explain in-place restore, side-cluster restore, and restore into an external environment. You can choose the right recovery pattern for your topology more easily.

  • K8SPG-793 - Documented the trackLatestRestorableTime logic, including when to use them and how they interact with read-only filesystem policies. Also reorganized point-in-time recovery guides.

  • K8SPG-963 - Documented Custom Resource statuses and conditions so you can interpret cluster health from status without reading Operator source code.

  • K8SPG-1123 - Added documentation for deploying the Operator with Community PostgreSQL images. The guide covers how to set custom image fields and what Percona-only features are unavailable with community builds.

Supported software

This Operator version is developed and tested with Percona Distribution for PostgreSQL and PostgreSQL Community.

Percona Distribution for PostgreSQL

  • PostgreSQL 14.24-1, 15.19-1, 16.15-1, 17.11.1-1, 18.6.1-1 as the database. Other versions may also work but have not been tested.
  • pgBackRest 2.59.0-1 for backup and recovery
  • pgBouncer 1.25.2-6 for connection pooling
  • Patroni version 4.1.5 for high-availability
  • PostGIS version 3.5.7
  • PMM Client version 3.9.1
  • cert-manager 1.21.1

PostgreSQL Community

  • PostgreSQL 14.24, 15.19, 16.15, 17.11, 18.6, and 19 (tech preview) as the database.
  • pgBackRest 2.59.0 for backup and recovery
  • pgBouncer 1.25.2 for connection pooling

Supported platforms

Percona Operators are designed for compatibility with all CNCF-certified Kubernetes distributions.

Our release process includes targeted testing and validation on major cloud provider platforms and OpenShift, as detailed below:

This list only includes the platforms that the Percona Operators are specifically tested on as part of the release process. Other Kubernetes flavors and versions depend on the backward compatibility offered by Kubernetes itself.

PostgreSQL Community images

Community images are available on UBI 9 and UBI 8. PostgreSQL 19 is a tech preview and ships on UBI 9 only. pgBouncer and pgBackRest community images are compatible with both UBI 8 and 9. See PostgreSQL community images.

Contributors

Avapaa
Assets 2
Loading