v4.5.0 Silo, Valkey, Kafka

[Vonng]

Pigsty v4.5.0 is a feature release spanning data services, the PostgreSQL ecosystem, safer automation, observability, and the software supply chain. It introduces pilot modules for Kafka KRaft and MySQL 8.4, adds Valkey support to REDIS, converges the MINIO module on Silo, and expands the packaged PostgreSQL extension catalog from 531 to 575.

This release covers 100 commits and 416 files since v4.4.0. See the complete source comparison.

Highlights

• 575 PostgreSQL extensions: 46 additions and 2 removals, for a net gain of 44. Package maps, RPM/DEB coverage, aliases, and a broad set of Rust extensions rebuilt with pgrx 0.19.1 are included.
• Kafka KRaft pilot module: Native multi-cluster orchestration, dynamic controller/broker enrollment and retirement, SCRAM-SHA-512, TLS, credential rotation, lifecycle safety checks, exporters, alerts, and four Grafana dashboards.
• MySQL 8.4 pilot module: Standalone and three-node InnoDB Cluster deployments with MySQL Router, Shell, TLS, XtraBackup, user/database provisioning, conservative reconciliation, alerts, and five Grafana dashboards.
• Valkey and Silo: REDIS now supports redis_type: valkey. The MINIO module now accepts only minio_type: silo; the RustFS backend developed during this cycle was withdrawn before release.
• Safer orchestration: PGSQL, REDIS, MINIO, KAFKA, and MYSQL workflows are cluster-identity-aware. etcd delegation, DBSU key exchange, PITR cleanup, backup markers, and destructive removal paths are more tightly bounded.
• Observability refresh: Grafana dashboards move to Dashboard API v2, MinIO/Silo monitoring moves to Metrics V3, and pg_exporter gains PostgreSQL 19, transaction-age, lock, replication, and vacuum-pressure metrics.
• Repository and supply-chain work: SOW now generates local RPM/APT repositories atomically, synthetic ModuleMD is removed, package inputs are checksum-pinned, and Pigsty-built Infra packages use SPDX metadata and the 1PGSTY release convention.
• 51 standalone configuration templates: New Kafka, MySQL, and eight-node HA simulation examples join the existing configuration library.

New data services

Kafka KRaft

The new Kafka module uses node state as the source of truth and supports multiple clusters in one inventory. It manages dynamic quorum joins, broker admission, member retirement, and three-step dead-node replacement. Partial initialization limits are rejected, while kafka-rm.yml requires an explicit non-empty limit and validates data paths and surviving quorum anchors before stopping services.

Kafka remains a pilot module. Clients must resolve and reach every broker directly; the Kafka data plane must not be placed behind HAProxy, a VIP, or a generic L4 load balancer.

MySQL 8.4

The new MySQL pilot module supports either one node or a three-node InnoDB Cluster. It includes MySQL Router and Shell, scheduled XtraBackup full backups, TLS, account and database provisioning, primary-key policy checks, and guarded member removal.

Valkey, Silo, and Mongo mode

• REDIS keeps Redis as the default engine and deploys Valkey only when redis_type: valkey is selected. Existing Redis-style configuration paths, service interfaces, and monitoring labels remain compatible.
• The MINIO role now deploys Silo as its sole backend. Silo preserves the S3/Admin APIs, /minio/* routes, MINIO_* environment variables, and compatible disk format, while using the silo package, binary, and service.
• The standalone FERRET role and mongo.yml are removed. Use PostgreSQL Mongo mode for the DocumentDB data layer and the FerretDB Docker app for the MongoDB protocol layer.

PostgreSQL and extensions

• PostgreSQL 18.6 is the default. Standard Patroni templates allow pgoutput, test_decoding, and wal2json through output_plugin_libraries; Patroni filters the parameter on older PostgreSQL versions.
• The PostgreSQL 19 beta template gains pgBackRest 2.59 and scheduled-backup support.
• Percona PostgreSQL 18 TDE uses cluster mode. IvorySQL receives corrected default-database initialization and compatible WAL compression.
• Notable extension upgrades include Citus 14.2.0, pg_search 0.25.2, TimescaleDB 2.29.1, pgvector 0.8.6, DocumentDB 0.114, pg_partman 5.5.0, pgmnemo 0.16.1, plpgsql_check 2.10.4, provsql 1.12.0, and pgbson 2.1.0.
• Notable additions include pg_lake, pg_local_cache, pg_turbovec, pg_vault_tde, pg_oidc_validator, pg_statviz, pgcontext, pgmemento, pgwasm, plruby, postbis, and qdgc.
• pg_analytics and spat are removed from the catalog. emailaddr, explain_ui, oidc_validator, pg_summarize, and smlar are removed from default install groups because upstream does not provide a redistributable license.
• pg_relation_sql is packaged as a standalone SQL toolkit and does not count toward the 575 extensions because it does not support CREATE EXTENSION.

The full package-by-package RPM/DEB matrix is available in the English release notes and Chinese release notes.

Observability, repositories, and platforms

• Adds four Kafka and five MySQL dashboards and refreshes the Node, PostgreSQL, Redis, and Infra dashboard sets.
• Moves Silo collection to /minio/metrics/v3 and drops high-cardinality bucket samples.
• Updates pg_exporter to the 1.4 series and fixes duplicate pg_subrel time series.
• Uses sow create --pigsty to generate local repositories and SHA-256 repo_complete markers atomically.
• Renames exporter RPMs from underscore to hyphen form, such as node_exporter → node-exporter and redis_exporter → redis-exporter.
• Notable Infra updates include etcd 3.7.1, Grafana 13.1.3, VictoriaMetrics 1.149.0, VictoriaLogs 1.52.0, pg-exporter 1.4.1, pg-timetable 7.0.0, PostgREST 16.1, Vector 0.57.0, and SOW 0.3.0.
• Refreshes the validated platform baseline to Rocky Linux 9.8/10.2, Debian 12.15/13.6, and Ubuntu 22.04.5/24.04.4/26.04.0 on both x86_64 and ARM64.
• Vagrant gains a configurable root_disk (64 GiB by default; 128 GiB in package-build matrices) while disk continues to describe the optional /data disk.

Important upgrade notes

1. Silo replaces the MinIO server: minio_type accepts only silo. Protocol and disk-format compatibility do not constitute a tested migration; validate backups, real reads/writes, rollback, and in-place compatibility before switching an existing object-storage deployment.
2. ha/trio changes topology: New deployments use a three-node, single-drive Silo pool with EC:1. An existing single-node pool cannot be expanded into this topology in place; create a new pool and migrate objects.
3. FERRET is split: Remove the old FerretDB systemd service and redeploy the data and protocol layers through Mongo mode plus the Docker app.
4. Explicit cluster identity is required: Custom inventories must define the relevant pg_cluster, redis_cluster, minio_cluster, kafka_cluster, or mysql_cluster identity.
5. Full pgsql.yml is for initial bootstrap: It restarts Patroni/PostgreSQL and reapplies managed configuration and bootstrap SQL. Use precise tags for routine maintenance.
6. HAProxy service semantics changed: Pigsty no longer renders /etc/default/haproxy, although the unit reads it when present. Use EXTRAOPTS, preserve -S /run/haproxy-master.sock, and do not add -f.
7. SOW is now a core REPO/CACHE dependency: Older offline repositories may need SOW 0.3.0 before regeneration.
8. Exporter RPM names changed: Update private repositories and automation to use the hyphenated package names.
9. Docker purge is immediate: make purge deletes the repository-local ./data directory without a countdown.
10. Vagrant disk variables are distinct: root_disk controls the system disk; disk controls the extra /data disk required by distributed Silo.

Install

For a fresh installation on a supported Linux node:

curl -fsSL https://repo.pigsty.io/get | bash -s v4.5.0
cd ~/pigsty
./bootstrap
./configure
./install.yml

A production upgrade is not an in-place overwrite. Review the compatibility notes, verify backups and live cluster state, and scope every maintenance operation explicitly.

Release assets and signatures

This release provides the source archive plus offline packages for Rocky Linux 9/10, Debian 12/13, and Ubuntu 22.04/24.04/26.04 on both x86_64 and ARM64.

All 16 original assets have matching GPG ASCII detached signatures, for 32 uploaded assets in total. The signing key is published in KEYS:

9592 A7BC 7A68 2E73 3337  6E09 E793 5D8D B9BD 8B20

Verify an asset with:

curl -LO https://raw.githubusercontent.com/pgsty/pigsty/v4.5.0/KEYS
gpg --import KEYS
gpg --verify pigsty-v4.5.0.tgz.asc pigsty-v4.5.0.tgz
grep 'pigsty-v4.5.0.tgz$' checksums | md5sum -c -

The uploaded files were reconciled against their local names, byte sizes, and GitHub SHA-256 asset digests before publication.

Contributors

Thanks to @Mosney for the ha/trio MinIO user fix in #773, and to Dependabot for the Docker and GitHub Actions updates in #758 and #757.

Full changelog: v4.4.0...v4.5.0

---

This discussion was created from the release v4.5.0 Silo, Valkey, Kafka.