Repository navigation
Karakoram v0.4.2
Karakoram v0.4.2 is a fix release for v0.4.1, based on further live checks, principally on Zerops and Dokploy. It corrects resume and exposure recovery, engine formation, backups and restores, clone readiness and deployment retries. There are no new engines or features. Follow the v0.4.1 → v0.4.2 upgrade guide.
Fixed
Unless stated otherwise, these defects affected v0.4.0 and v0.4.1. “Control-plane-side” means a control-plane upgrade applies the fix; “agent-side” means existing member executables must be replaced through the supported procedures in Upgrade notes. Provider-specific live evidence is scoped below, not a promise of complete acceptance on every provider.
Maintenance, resume and exposure
- High — v0.4.1 regression: resume/recovery under a closed maintenance window could fail to restore already-admitted public TCP or HTTPS exposure, and a resume operation could remain running. Previously admitted exposure now recovers without admitting deferred exposure changes or acknowledging them prematurely. This is control-plane-side, covering PostgreSQL/Supabase (including its gateway), MongoDB, RabbitMQ, Elasticsearch, OpenBao, MySQL/PXC, Neo4j, Weaviate and Keycloak. Live-verified on Dokploy with a held OpenBao resume completing without an override or second resume, RabbitMQ suspend/resume, PostgreSQL public-route recovery and a deferred private-to-public change remaining private until overridden; on Zerops, PostgreSQL public IPv6 exposure recovered after one resume with TLS-verified data readback. Other affected engine paths have regression-test coverage, not a separate live exposure-recovery pass. v0.4.0 is not affected.
OpenBao
- v0.4.1 regression — briefly reporting Ready during a rolling update: Updating/Upgrading now remains visible between member replacements until the admitted generation converges; a deferred update still leaves a healthy cluster Ready. Control-plane-side, live-verified on Dokploy, with no intermediate Ready transition during the roll. v0.4.0 is not affected by this intermediate Ready defect.
MySQL / PXC formation and switchover
- High — MySQL Group Replication now forms: generated group-recovery credentials fit MySQL's length limit; legacy single-member credentials are corrected before first adoption into a group. The credential-generation/adoption fix is control-plane-side. Never-formed legacy groups must be deleted and freshly created; do not force bootstrap or edit coordinator state. Formation and proxied SQL were live-verified on Zerops; a new Dokploy/Nomad Group Replication pass is not established for this release.
- High — PXC now forms on Zerops: startup durably writes required files and their parent directories without a filesystem-wide synchronization call rejected by the platform. This includes agent-side launcher/durability changes and control-plane reload/observation handling. Live-verified on Zerops with three synchronized members, ProxySQL and TLS-verified writes; fresh PXC formation also passed on Dokploy.
- High — MySQL Group Replication primary switchover and clustered restart: the native primary-election function now receives a validated literal UUID rather than an unsupported prepared parameter. Agent-side, live-verified on Zerops: clustered restart completed with three ONLINE members and 367 acknowledged writes read back without loss. Dokploy/Nomad re-proof is not established.
- Group startup failures retain the newest useful diagnostic and omit credential-bearing records; clone failures report a safe stage and, when available, the MySQL error number instead of an undifferentiated failure. These are agent-side diagnostics, not permission to expose raw SQL or error messages. Sanitized startup diagnostics were exercised on Zerops; clone-stage diagnostics identified the follower recovery failure there. Redaction and diagnostic-boundary cases are covered by tests.
MySQL / PXC backups, restore and PITR
- High — full and incremental recovery chains: the control plane now recognizes XtraBackup's native full-backup checkpoint type, unblocking full backups and dependent incremental chains, restore/clone baselines and PITR baselines. Control-plane-side; take and verify a new full backup rather than relying on a previously failed one. Full plus incremental backups passed on Zerops for MySQL Group Replication and PXC, and on Dokploy for PXC.
- High — in-place restore completes re-formation: the control plane releases durable server-stop holds so restored seeds and followers can start. MySQL Group Replication followers fence clients while native re-clone replaces old data, and retry a rejected native start only when copying never began. Stop-hold release is control-plane-side; Group Replication follower re-clone is agent-side. Live-verified on Zerops for Group Replication restore from an incremental chain and PXC full restore, with per-member data checks and writes after recovery. PXC uses its own Galera recovery path. PXC PITR restore also passed natively on Dokploy. See the still-open Zerops PXC post-restore issue below.
- High — PITR binlog archiving: configuration refresh no longer continually postpones the archive timer; status uses completed archives and real lag rather than a missing archive index or refresh timing. Agent-side archiving plus control-plane status fixes, live-verified on Zerops for MySQL Group Replication and PXC: binlog/archive-index pairs appeared every 300 seconds and PITR recovered data to a point after the last full backup. PXC archive cadence and PITR data also passed on Dokploy. Nomad re-proof is not established.
Clones and create-from-backup
- OpenBao, MySQL/PXC, Neo4j and Weaviate now publish completed initial-restore receipts to the readiness gate, allowing a successful clone to report Ready and finish creation. Control-plane-side. Live-verified on Dokploy for OpenBao (the held operation completed without replacing containers/volumes), and on Zerops for MySQL restored from a full-plus-incremental chain and Neo4j with TLS-verified source-marker readback. PXC and Weaviate completion-gate fixes have tests; completed live clone acceptance for those engines is not established in the release evidence.
- Temporal now rejects unsupported initial data sources at admission instead of accepting a create-from-backup request it cannot complete. Control-plane-side, verified by tests; no live admission re-proof is claimed. Its separate restore functionality is unchanged.
Dokploy and Keycloak
- Dokploy initial-build retry, including Neo4j: failed first builds remain observable so ownership checks can reach the retry path. A durable maximum of four build attempts per rendered content/retry epoch survives control-plane restarts; exhaustion reports
DeploymentRetriesExhaustedinstead of retrying indefinitely. Control-plane-side, affecting any engine on Dokploy. Live-verified on Dokploy with the previously blocked Neo4j deployment converging after one new build and native query/marker checks passing; the original terminally failed create operation remains historical. Budget exhaustion/restart persistence is test-covered, not a live fault-injection pass. - Keycloak omitted resource defaults: each omitted CPU, memory or disk field receives its catalog default independently, including requests supplying disk only. Control-plane-side, live dry-run verified on Dokploy; other provider paths are covered by tests, not a new live defaulting pass.
ProxySQL
- Unchanged MySQL/PXC proxy configuration no longer reloads into ProxySQL every minute, avoiding brief replica routing and failed writes. Agent-side, live-verified on Zerops PXC with 975/975 writes acknowledged and read back, zero failures and no runtime LOAD/SAVE events over 15 agent-apply cycles. Group Replication and Dokploy/Nomad live re-proof of this particular fix is not established. Identical configuration-file rewrites/log messages may still occur; they do not imply a runtime reload.
Upgrade notes v0.4.1 → v0.4.2
Follow the upgrade guide. Take a consistent control-plane state backup and preserve the original master key separately before deploying:
git fetch origin --tags && git checkout v0.4.2
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs- No new SQLite migrations:
internal/store/migrationsstill ends at017_backup_drills.sql. Retain pre-upgrade state and the old executable; reverting the control plane does not undo engine or infrastructure changes. - No new resource minimums or dependency/native-artifact lock changes. The v0.4.1 floors still apply: Group Replication and PXC need at least 2 GiB RAM per data member; single-member MySQL retains its 1 GiB floor. No automatic resize is performed.
- Changed behavior: Dokploy has the bounded build-attempt budget and explicit exhaustion condition described above; Temporal rejects unsupported initial data sources; Keycloak defaults partial resource requests; closed-window recovery restores only admitted exposure, leaving deferred changes pending.
- AGENT ROLLOUT — MySQL/PXC:
karactl clusters upgrade-agentscovers PostgreSQL (including Citus), Supabase, MongoDB, RabbitMQ and Elasticsearch only. It does not cover MySQL/PXC. Their agent-side switchover/clustered-restart, Group Replication restore-follower re-clone, PITR binlog archiving, ProxySQL reload and clone/startup diagnostics fixes reach existing clusters only by re-creating them: verified backup → clone/restore into a new v0.4.2 cluster → verify member builds, native data and client writes → switch clients. PXC startup durability also needs new agents. Preserve an independent verified recovery path before deletion. Restarting the control plane or database process does not install new agent code. Full-backup metadata admission and restore stop-hold release themselves need only the control-plane update. - OpenBao, Neo4j, Weaviate, Keycloak and Temporal: the fixes introduced in v0.4.2 relative to v0.4.1 are control-plane-side, so a control-plane upgrade is enough for those fixes; no re-creation is needed for them. These engines still do not support
upgrade-agents. If coming from v0.4.0, also follow the v0.4.1 agent guidance: this release does not retroactively install older agent-side fixes.
Known issues
- PostgreSQL administrator password rotation remains unavailable. The revealed
postgrespassword cannot be rotated in place. Follow administrator credential containment; do not treat a control-plane upgrade as credential rotation. - ProxySQL can treat a healthy MySQL Group Replication member as offline (also present in v0.4.0 and v0.4.1). The proxy's health monitor can connect to a member without TLS, which the member rejects; ProxySQL then logs
Access denied for user 'kk_monitor'andAssuming offlinefor that member and sends it no traffic, although the member is ONLINE in the group. If Group Replication elects that member primary, writes through the proxy can fail. Seen on Zerops; other providers may be affected. Don't disable TLS or change the monitor account to work around it. A fix is planned for v0.4.3; existing clusters will need their proxy redeployed to pick it up. - MySQL point-in-time recovery makes member agents use more and more memory (v0.4.2; in v0.4.0 and v0.4.1 the normal trigger is blocked because MySQL full backups fail there). The binlog archiver decodes each interval's binlogs in memory, so busy members need a lot of memory per archive pass. Separately, once a cluster has about a day or more of archived binlogs, the control plane's routine recovery-coverage checks start timing out, and each one leaves memory behind in the member agent — up to about 0.5 GB per hour, even when the cluster is idle and whatever the archive interval. Eventually
mysqldis killed for running out of memory, restarts repeatedly and refuses writes; a Group Replication cluster can lose its majority and stay without a writer until an operator intervenes. Acknowledged writes are preserved. Workaround: disable point-in-time recovery on all MySQL, Group Replication and PXC clusters until the fix (spec.backup.pitr.enabled: false, applied withkaractl clusters patch CLUSTER -f change.yaml); full backups keep working. Disabling it stops further growth but doesn't free memory already used:karactl clusters restartdoesn't help, because it restarts only the database and proxy processes. To free it, make a resource change (for example CPU or memory), which restarts members one at a time; do this in a maintenance window, expect a short write interruption while the writer moves, expect downtime on single-member clusters, and don't do it on a cluster that has already lost its majority. Don't lower binlog retention to work around it. A fix is planned for v0.4.3. - MySQL Group Replication doesn't recover by itself after all members go down at once (also present in v0.4.0 and v0.4.1). After a whole-group outage — for example a power loss or all nodes rebooting — or after suspending and resuming a Group Replication cluster that keeps its volumes, the cluster can stay Degraded with no writer, reporting
GroupReformationBlocked(UnsafeOfflineGTID), even though every member has the same data. The check that compares members before re-forming the group times out. Recovery: don't force a bootstrap or reset GTIDs; restore the cluster in place from its latest full backup (karactl restores create CLUSTER NAME --backup FULL_BACKUP --yes). Data written after that backup is lost, because point-in-time restore isn't available on a cluster without a writer in this release. Until the fix, take full backups frequently and avoid suspending Group Replication clusters. PXC is not affected. A fix is planned for v0.4.3. - PXC after in-place restore on Zerops: following either full or PITR restore, one member can be flagged
GTIDHistoryDivergedand the cluster reports Degraded although checked data and writes are correct. Investigation is ongoing. Verify data and replication natively; the flag can block incremental backups and PITR continuity, and a new full backup has not reliably cleared it. This was not reproduced on Dokploy in the completed native checks. - Despite the clone fix above, creating a multi-member Percona XtraDB Cluster from a backup never completes (also present in v0.4.1). This applies to clones from full, incremental and point-in-time backups on every provider. The create operation fails after about 20 minutes; the new cluster stays not Ready with all members unreachable and no proxy. Your source cluster and its backups are not affected. Delete the failed cluster. A fix is planned for v0.4.3.
- Replacing a Percona XtraDB Cluster member or scaling PXC in never completes (also present in v0.4.0 and v0.4.1). Automatic healing of PXC members (restart or rejoin of the same member) is not affected. The new member joins, but the old member's retirement is rejected repeatedly; the operation stays Updating or Scaling and fails after about 20 retries, the old member stays in Removing, and further operations on the cluster are blocked. The remaining members keep serving reads and writes with reduced redundancy. Cancelling only helps before the old member starts retiring. Workaround: don't replace PXC members or scale PXC in until the fix. If an operation is already stuck, leave the remaining members alone (don't retry, restart, scale back or remove volumes), keep a validated backup, and upgrade the control plane once the fix is released — it is control-plane-only and completes the stuck retirement without new member agents. Group Replication and other engines are not affected. A fix is planned for v0.4.3.
- Full backups can run out of memory on single-member MySQL clusters with 1 GiB of memory (also present in v0.4.0 and v0.4.1). The kernel can kill
mysqld, causing the member to restart and the backup to fail. Workaround: give MySQL members at least 2 GiB of memory before taking full backups (3 GiB recommended for Group Replication members), plus headroom for your workload. A fix is planned for v0.4.3. - Nomad live coverage of the new engines is incomplete. Implemented runtime support does not establish lifecycle/recovery acceptance; no new Nomad pass for the fixes above is claimed.
- Temporal is not supported on Zerops: membership requires private IPv4 networking while Zerops private networking is IPv6. Use Dokploy or Nomad within their documented capability limits.
- Weaviate cannot start on Zerops (also present in v0.4.0 and v0.4.1): members never become Ready because Zerops private networking is IPv6-only and Weaviate's cluster address is not advertised for IPv6. Nomad and Dokploy are not affected. Use Weaviate on Dokploy or Nomad until the next release; a fix is planned for v0.4.3.
- Dokploy may retain empty Docker volume objects after deletion; data wiping and object removal are distinct. See Dokploy lifecycle limits.
- PostgreSQL can remain Updating while healthy members acknowledge binding/access configuration;
ConfigurationPendingexplains the wait and backups still require readiness. This release does not remove that convergence latency.
Review limitations before deployment. The control plane remains one process with one SQLite writer. Preserve its state, master key and independent provider configuration; control-plane state recovery does not recover engine data.
Experimental status and live coverage
OpenBao, Temporal, MySQL, PXC, Neo4j, Weaviate, Keycloak, Citus and Data MCP remain EXPERIMENTAL. None has complete live acceptance across Zerops, Dokploy and Nomad. The table lists scoped successful evidence available for this release, including earlier checks carried forward where stated; it is not a full provider pass or production certification. “Not established” includes checks not run or not completed. No subsequent-release work is counted here.
| Component | Zerops | Dokploy | Nomad |
|---|---|---|---|
| OpenBao | Earlier create/KV/snapshot/resize/idle/delete checks; no new rolling/clone re-proof | Rolling-status fix, snapshot clone Ready, native KV durability, suspend/resume recovery, Raft scale 3→5; broader lifecycle checks | Earlier Raft/resource and PostgreSQL-storage failover checks only; new fixes not live re-proven |
| Temporal | Unsupported | Earlier create/mTLS/workflow, healing and source-primary restart checks; new admission fix test-only | Not established |
| MySQL Group Replication | Formation/proxied SQL, clustered restart, full+incremental, incremental restore, single-target clone and PITR data checks | New Group Replication lifecycle/recovery acceptance not established | No Group Replication formation/recovery pass established; earlier single-member proxy SQL only |
| PXC | Formation, full+incremental, archive cadence, PITR/full restore data and ProxySQL write continuity; post-restore Degraded issue remains | Formation, full+incremental, archive cadence and native PITR data; no divergence flag in completed watch | Earlier three-member formation only; new backup/recovery fixes not live re-proven |
| Neo4j | Create, marker, resize, dump and clone Ready with TLS-verified marker | Failed-initial-build retry recovery, native user/query/marker checks; earlier TLS/healing | Not established |
| Weaviate | Cannot start (known issue above) | Earlier retained-store readiness, backup/restore and maintenance checks; new clone gate test-only | Not established |
| Keycloak | Earlier create/OIDC, healing, binding rotation, resize and idle | Disk-only resource-default dry-run; earlier OIDC, healing/binding and maintenance-route checks | Earlier create/healing/resize/idle; no new defaulting re-proof |
| Citus | Earlier create/scale-out/healing/retained-backup recovery/distributed SQL/fresh backup | Earlier create and worker-healing checks | Not established |
| Data MCP | Not established | Not established | Not established |
| Provider recovery/exposure | Closed-window PostgreSQL public IPv6 resume and TLS data check | OpenBao/RabbitMQ/PostgreSQL closed-window recovery and deferred-exposure fences; initial-build retry | New recovery/retry fixes not live re-proven |
See engine/provider availability for implemented capabilities, which are separate from live acceptance.
Assets
karakoram-v0.4.2-linux-amd64.tar.gzandkarakoram-v0.4.2-linux-arm64.tar.gz: control plane/member agent, embedded UI,LICENSEandNOTICE.karactl-linux-amd64,karactl-linux-arm64,karactl-darwin-amd64,karactl-darwin-arm64,karactl-windows-amd64.exe.- Native artifact locks and existing native bundle inventory are unchanged from v0.4.1. Upstream-only artifacts retain their locked official URLs or exact-byte mirrors; Weaviate is not redistributed as a Karakoram release asset.
- Verify
checksums.txtand its SSH signature before installation, following release verification.
Documentation: https://karakoram.dev/docs/