Repository navigation
Releases
Karakoram v0.4.9
Summary
Karakoram v0.4.9 adds agent heartbeats, shared platform snapshots and less frequent reconciliation for idle clusters. It fixes Dokploy proxy-rollout recovery and admission, database replacement placement, and cancellation of confirmed-loss MySQL Group Replication replacement.
Read the upgrade guide and limitations before upgrading. The Zerops API error-code fix is included since v0.4.8, not new in this release. Control-plane upgrades do not change selected database engine versions.
New features
- Agent heartbeats and a platform read budget. Updated agents push status and subscribed read-only native observations every three seconds over their existing authenticated connection. Steady clusters use shared platform snapshots and reconcile every 30 minutes; availability is still evaluated every five seconds. Lifecycle decisions, admission and pinned-dial authorization retain fresh reads. Old agents continue using live pulls until upgraded.
- Forced fresh reconciliation.
karactl -n NS clusters reconcile CLUSTERrequests an immediate admitted fresh pass without changing the specification. Repeated requests are coalesced with 30-second minimum spacing; command acceptance does not mean agents have applied a configuration. - Placement diagnostics and repair.
FailureDomainsSeparatedidentifies shared voting-member nodes, insufficient eligible nodes and unknown placement. Dokploy/Nomad replacement planning excludes surviving data-member nodes. Existing co-location is detected, not automatically moved. - Request and heartbeat metrics. Monitoring distinguishes platform reads by caller, snapshot use, steady-observation acceptance/fallback and heartbeat ingestion. Configuration HTTP traffic is counted separately from provider GETs.
Fixes
Affected versions identify documented released exposure; live coverage is scoped below. Earlier releases are not individually established unless named.
| Fix | Affected released versions | What changes |
|---|---|---|
| Dokploy request amplification, including PostgreSQL ProxySQL configuration-pull peer resolution | Dokploy polling: v0.2.0–v0.4.8; PostgreSQL proxy peer checks confirmed in v0.4.7/v0.4.8 | Control-plane snapshots and parallel observation reduce repeated status reads. Upgraded agents use certificate-bound conditional configuration pulls; unchanged steady inputs avoid repeated peer renders, while actual renders still validate peers freshly. Dokploy API-key accounting writes remain. Active operations can still generate large read bursts. |
| Dokploy GR status timeouts and falsely failed proxy-count changes | v0.4.3–v0.4.8 | Concurrent bounded observations and snapshots correct the serialized-read mechanism. This does not fix every network/latency failure or the long-operation status issue below; the earlier advice to keep two proxies on Dokploy no longer applies. |
| Paused Dokploy ProxySQL updates leave recovery stuck | v0.4.7/v0.4.8; MySQL, PXC, PostgreSQL and Supabase | Authoritatively paused/rolled-back attempts fail promptly. Repair the cause, then PATCH a different valid count/resource target. A correction already accepted but unapplied on the old control plane rolls forward after upgrade. Identical apply and forced reconcile do not re-arm a failed receipt. |
| Dokploy can retire serving proxies before a replacement is ready | v0.4.7/v0.4.8 | Fixed control plane plus fixed proxy agents enforce opted-in task/member/configuration-revision admission. Unapplied startup stays pending; stale Ready revisions remain rejected. Upgrade old marked proxies before count/resource changes. Spare capacity and executor-restart caveats remain. |
| PostgreSQL/Supabase proxy cancellation does not settle an accepted rollout after a lost reply | v0.4.7/v0.4.8 | Request-bound settlement observes the original serving update without another upload. Cancellation does not prove the update never committed; terminal failure can still require manual recovery. |
| Cancelling joined confirmed-loss MySQL GR replacement leaves the cluster stuck | v0.4.0–v0.4.8 | Joined replacement completes forward; unjoined work rolls back the new allocation and retains confirmed loss for a fresh replacement. Already-stuck v0.4.7/v0.4.8 clusters settle their rollback owner after the control-plane upgrade. Observe native membership, cleanup and ownership: cancellation acknowledgement is not completed recovery. |
| Replacement can co-locate voting data members on Dokploy/Nomad | Documented Dokploy exposure: MySQL GR/PXC v0.4.0–v0.4.8; PostgreSQL HA/Supabase v0.2.0–v0.4.8 | Preserve predecessor identity and exclude survivor nodes during import planning. Insufficient eligible placement refuses admission rather than falling back to an occupied node. Inspect and repair existing co-location separately; already-frozen imports retain their original request. |
| Neo4j creation admitted with insufficient native-memory headroom | v0.4.7/v0.4.8; live failure observed on Dokploy | New creates and engine-resource edits require at least 2 GiB minimum RAM on every provider. Unchanged smaller legacy shapes remain admitted, not automatically resized. No agent rollout is needed for this admission correction. |
| Data MCP companions miss a changed agent archive | v0.4.7/v0.4.8 | Enabled MongoDB/PostgreSQL/Supabase companions refresh once when the deployed archive changes, even if upstream URLs do not. Maintenance windows gate the refresh; companion access can be interrupted. Same-build control-plane restarts do not cause refreshes. |
Behaviour changes
- Idle drift repair is less frequent. Listing-visible platform changes are normally detected on the 60-second shared listing cadence; changes outside that listing can wait for the 30-minute deep reconcile. Force a fresh pass when needed. Health evidence retains its 15-second lease and is not made fresh by republishing it.
- Configuration pulls remain every 60 seconds, with heartbeat notices for changed inputs. The notice scheduling allowance is up to seven seconds, excluding transport/server time and failed-heartbeat retries. Listing-only changes still wait for listing detection and fresh render/apply work. With failing heartbeats, conditional pulls can take about 125 seconds before rendering; old agents retain unconditional minute pulls.
- Healthy GR replacement and scale-in require full GroupView-capable surviving agents. Upgrade healthy survivors first. Already-in-flight legacy work and confirmed-loss recovery with positively identified legacy survivors have documented weaker ONLINE/count exceptions; unknown or unreachable capability is not an exception.
- Downgrade is restricted. There is no SQLite schema migration, but older writers can discard new state intent and older control planes cannot admit tasks from opted-in Swarm templates. Do not downgrade while managing those templates or active new-protocol work.
MySQL/PXC lost-volume replacement
Automatic confirmed-lost-volume replacement remains experimental and off by default for both GR and PXC. With it off, confirmed loss leaves the cluster Degraded with guidance to verify survivors and run karactl clusters replace-member CLUSTER MEMBER --wait.
Explicit opt-in remains KARAKORAM_MYSQL_AUTO_REPLACE_LOST_VOLUMES=true in the control-plane environment, followed by restart. Native majority, group/history and donor checks still apply; this is not forced-quorum or lost-data recovery. Group Replication quorum loss was observed after automatic replacement earlier; its cause is not established. The write-stall issue below is also unresolved.
How to upgrade
- Back up consistent complete control-plane state, preserve the original master key separately, and retain the previous executable with its pre-upgrade state. Keep independent engine-native backups.
- Upgrade the control plane first through the provider's singleton procedure, preserving state, key, endpoint identity and provider settings. Use matching CLI tooling. Leave automatic MySQL/PXC lost-volume replacement off unless deliberately accepting its experimental risk.
- If legacy structural work or confirmed loss already exists, follow the upgrade guide's recovery order and observe the existing operation; do not submit another replacement over it. Already-stuck confirmed-loss cancellation owners can settle under the fixed control plane.
- In maintenance, run
karactl -n NS clusters upgrade-agents CLUSTER --wait --timeout 30m. Native process restart alone does not upgrade agents. Upgrade healthy GR survivors before new healthy replacement, scale-in or placement repair, and old marked proxies before count/resource changes. Expect singleton downtime, retiring-session disconnects and possible GR handoff rejections; inspect ambiguous commits before replaying them. - Inspect
PLACEMENTGROUPandFailureDomainsSeparatedafter earlier replacements. Take a fresh full backup, verify quorum/replication and spare capacity, then follow the documented one-member-at-a-time repair. Do not drain nodes to force placement or edit frozen import state. - Verify native readiness, replication, configuration acknowledgements, application reconnection, companion access and backup/restore. Compare idle request metrics only after convergence and agent rollout. Existing undersized Neo4j clusters need an intentional backed-up resize with singleton downtime; resource edits must meet the new minimum.
Known issues
- Deleting a retained backup can leave it stuck in
Deletingand unrestorable (also present since v0.1.0 for Elasticsearch, v0.4.0 for Weaviate, v0.2.0 for failed PostgreSQL backups). After source deletion with--keep-backups, Karakoram marks the recordDeletingbefore provider validation; a refusal can leave it there although no objects were deleted. This affects Elasticsearch r...
Karakoram v0.4.8
Summary
Karakoram v0.4.8 is a hotfix of v0.4.7 with one change: the Zerops provider classifies Zerops API errors by their error code instead of their HTTP status, so it works both before and after Zerops changes how its API reports missing objects. Features planned for the next feature release are deferred to v0.4.9.
Read the v0.4.7 → v0.4.8 upgrade guide before upgrading.
What was wrong / who is affected
Affected: every installation that manages clusters on Zerops (v0.2.0–v0.4.7), from the day Zerops deploys its API change. Nomad and Dokploy installations are not affected.
Zerops is changing how its API reports a missing object: the error code stays the same (for example serviceStackNotFound or projectNotFound), but the HTTP status changes from 400 to 404, and the error metadata gains the object's ID. Zerops has not announced a date and the switch has no compatibility period.
v0.2.0–v0.4.7 treat every HTTP 404 as "this service or app version no longer exists" and every other 4xx answer as a rejected request. Deleted services, missing app versions and expired platform Processes are still recognized after the change. But those versions then read any other missing object as a missing service or app version:
- A create or import naming a service type or version Zerops does not offer, or a resource change Zerops refuses because of a missing type version or autoscaling profile, is no longer reported as rejected. MySQL, PXC and Weaviate resource changes that would stop at a rejection keep retrying instead. When Zerops refuses to delete a PostgreSQL or MySQL member's volume for such a reason, the volume is not moved to the orphaned-volume inventory and retirement keeps retrying.
- A project-level answer such as
projectNotFound(for example a wrong project ID or a deleted project) reads as the cluster's services being absent. Keycloak, Supabase and Temporal component checks report missing components, Temporal deletion can be reported complete, and MySQL formation skips such members, instead of failing with a clear rejection. - A 404 from a gateway or an unknown API route reads as a deleted service.
No data loss path was identified, because creating or replacing a service fails against the same missing project, but these misreadings can stall operations and hide the real cause.
v0.4.8 decides by the error code, with HTTP 400 and HTTP 404 alike. Only a missing service or app version counts as absence, and a missing Process as expiry. Another object's not-found code stays a rejected request, as before. An HTTP 404 without a known Zerops error code is handled like a temporary platform outage: it is retried, a write answered this way has an unknown outcome, and it is never evidence that a service was deleted. The new error metadata is not shown in diagnostics.
How to upgrade
Follow the upgrade guide.
- Back up consistent control-plane state and preserve the original master key separately, retaining the previous executable with its state.
- Upgrade the control plane through the provider's documented singleton path. This applies the fix. Member agents never call the Zerops API: v0.4.7 agents keep working with the v0.4.8 control plane and need no rollout. No database restart, engine upgrade, backup or configuration change is needed.
Upgrade before Zerops deploys its change if you can; upgrading afterwards is also safe.
Verification
Automated gates (on the release tree): build, go vet (default, e2e and integration tags), gofmt, staticcheck 2026.2.2 (default and integration tags, no findings), OpenAPI generation check, Web UI type check, lint, 409 tests and build, and govulncheck (no vulnerabilities affecting Karakoram code) passed. The full go test ./... passed, as did go test -race on the Zerops provider and the packages that import it. The tree stayed clean after testing. The native artifact lock is unchanged from v0.4.7, and the 19 Karakoram-built bundles are byte-identical to the v0.4.7 assets.
New unit and replay tests cover every moved not-found code with both HTTP 400 and HTTP 404, recorded old and new error bodies (including the new metadata) decoded through the real API client, and a 404 without a Zerops error envelope. Running the new tests against v0.4.7 code fails the new-API cases. No live Zerops test was run against the changed API, because Zerops has not deployed it yet.
Known issues
-
Deleting a retained backup can leave it stuck in
Deletingand unrestorable (also present since v0.1.0 for Elasticsearch, v0.4.0 for Weaviate, v0.2.0 for failed PostgreSQL backups). After source deletion with--keep-backups, Karakoram marks the recordDeletingbefore provider validation; a refusal can leave it there although no objects were deleted. This affects Elasticsearch repositories with multiple snapshots or an edited storage bucket/prefix, Weaviate after storage bucket/endpoint changes, and failed or incomplete PostgreSQL/Supabase records. Workaround: do not delete individual retained Elasticsearch snapshots while other snapshots remain; preserve the original storage binding. A stuck Elasticsearch snapshot is still intact and can be restored natively by registering its repository read-only in Elasticsearch. For Weaviate, restore the original storage binding and retry deletion: deletion can then complete, but Karakoram restore remains unavailable while the record isDeleting. There is no supported way to reset a stuck record in these releases; do not delete repository objects by hand. A fix is planned for v0.4.11. -
A failing PostgreSQL WAL archive is not reported as Degraded (also present in v0.2.0–v0.4.6). When WAL archiving fails, the cluster shows phase
Updatingand conditionWALArchive=False(ArchiveFailing), but it is not markedDegraded, noClusterDegradedwarning is emitted and the degraded metric stays 0. Unarchived WAL keeps accumulating until the data volume fills, after which the member stops. For single and high-availability clusters, while archiving fails, point-in-time restore to a timestamp is refused, even for times before the failure; restoring a named full, differential or incremental backup still works, and point-in-time restore becomes available again after the next successful archive. Workaround: alert on theWALArchivecondition (karakoram_cluster_condition{type="WALArchive",status="False"}) and on risingkarakoram_postgres_archiver_failed_total/karakoram_postgres_archive_pending_segments, and checkkaractl clusters get CLUSTERconditions after configuration changes and failovers. MySQL/PXC binlog archive failures and MongoDB PBM inventory failures similarly leave the clusterReady; watch theBinlogArchiveandBackupconditions. A fix is planned for v0.4.11. -
After a lost MySQL Group Replication member is replaced, commits can stall for several seconds at a time for about five minutes. When
karactl clusters replace-memberreplaces a member whose volume was lost, the remaining members keep trying to reach the removed member for about five minutes after its name is withdrawn. During that time a single commit can take 4–5 seconds roughly every 14 seconds; writes continue between the stalls and no acknowledged write is lost. A client with a short read timeout can get error 2013 (Lost connection to MySQL server) at COMMIT even though the transaction was committed. Replacing a healthy member is not affected. Workaround: treat a timeout or error 2013 at COMMIT as an unknown outcome — check whether the write was applied before retrying, or make writes idempotent; a longer client read timeout (for example 10 seconds) can turn these errors into slower commits; expect slower commits for about five minutes after a lost-member replacement; keep automatic replacement of lost MySQL volumes (KARAKORAM_MYSQL_AUTO_REPLACE_LOST_VOLUMES) disabled. A fix is planned for v0.4.10. -
During a long operation, status can report a serving cluster as Degraded (status only; also present in v0.4.7 and v0.4.9). While
karactl clusters upgrade-agents,karactl clusters replace-memberor a count or resource change waits on a stalled platform rollout, status can showHealthy=Unknown(ObservationStaleorObservationFailed),AgentsCurrent=Unknownand phase Degraded although the members and proxies keep serving. One episode on Dokploy under heavy read load lasted 692 seconds. The status clears when health observation succeeds again or the operation ends. Workaround: check serving directly withkaractl clusters membersand client checks, and the operation's progress withkaractl operations get; resolve the stalled rollout (for example, free scheduler capacity) rather than restarting members or the control plane. A fix is planned for v0.4.10.
All known issues and limitations of v0.4.7 remain in v0.4.8, unchanged: -
On Dokploy, a ProxySQL count or resource change can leave a cluster stuck (v0.4.7 only; ProxySQL in front of MySQL, PXC, PostgreSQL and Supabase). If the new proxy task does not become healthy (for example, the node lacks CPU or memory), Docker Swarm pauses the update. The existing proxies keep serving, but the operation stays Running for about 15 minutes before failing, cannot be cancelled, and afterwards the cluster stays Degraded: changing the proxy settings back is accepted but not applied, and agent upgrades are blocked. There is no supported way to recover in this release. Before changing proxy count or resources on Dokploy, make sure the nodes have spare capacity for the new proxy tasks. If an update pauses, keep the existing proxies running, do not edit the Swarm service by hand, and upgrade when the fix is released, or open an issue. A fix is planned for v0.4.9.
-
**Cancelling the replac...
Karakoram v0.4.7
Summary
Karakoram v0.4.7 adds agent upgrades for every member-agent engine, PostgreSQL administrator rotation, automatic MySQL/PXC PITR re-anchoring and Temporal on Zerops. It includes engine security updates and fixes for Data MCP recovery, PXC formation and failed-deployment recovery.
Read the full upgrade guide and limitations before upgrading. The v0.4.4 and v0.4.5 MySQL hotfixes and v0.4.6 compression-library security fix are inherited, not new fixes in this release. Database engine versions do not change merely because the control plane is upgraded.
New features
- Agent upgrades for all engines.
karactl clusters upgrade-agents NAME --waitnow covers MySQL single/Group Replication, PXC and ProxySQL, OpenBao, Keycloak, Temporal, Neo4j and Weaviate, alongside the previously supported engines. Rollouts check native readiness and maintenance admission. Singletons can have downtime; this is not an engine-version upgrade. - PostgreSQL administrator rotation. PostgreSQL, Supabase and Citus can rotate the managed
postgressuperuser withkaractl -n NS system-users rotate NAME postgres --yesafter every data-member agent is upgraded. A sealed journal supports forward recovery after interruption. Existing sessions are not revoked; external administrator clients must reconnect with the new password. Supabase credential-consuming services restart, and Citus can have temporary cross-group authentication failures while rotation converges. - MySQL/PXC PITR re-anchoring and restorable intervals. With PITR and backup storage configured, a stable archive-writer change can trigger a new full backup automatically. The PITR API, CLI, MCP and restore UI expose all proven recovery intervals, including older ranges hidden by the previous summary. Gaps remain unrestorable; a new full does not recover the gap after the previous writer's last proven endpoint.
- Temporal on Zerops. Updated agents support private IPv6 in combined and split layouts. Both the control plane and agents must be upgraded.
- Serving-preserving ProxySQL changes. Eligible count/resource changes on Zerops retain serving containers; eligible Dokploy deployments use serial start-first replacements with native serving admission. Dokploy needs a supporting deployed runtime and spare capacity. Legacy runtimes can require one disruptive deployment first. Nomad retains its count-only path, not resource-preservation guarantees. Existing connections can still be lost when their instance is removed or replaced.
- MCP quality improvements. Built-in tools have fuller descriptions and documented parameter defaults and limits. Output schemas are opt-in with
KARAKORAM_MCP_OUTPUT_SCHEMAS=truein the control-plane environment, followed by a restart; the default isfalseto keep discovery smaller. Successful calls retain text andstructuredContentin both modes. Data MCP diagnostics classify upstream failures without exposing upstream messages or credentials. - Provider-neutral web UI. Creation, access, settings and login wording now follows the selected provider. Zerops-specific networking and pricing guidance appears only for Zerops namespaces. UI tests and synthetic-API screenshots cover this change; no live-installation browser check is claimed.
Fixes
Affected versions below distinguish confirmed source exposure from likely inheritance; live coverage is scoped in Verification.
| Fix | Affected released versions | What changes |
|---|---|---|
| Data MCP keeps using a closed companion connection | v0.3.0–v0.4.5; v0.4.6 likely inherits it | Reconnect after a closed connection or missing session. Tool listing retries once; tool execution is never replayed. No control-plane restart is needed. |
| Native health goes stale while a Data MCP companion hangs | v0.4.1–v0.4.6 | Bound companion readiness observation so database health stays fresh while Data MCP is reported unavailable. |
| Fresh PXC creation fails behind ProxySQL on Nomad | v0.4.3–v0.4.6 | Checkpoint native formation without waiting on a proxy whose serving listener is deliberately fenced. Zerops and Dokploy are not affected by this released formation deadlock. |
| Weaviate response-header/write-request timeouts are misclassified | v0.4.4–v0.4.6 | Preserve timeout classification after connection establishment; do not replay writes. This is a diagnostic correction, not a new availability guarantee. |
| Corrected resources are ignored after a failed first deployment | v0.4.3–v0.4.5 source exposure; v0.4.6 likely inherits it | Apply corrected desired resources before a re-armed deployment on Zerops, Dokploy and Nomad. Patch resources, then run clusters retry-deployments; the resource patch alone does not re-arm it. Preserve existing services, volumes and data. Zerops resource application was demonstrated, but the affected Functions service did not reach native Ready; Dokploy was not live-tested. |
| False PXC GTID-history incidents after restart/rejoin under writes | v0.4.3 confirmed; v0.4.6 likely inherits it; other tags not individually established | Require stationary comparison evidence before latching prefix divergence. Recovery full backups can be admitted during an already-applied update waiting for health, subject to native convergence. The latter exception was not live-exercised. |
Other corrections:
- MySQL rolling changes update proxies first; cluster mutations are isolated per cluster from backup cleanup.
- PXC comparison budgets and history-persistence diagnostics are more robust.
- Neo4j verifies the complete administrator bootstrap on replay. Already-written incorrect completion markers still require the documented stopped-member repair.
- Nomad host-volume placement accounts for available task capacity.
- The example Nomad job accepts a positive
member_ephemeral_disk_mbvalue without the unsupportedtostringfunction (affects v0.2.0–v0.4.6; rendered offline, not live-registered with a positive value).
Engine and dependency updates
Engine-version changes are explicit operations, separate from control-plane and agent rollout. New defaults do not automatically patch existing databases.
| Component | New versions / changes |
|---|---|
| Percona Server for MongoDB | 7.0.45-24 and 8.0.34-15, with 8.0.34-15 the default; database tools 100.19.1-2. Includes memory-disclosure and denial-of-service fixes. |
| mongosh for MongoDB 7.0/8.0 | 2.14.1 replaces 2.10.0, fixing compromised bundled TLS roots. MongoDB 6.0 retains its old shell and unresolved exposure. |
| RabbitMQ / Erlang | RabbitMQ 4.3.7 and Erlang 27.3.4.19 include broker and runtime security fixes. |
| Elasticsearch, Kibana, Logstash and APM Server | Aligned 9.5.6, 9.4.9 and 8.19.24 security patches; 9.5.6 is the default. |
| OpenBao / plugins | OpenBao 2.7.2, AWS KMS plugin 0.1.1 and MongoDB plugin 0.0.2. Review the changed SSH wildcard-principal default and verify unseal and lease lifecycles. |
| Temporal / UI | Server 1.31.5 and 1.32.2, with 1.32.2 the default; UI 2.58.0. ValidateWorkerDeploymentVersionComputeConfig now requires namespace Write permission. |
| Neo4j / Java | Neo4j Community 2026.10.1 and Temurin JRE 21.0.13+9, including snappy-java and Java trust-store fixes. |
| Weaviate | 1.39.15 and 1.40.3, with 1.40.3 the default; includes backup-authentication and security-dependency updates. |
| PostgreSQL vector profiles | pgvector 0.8.7 native bundles for PostgreSQL 16/17/18 and Supabase PG17 fix CVE-2026-103484. Installed extensions receive per-database SQL updates after member rollout. |
| Supabase | Envoy 1.39.6, Vector 0.61.0, Storage 1.81.5, Realtime 2.153.3, Edge Runtime 1.77.9 and imgproxy 4.0.21; rebuilt postgres-meta, Studio, Supavisor and Logflare bundles with patched runtimes. |
| Supabase PG17 extensions | Bundle 17.11.1-2 updates pg_net, Vault, pgsodium, pg_graphql, pg_jsonschema, HypoPG and index_advisor, including pg_graphql and pgsodium security fixes. |
| Control-plane dependencies | go-jose/v4 4.1.8 and embedded SQLite 3.54.1; Vite 6.4.5 and minimatch 11.0.4 security updates affect development/build tooling. The v0.4.6 klauspost/compress 1.20.2 fix remains included. |
The MongoDB MCP 3.1.0 native bundle is rebuilt with body-parser 2.4.0, @hono/node-server 2.1.6 and basic-ftp 6.2.6, rather than only changing the npm lock. Supabase, pgvector and imgproxy security rebuilds are pinned in the native artifact lock. Imgproxy v4 changes ETag generation and SVG rasterization; review the compatibility notes before rollout.
EOL create blocks: MongoDB 6.0 and Percona Server/PXC 8.0 are recovery/migration-only. Fresh empty creates are rejected. Existing workloads, supported forward upgrades and validated same-major restore-to-new recovery remain available; this does not extend vendor support.
Fixed upstream, pending vendor packages: etcd 3.5.35, pgBackRest 2.59.3 and set_user 4.3.0 were unavailable in the checked Percona Noble indexes. This release retains etcd 3.5.33, pgBackRest 2.59.1 and set_user 4.2.0 and does not claim those issues fixed. pgBackRest's weak-encryption issue occurs on OS RNG failure; existing weak stanza subkeys remain weak even after a binary update. Follow the upgrade guide's private key check and new-repository/full-backup procedure rather than assuming an incremental repairs old keys.
Behaviour changes
- PostgreSQL/Supabase proxy count increases wait for the maintenance window when the backend or deployed runtime cannot preserve serving. All proxy resource changes and count reductions remain maintenance-gated.
- MongoDB and PostgreSQL/Supabase Data MCP companions redeploy once after upgrade, honoring each cluster's maintenance window. Data MCP is briefly unavailable for that cluster; database members and volumes are not replaced. Preparation is bounded and failed work reports the retry command....
Karakoram v0.4.6
Summary
Karakoram v0.4.6 is a security hotfix of v0.4.5 with one change: the
github.com/klauspost/compress library used by the MongoDB driver is updated
from v1.18.7 to v1.20.2 to fix
GO-2026-6865
(GHSA-fpmh-frq8-7rj8).
The MongoDB driver stays at v2.4.3. Features planned for the next feature
release are deferred to v0.4.7.
What was wrong / who is affected
Affected: v0.4.5 and earlier installations that manage MongoDB (PSMDB)
clusters.
A crafted zstd-compressed MongoDB wire response could make the decoder panic
instead of returning an error. The control plane administers MongoDB clusters
from background reconciliation, so such a panic stops the whole control-plane
process: the API, UI and management of every cluster it serves are
interrupted until it restarts, and a server that keeps sending crafted
responses can repeat the crash.
Impact is a control-plane crash (denial of service); no data corruption is known.
Neither the advisory nor the inspected code shows code execution or corruption
of database or control-plane data. A control-plane crash does not stop database
processes directly, but management and in-flight operations are interrupted
until the control plane is back. The Karakoram impact is traced from source; no
end-to-end exploit was reproduced.
Exposure needs a malicious or compromised managed MongoDB server, or response
injection. An attacker must get crafted responses onto a control-plane
connection to a managed mongod or mongos, by compromising or controlling
that server, or by impersonating it or injecting into its connection.
- With the default
requireTLSmode, verified TLS blocks ordinary network
attackers; a compromised server can still send crafted responses over its
valid TLS connection. - In
disabled,allowTLSandpreferTLSmodes the control plane can use
plaintext connections, so an attacker who can alter that network path can
also inject responses. - Storing data, ordinary database credentials or Karakoram API access alone are
not sufficient. - Disabling MongoDB network compression is not a mitigation: the driver
decodes a compressed response from the server even when compression was not
negotiated (Karakoram does not request compression for these connections). - Installations without MongoDB clusters never decode MongoDB responses and are
not exposed; upgrading still removes the vulnerable library.
How to upgrade
Follow the v0.4.5 → v0.4.6 upgrade guide.
- Back up consistent control-plane state and preserve the original master
key separately, retaining the previous executable with its state.
Upgrade the control plane through the provider's documented singleton
path. This applies the fix. No database restart, engine upgrade, backup or
configuration change is needed, and v0.4.5 member agents keep working with
the v0.4.6 control plane. - Optionally roll MongoDB member agents. Agents run the same executable and
also contain the library, but only connect to themongod/mongoson their
own member over loopback: exploiting one requires control of that local
endpoint or the ability to inject its replies, subject to TLS verification.
The process that crashes is that member's agent, not the shared control
plane; the cluster effect depends on the member's role and quorum. To remove
the library from MongoDB members too, run
karactl clusters upgrade-agents NAME --waitper MongoDB cluster in a
maintenance window. Agents for other engines never open MongoDB connections
and need no relaunch.karactldoes not contain the library.
After the upgrade, reachable agents reporting the older build appear in
status.agents as outdated (unreachable or unreported builds as unknown) until
rolled; the AgentsCurrent condition is informational and does not degrade
clusters. If
automatic agent rollout is enabled, it rolls outdated agents of supported
engines as for any release.
Verification
Automated gates (on the exact release tree): Go 1.27.3 build, vet
(including e2e and integration tags), gofmt, race tests on every package that
links the updated library, OpenAPI check, native-artifact Python tests, and UI
typecheck/lint/tests/build passed. The full Go test run passed except one
wall-clock latency test (a 5-second sweep budget) in an unchanged package that
does not use the library: it failed repeatedly while the build host was heavily
loaded, then passed in a full package rerun and five consecutive runs at much
lower host load. The tree stayed clean after testing.
Staticcheck has 0 new findings versus v0.4.5. govulncheck no longer reports
GO-2026-6865 and reports 0 vulnerabilities affecting Karakoram's code. The
module build list differs from v0.4.5 only in github.com/klauspost/compress,
and the native artifact lock is unchanged.
The fix is a dependency update; no exploit was reproduced against Karakoram,
and no live MongoDB cluster test was run for this hotfix.
Known issues
-
Creating a Percona XtraDB Cluster on Nomad fails (also present in v0.4.3–v0.4.5; Zerops and Dokploy are not affected). A new three-member PXC cluster on the Nomad provider fails after about 5–9 minutes with
GTIDMappingCertificationBlocked: the last member is never started and the failed create shuts down another member. No existing data is affected; this happens only when creating a cluster. There is no workaround on Nomad: PXC requires its proxy. Until the fix, use MySQL Group Replication on Nomad, or create PXC clusters on Zerops or Dokploy. A fix is planned for the next release. -
A cluster can report Degraded while its Data MCP companion is unresponsive (also present in v0.4.1–v0.4.5). If a cluster's Data MCP companion accepts connections but stops answering — for example while it restarts or when it hangs — the control plane waits too long for it, the cluster's health observations go stale, and the cluster reports
Degraded(ObservationStale) for as long as the companion stays unresponsive, even though the database itself is healthy. It recovers by itself once the companion answers again. If it persists, disable Data MCP for that cluster (spec.dataMcp.enabled: false, applied withkaractl clusters patch CLUSTER -f change.yaml) and re-enable it later. A fix is planned for the next release.
All known issues and limitations of v0.4.5 remain in v0.4.6. Fixes for the Data
MCP closed-connection, Zerops failed-deployment resource recovery and false PXC
GTID-history incident issues are planned for v0.4.7; the other limitations keep
their individually documented status. -
Data MCP can keep failing with
tool listing unavailableafter the
connection to a cluster's companion drops (v0.3.0 onwards). The gateway
keeps reusing the closed connection until about 10 minutes without Data MCP
requests for that cluster, or until the control plane restarts. Databases are
not affected. Workaround: pause Data MCP requests to that cluster for about
10 minutes, or restart the control plane (clusters keep running). -
False PXC GTID-history incidents after restart/rejoin remain unfixed.
Rolling resource/parameter changes, upgrades, crashes, agent restarts, TLS
rotation, suspend/resume or restore while writes continue can falsely degrade
a cluster, block incrementals and freeze PITR coverage. A full backup clears
the flag about 20 seconds after completion. If an already-applied change is
only waiting for health and manual backup/cancellation are refused, add an
enabled full-backup schedule a few minutes ahead, preserving existing
schedules and changing nothing else in the patch; restore the original
schedules afterwards. This needs backup storage and healthy members. If the
change is still rolling, that scheduled full can remain Pending: leave the
serving cluster alone until the fix. -
On Zerops, raising resources after a member's first deployment exhausts
retries still does not redeploy at the new size. Optional extras/proxies
can be disabled and re-enabled at the larger size; a never-started scale-out
data member can be scaled in and added again. Delete/recreate only a brand-new
cluster with no data. Never delete an existing data-bearing cluster to repair
a stuck replacement. Engine-specific cancellation/timeout and leave-in-place
guidance remains in the upgrade guide.
Neo4j and Data MCP are unaffected. -
PostgreSQL's revealed
postgresadministrator password cannot be rotated in
place; follow credential containment. -
Temporal remains unsupported on Zerops because it requires private IPv4.
-
Nomad live coverage remains incomplete. Earlier scoped results are not a
current full lifecycle/recovery pass; experimental engine/provider status is
unchanged. -
PXC's
GTIDHistorySafechecks GTID compatibility, not row equivalence.
Privileged unlogged writes can cause invisible row drift; restrict privileges
and verify data independently. -
PITR still displays only the interval with the latest end, and still
does not bridge physical archive-writer changes: take a fresh full from the
current writer after every writer change and verify the window. -
PXC can report
GTIDHistorySafe=Unknown/QuorumComparisonUnavailableafter
scale-out or on idle clusters; self-clearance is not guaranteed.
See limitations
before deployment.
Documentation: https://karakoram.dev/docs/
Karakoram v0.4.5
Summary
Karakoram v0.4.5 is a hotfix of v0.4.4 with only two MySQL fixes:
PITR windows that stopped extending on writers provisioned with copied data,
and first full backups rejected when executed GTIDs were empty. Features
planned for the next feature release are deferred to v0.4.6.
What was wrong / who is affected
v0.4.0–v0.4.4 — both defects:
MySQL single-instance, Group Replication and PXC sources
with imported executed GTIDs absent from their local binlog history can stop
extending PITR coverage. A typical trigger is a writer change to a member
provisioned by Group Replication clone or PXC state snapshot transfer (SST),
including during ordinary cluster formation. A restored singleton or the
initial writer of a restored/cloned cluster can also be affected without a
writer change. Ordinary expiration of formerly logged binlogs alone is not
this defect. Binlog uploads continue and BinlogArchive can remain healthy,
but karactl backups pitr-window CLUSTER stops advancing past the full's own
snapshot file. Repeated fulls on that same affected writer do not repair forward
coverage. Previously proven targets and physical full backups remain usable.
A first full backup can also fail with metadata_invalid if
gtid_executed is empty. XtraBackup legitimately writes only a binlog filename
and position in that case; the old parser required a third GTID column. This
shared parser affects MySQL single-instance, Group Replication and PXC on all
providers whenever the source has no executed GTIDs; not every fresh clustered
source has an empty set. Managed bootstrap account creation is not binary-logged
and does not guarantee a nonempty set. Previously, a committed binary-logged
application write followed by a new full could avoid this parser error.
What the fixes do
Coverage and restore selection verify the physical full's exact source,
snapshot file and position to distinguish imported baseline history from
logged dependencies. Imported/purged-only GTIDs need not reappear in every
Previous_gtids event. Missing exact anchors and post-snapshot dependencies
still fail closed; no GTID reset or unsafe history skip is introduced.
Verified same-source chains that cannot extend now report
PITR=False/CoverageBlocked, the reason and verified endpoint, plus a
deduplicated Warning event. The safe prefix remains usable. BinlogArchive
reports upload health independently and can remain True; idle time alone
does not raise a coverage failure.
The coverage-cache identity changes so corrected agents rebuild existing fulls'
proofs. An existing full blocked only by imported baseline history can resume
across previously archived transactions if the physical source is unchanged
and the exact anchor and contiguous dependencies remain available. This does
not reconstruct missing logs or bridge different physical writers.
Backup metadata now accepts a missing or empty GTID column while still requiring
and validating the binlog filename and position. No application write is
needed with corrected agents. Existing failed fulls stay failed: request a new
full rather than assuming the upgrade repairs or retries the old operation.
How to upgrade
Follow the v0.4.4 → v0.4.5 upgrade guide.
- Back up consistent control-plane state and preserve the original master
key separately, retaining the previous executable with its state.
Upgrade the control plane first through the provider's documented path. - Relaunch every affected MySQL/PXC data-member agent through a genuinely
changed CPU or memory resource target. Rolling maintenance stops and
redeploys members with the current executable. Check capacity and majority
health, preserve a validated backup and independent recovery path, and plan
handoff interruptions (downtime for a singleton). Verify member builds,
replication and client writes. Unchanged patches,karactl clusters restart
and control-plane restarts do not replace agents. MySQL/PXC have no
supportedupgrade-agentsoperation in v0.4.5, although the shared CLI
exposes that command for other engines. These fixes need data-member agents,
not a PXC proxy rollout. - Re-enable PITR if disabled (
spec.backup.pitr.enabled: true) only after
all affected data agents run corrected code. Preserve native binlogs and
archived objects while coverage checks finish. - Take a successful new full from the current writer. Failed first fulls
stay failed and no application write is required. A same-source existing
full can conditionally resume after its proof is rebuilt, as described
above. The rolling GR/PXC relaunch moves the writer at least once, so
establish coverage from the new full; do not assume PITR beyond the old
verified endpoint into the old-to-new-baseline gap is recovered. Preserve
earlier backups and proven windows. Purged unarchived logs cannot be
reconstructed by upgrading or taking a full. - Check
karactl backups pitr-window CLUSTERadvances past the previously
frozen endpoint after post-backup writes and completed uploads/coverage
checks. InspectPITR,BinlogArchive, Warning events and writer-agent
diagnostics. A snapshot-only window is not continuous PITR coverage.
Do not downgrade the control plane after rolling agents. In particular,
versions below v0.4.3 cannot negotiate the pending/freshness coverage contract.
Control-plane rollback does not undo database or infrastructure changes.
Verification
Live acceptance (Nomad only): Both fixes passed live checks on Nomad with
identical patches in a development build; the release build itself was verified
by automated gates. Corrected agents
extended previously frozen Group Replication full-backup windows; a real
PITR clone recovered all 26,223 expected acknowledged rows, with no missing
or changed rows and none of 951 later rows. A fresh single-instance source's
first successful full after the upgrade recorded an empty GTID set; its
earlier full attempts had failed. Native metadata bytes were not captured
in that live run; the two-column format is covered by deterministic parser
tests and checked against the native tool's source. PXC/SST was not live-tested.
Automated gates (on the exact release tree): Go 1.27.3 build, vet
(including e2e and integration tags), gofmt, full Go tests, race tests on the
changed MySQL packages, OpenAPI check, native-artifact Python tests, and UI
typecheck/lint/tests/build passed. The tree stayed clean after testing.
Staticcheck has 0 new findings versus v0.4.4 (20 normal / 22 integration
baseline findings); govulncheck reports 0 reachable vulnerabilities.
The native artifact lock is unchanged versus v0.4.4, and all 21 local
native bundles match the tagged lock. No new MongoDB MCP bundle is included.
The fixes include regression coverage for imported baseline GTIDs with exact
physical anchors (including native binlog-derived fixtures), old and newly
taken fulls on the same source, missing dependencies and regressing logged
boundaries, safe-prefix retention and deduplicated CoverageBlocked Warnings
despite healthy uploads, and valid absent/empty GTID metadata columns alongside
invalid required coordinates. This describes the included tests, not release-gate
execution. Native artifact locks are unchanged by these two fixes.
Known issues
- Data MCP can keep failing with
tool listing unavailableafter its connection to a cluster's companion drops (also present in v0.3.0–v0.4.4). If the connection between the control plane and a cluster's Data MCP companion closes — for example after the companion restarts or a network interruption — the gateway keeps reusing the closed connection, and Data MCP requests for that cluster keep failing; the stale connection is only dropped after about 10 minutes with no Data MCP requests for that cluster, or when the control plane restarts. Your databases are not affected. Workaround: stop sending Data MCP requests to that cluster for about 10 minutes, or restart the Karakoram control plane (clusters keep running while it restarts). A fix is planned for the next release. - False PXC GTID-history incidents after restart/rejoin remain unfixed.
Rolling resource/parameter changes, upgrades, crashes, agent restarts, TLS
rotation, suspend/resume or restore while writes continue can falsely degrade
a cluster, block incrementals and freeze PITR coverage. A full backup clears
the flag about 20 seconds after completion. If an already-applied change is
only waiting for health and manual backup/cancellation are refused, add an
enabled full-backup schedule a few minutes ahead, preserving existing
schedules and changing nothing else in the patch; restore the original
schedules afterwards. This needs backup storage and healthy members. If the
change is still rolling, that scheduled full can remain Pending: leave the
serving cluster alone until the fix. Fix planned for v0.4.6, not this hotfix. - On Zerops, raising resources after a member's first deployment exhausts
retries still does not redeploy at the new size. Optional extras/proxies
can be disabled and re-enabled at the larger size; a never-started scale-out
data member can be scaled in and added again. Delete/recreate only a brand-new
cluster with no data. Never delete an existing data-bearing cluster to repair
a stuck replacement. Engine-specific cancellation/timeout and leave-in-place
guidance remains in the upgrade guide.
Neo4j and Data MCP are unaffected. Fix planned for v0.4.6. - PostgreSQL's revealed
postgresadministrator password cannot be rotated in
place; follow [credential containment](https://github.com/KarakoramDev/Karakoram/blob/v0.4.5/docs/upgra...
Karakoram v0.4.4
Summary
Karakoram v0.4.4 is a hotfix of v0.4.3 with two MySQL/PXC point-in-time
recovery fixes: binlog archiving that silently stopped (archive failures are now
visible), and point-in-time restores/clones rejected when the writer's clock
lags the control plane. Unreleased features previously described as v0.4.4 are
deferred to v0.4.5.
What was wrong / who is affected
v0.4.3 only: MySQL single-instance, Group Replication and PXC clusters with
PITR enabled can silently stop archiving after schema/account statements,
including changes made through Karakoram. The parser expected a space before a
mysqlbinlog Query event, but native output uses a tab. Standalone queries
without Xid, including no-op schema/account statements, could trigger the stall;
ordinary atomic statements carrying Xid could continue, so a successful
CREATE DATABASE, CREATE TABLE, ALTER, CREATE USER or GRANT alone does
not prove archive health. The archive loop discarded errors and BinlogArchive
could still appear healthy. Full backups are unaffected.
v0.4.3 only: MySQL point-in-time restores and clones could be rejected with
"agent returned PITR verification predating restore admission" whenever the
writer node's clock was even milliseconds behind the control plane's. Earlier
releases lack this check.
What the fixes do
The parser accepts native space/tab Query headers and correctly handles
standalone queries, transaction-wrapped row/statement DML, compressed
transactions and durable XA prepare boundaries. Failed decode, rotation or
upload attempts report BinlogArchive=False/ArchiveFailed, a Warning event
with reason ArchiveFailed and a sanitized cause in the writer agent log. Only
a completed successful pass clears the failure; failures never extend proven
coverage.
Corrected agents automatically retry retained, unarchived closed binlogs in
native order on the next scheduled pass, preserving durable receipts. Keep
native logs until catch-up finishes. Purged unarchived binlogs leave a
permanent PITR gap: upgrading or taking a new full cannot reconstruct them.
A missing pending log reports unarchived_binlog_purged; a new successful full
establishes a recovery baseline after the gap.
Point-in-time restore and clone admission now trusts the agent's per-request
freshness attestation instead of comparing timestamps from different clocks.
This fix is control-plane-only: it works with v0.4.3 agents and needs no
agent relaunch.
How to upgrade
Follow the v0.4.3 → v0.4.4 upgrade guide.
- Back up consistent control-plane state and preserve the original master
key separately, retaining the previous executable with its state.
Upgrade the control plane first through the provider's documented path. - Relaunch every affected MySQL/PXC data-member agent through a genuinely
changed CPU or memory resource target (rolling maintenance; on Group
Replication and PXC this costs one writer move). Check
capacity and majority health, preserve a validated backup and independent
recovery path, and plan handoff interruptions (downtime for a singleton).
Verify member builds, replication and client writes. Unchanged patches,
clusters restartand control-plane restarts do not relaunch agents;
MySQL/PXC have noupgrade-agentscommand in v0.4.4. This correction
needs data-member agents, not a PXC proxy rollout. - Re-enable PITR if disabled (
spec.backup.pitr.enabled: true) only once
all affected data agents run corrected code. Retain native logs for catch-up. - Take a successful full backup from the current writer before relying on
PITR. On single-member clusters (and any cluster whose writer did not change
during the relaunch), retained binlogs from the stuck period are archived and
point-in-time recovery covers that period again, starting from the original
full backup — verified by automated tests, not live. On Group Replication
and PXC clusters the relaunch moves the writer, so coverage restarts from the
next full backup and the stuck period cannot be restored to a point in time;
restore from a full backup instead. Preserve earlier backups. - Check
karactl backups pitr-window CLUSTERadvances after post-backup
writes and completed archive/coverage checks; inspectBinlogArchive,
Warning events and writer-agent diagnostics. A snapshot-only window is not
continuous PITR coverage.
Do not downgrade the control plane after rolling agents. In particular,
versions below v0.4.3 cannot enforce the agents' pending/freshness coverage
contract. Control-plane rollback does not undo database/infrastructure changes.
Verification
Live acceptance (Nomad only): a v0.4.3 Group Replication cluster with a
proxy, stuck in the archiving stall, was upgraded control plane first (no
member was redeployed), then its agents were relaunched with a rolling memory
change. The retained backlog was archived on the first pass; after a full
backup on the new writer, the recovery window advanced across new database,
user and grant changes made through the API without stalling again. A
single-member cluster archived past CREATE DATABASE. A point-in-time clone
and an in-place point-in-time restore succeeded with the writer's clock behind
the control plane's and matched the target state exactly. An induced
600-second storage failure reported BinlogArchive=False/ArchiveFailed with
exactly one Warning event and recovered afterwards. No acknowledged write was
lost. PXC (same archive code) and other providers were not live-tested for
this hotfix.
Automated gates (on the release commit): build, go vet (including e2e and
integration tags), gofmt, the full Go test suite, race tests of the changed
packages, staticcheck (no new findings vs v0.4.3), govulncheck (no reachable
vulnerabilities), UI typecheck/lint/test/build and the OpenAPI check passed.
Native artifacts are unchanged since v0.4.3.
The fixes add regression tests for native Query-header whitespace and captured
Percona mysqlbinlog schema/account, no-op, row/statement DML, compressed and XA
fixtures; retained-backlog recovery after restart versus purged-log gaps;
archive failure/recovery without fabricated coverage; the ArchiveFailed
condition and deduplicated Warning event; and restore/clone admission with the
agent clock behind or ahead of the control plane.
Known issues
- MySQL point-in-time recovery can silently stop extending after the writer changes in Group Replication and PXC clusters (also present in v0.4.0–v0.4.3). When the writer moves — automatic failover, switchover, or rolling maintenance — to a member whose data was copied in rather than replayed from its own binlogs, binlogs keep uploading and
BinlogArchivestays healthy, but the end of the point-in-time window stops advancing. This applies to most Group Replication and PXC clusters, because members that join when the cluster forms are copied by Group Replication clone or PXC state transfer, and to any cluster created from a backup, cloned or restored in place. Restores to points before the freeze still work, and full backups are not affected. A new full backup does not restart coverage. To check, compare the window end fromkaractl backups pitr-window CLUSTERwith the current time. Until the fix, rely on full backups and schedule them as often as your recovery point requires. A fix is planned for the next release. - On Zerops, raising the resources of a member that failed its first deployment doesn't take effect (also present in v0.4.3). If a newly added member, extra component or proxy exhausts its deployment retries — for example because it was given too little memory to start — a later resource increase is recorded but never deployed, and retrying deployments redeploys it at the old size. These members never started, so they hold no data. Recovery: for optional extras and proxies, disable the component and enable it again with the larger resources; for a data member added by scaling out, scale back in to remove it, then scale out again with the new resources; for a brand-new cluster that never became Ready, delete it and create it again. If a replacement member is stuck in a cluster that already holds data, never delete the cluster; instead: for PostgreSQL and Supabase, cancel the replace operation (
karactl operations cancel OPERATION), which removes only the new member and keeps the original; for MySQL Group Replication and PXC, cancel the replace operation before the old member starts retiring; for MongoDB, wait for the operation to time out (about 2 hours), which removes the new member; for RabbitMQ, Elasticsearch, Weaviate and OpenBao, leave the cluster as it is and upgrade once the fix is released. After cancelling or a timeout, check that the new member is actually gone before making further changes. Neo4j and the Data MCP companion are not affected. A fix is planned for the next release. - False PXC GTID-history incidents after restart/rejoin remain unfixed
(also present in earlier v0.4 releases). Rolling resource/parameter changes,
upgrades, crashes, agent restarts, TLS rotation, suspend/resume or restore
while writes continue can falsely degrade a cluster, block incrementals and
freeze PITR coverage. A full backup clears the flag about 20 seconds after
completion. If an already-applied change is only waiting for health and
manual backup/cancellation are refused, add an enabled full-backup schedule
a few minutes ahead, preserving existing schedules and changing nothing else
in the patch; restore the original schedules afterwards. This requires
configured backup storage and healthy members. If the change is still rolling
through members, that scheduled full can remain Pending: leave the serving
cluster alone until the fix. **Fix planned for v0.4.5, no...
Karakoram v0.4.3
Karakoram v0.4.3 is a fix release for v0.4.2, led by a high-priority correction for MySQL point-in-time recovery (PITR) memory growth. On v0.4.2, PITR could exhaust a member's memory, OOM-kill mysqld repeatedly and leave Group Replication without a writable majority. This release also improves recovery, PXC history certification, maintenance handoffs and IPv6 listeners. No new engines are introduced. Follow the v0.4.2 → v0.4.3 upgrade guide.
MySQL / PXC PITR: memory and recovery
Unless stated otherwise, fixes below affect v0.4.0–v0.4.2. Control-plane-side fixes apply with the control-plane upgrade; agent-side fixes require the affected member executable to relaunch with v0.4.3 code. Provider evidence is scoped to the stated operation, not complete provider acceptance.
-
High priority — bounded PITR memory: the archiver now streams binlog decoding into a compact index instead of retaining the complete decoded log. Recovery-coverage checks cache unchanged history, stream rebuilds and extend verified appends instead of loading every historical sidecar on every poll. S3 listing now uses iterators with owned cancellation: returning early from a listing previously left producer goroutines holding listing pages. Routine coverage polling could therefore grow agent memory even on an idle cluster; increasing the archive interval did not prevent it. Per-call S3 transports are also closed. Agent-side for MySQL/PXC, with control-plane-side listing corrections for Elasticsearch, PostgreSQL/Supabase and Weaviate. Those other-engine paths do not establish the same continuous idle-leak impact. v0.4.2 is affected with PITR enabled at the default 300-second interval. v0.4.0/v0.4.1 share the affected archiver: normal configuration refreshes usually starved the default timer, but archiving could run with intervals below 60 seconds or when refreshes were delayed. Their ordinary full-backup admission defect also normally blocked post-full coverage polling, without removing exposure where accepted recovery inventory existed. The idle listing defect itself is independent of archive interval once coverage checks run. Live-verified on Nomad: with PITR enabled, three 2 GiB GR members, archive intervals of 1 and 300 seconds, at least 1,000 archived files, idle periods and about 1 MiB/s writes, member-agent RSS stayed at or below 48 MiB in steady state, peaking at 54.9 MiB during a 3.8 GB full backup. The idle cell used 30 ms injected storage latency; 30-minute memory ranges were at most 5.2 MiB, with no out-of-memory kills. This is a scoped acceptance measurement, not a capacity guarantee. For contrast, released v0.4.2 code reproduced listing-page leakage in tests; earlier live runs of affected code reached about 1 GB per idle agent and an agent was OOM-killed. The repo-wide listing-lifecycle corrections are verified by tests; no equivalent fresh live memory-bound measurement is claimed for Zerops or Dokploy.
Streaming and caching remove the demonstrated unbounded allocation paths; object-key inventories, fragmented GTID histories and returned recovery windows still have input-dependent size. No unconditional fixed-byte bound for arbitrary recovery repositories is claimed.
-
Coverage converges after relaunch or failover: long archive histories could leave PITR coverage pending indefinitely after an agent restart, writer failover or a large archive batch. A single agent-owned background refresh now persists progress across short polls and restarts; ordinary polls read compact state rather than rescanning history. The updated control plane negotiates explicit pending/progress responses without claiming an unverified window. Completed windows from other full backups remain available: a covered restore target can be admitted, while an uncovered target remains pending if an eligible baseline is still verifying or queued, rather than being prematurely rejected as out of range. With a reachable source agent, restore admission requires verification started after that admission request, not a cached status result. At most 32 baseline entries reside in memory; compact proofs and resumable checkpoints remain on disk. Metadata cleanup expires baselines unrequested for 24 hours and retains up to 256 inactive baselines in addition to the protected current working set; these are not backup-retention limits. v0.4.0–v0.4.2; agent-side background convergence plus control-plane-side pending/freshness handling. Live-verified on Nomad for coverage convergence and agent relaunch on long history without injected latency; convergence under storage latency is verified by tests.
-
PITR restore without an online writer: single-instance MySQL and Group Replication can validate archived recovery evidence without requiring an
ONLINEsource writer, instead of returning HTTP 503 when recovery is needed most. Archived coverage, lineage and recovery-safety checks still apply. Cloning into a new cluster still needs the source cluster specification; this does not mean that its database writer must be online. v0.4.0–v0.4.2; control-plane-side. Live-verified on Nomad for an in-place point-in-time restore of writerless GR after a real whole-group outage and a separate PITR restore after writer changes, with exact recovered data and post-target absence controls. This does not relax the physical-source lineage rules below. Single-instance and clone paths are verified by tests only. -
Complete GTID headers: binlog
Previous-GTIDsspanning multiple server UUIDs are preserved, avoiding truncated recovery coverage and loss of predecessor-history protection. v0.4.0–v0.4.2; agent-side; verified by tests only. Upgrading does not repair already-uploaded incomplete sidecars; take and verify a new full recovery baseline. -
Honest frozen-window reporting: after an archive-writer change, recovery can remain frozen at the last provable point until a new full backup establishes a baseline. This applies to GR primary changes from automatic failover or rolling maintenance, and to PXC when the physical archive source changes; newer uploads from a different source cannot extend the previous baseline. Restores within the proven old window, up to its frozen end, remain available. Status now reports
WindowFrozenWriterChangedwhen newer foreign-writer uploads are excluded and distinguishes upload lag from the recoverable endpoint instead of implying that healthy uploads protect current writes. v0.4.0–v0.4.2; control-plane-side reporting correction. The frozen-endpoint behavior was live-verified on Zerops; the corrected status reason is verified by tests only. Take a full backup from the current writer before relying on newer targets; the lineage restriction itself is not fixed by this release.
MySQL / PXC memory sizing and full backups
- Single-member full-backup OOM: default InnoDB sizing reserves memory for backup children, and XtraBackup streams with
--parallel=1 --read-buffer-size=1M. Backup errors retain safe reason codes and native exit statuses/signals; a SIGKILL is a possible OOM indication, not proof by itself. v0.4.0–v0.4.2; control-plane-side pool sizing plus agent-side streaming/diagnostics. Live-verified on Dokploy with a fresh 1 GiB member, full backup, clone and exact marker readback; no OOM. - Small clustered-member headroom: GR/PXC members at or below 2 GiB now default to a 768 MiB InnoDB buffer pool, and GR uses a 128 MiB message cache. Explicit operator pool settings retain precedence. v0.4.0–v0.4.2 memory exposure; control-plane-side dynamic pool sizing plus agent-side GR cache rendering. Dynamic pool shrink on existing agents was live-verified on Nomad; the new cache default is verified by tests and belongs to the fresh-agent PITR acceptance above, not a separate all-provider capacity guarantee. Existing GR agents continue rendering pool/4, or 192 MiB with the new 768 MiB pool, until those agents relaunch.
Measured full-backup peaks with PITR off, on Dokploy, were:
| Topology | Member limit | Backup-member peak |
|---|---|---|
| Single MySQL | 1 GiB | 84.4% |
| PXC | 2 GiB per member | 85.0% |
| Group Replication | 2 GiB per member | 90.7% |
These scoped measurements are not workload capacity guarantees or PITR headroom measurements. Admission floors remain 1 GiB for single MySQL and 2 GiB for GR/PXC. At least 3 GiB per GR/PXC member is recommended, plus workload and backup headroom. Old singleton agents with only the new control plane peaked at 96% of 1 GiB: update the agents rather than treating the thin margin as safe capacity.
PXC formation, history and member lifecycle
- GTID history after restore and during formation: compatible restored inverse-namespace histories no longer cause false divergence latches. Acknowledged history differences are bounded, tied to member identities and retired when consumed. Formation and restore now certify replicated GTID mapping before declaring it safe; empty joins are serialized and eligible followers use a seed-pinned SST remedy. Missing or inconsistent proof fails closed with explicit operator exits, rather than treating matching rows or Galera health as a history certificate. Existing incident latches are diagnosed, not silently cleared. v0.4.0–v0.4.2; control-plane-side history/lifecycle handling plus agent-side certification. Live-verified on Zerops for formation, PITR/full restore, serving latches and documented exits; Dokploy for formation, scale and full/PITR clone data. Nomad re-proof of these corrections is not established. Privileged unlogged writes remain outside this check.
- Member retirement completes: replacing a member or scaling in no longer fails repeatedly because the retiring member's reload omits its own Galera identity. An already-stuck re...
Karakoram v0.4.2
Karakoram v0.4.2 is a fix release for v0.4.1, based on further live checks, principally on Zerops and Dokploy. It corrects resume and exposure recovery, engine formation, backups and restores, clone readiness and deployment retries. There are no new engines or features. Follow the v0.4.1 → v0.4.2 upgrade guide.
Fixed
Unless stated otherwise, these defects affected v0.4.0 and v0.4.1. “Control-plane-side” means a control-plane upgrade applies the fix; “agent-side” means existing member executables must be replaced through the supported procedures in Upgrade notes. Provider-specific live evidence is scoped below, not a promise of complete acceptance on every provider.
Maintenance, resume and exposure
- High — v0.4.1 regression: resume/recovery under a closed maintenance window could fail to restore already-admitted public TCP or HTTPS exposure, and a resume operation could remain running. Previously admitted exposure now recovers without admitting deferred exposure changes or acknowledging them prematurely. This is control-plane-side, covering PostgreSQL/Supabase (including its gateway), MongoDB, RabbitMQ, Elasticsearch, OpenBao, MySQL/PXC, Neo4j, Weaviate and Keycloak. Live-verified on Dokploy with a held OpenBao resume completing without an override or second resume, RabbitMQ suspend/resume, PostgreSQL public-route recovery and a deferred private-to-public change remaining private until overridden; on Zerops, PostgreSQL public IPv6 exposure recovered after one resume with TLS-verified data readback. Other affected engine paths have regression-test coverage, not a separate live exposure-recovery pass. v0.4.0 is not affected.
OpenBao
- v0.4.1 regression — briefly reporting Ready during a rolling update: Updating/Upgrading now remains visible between member replacements until the admitted generation converges; a deferred update still leaves a healthy cluster Ready. Control-plane-side, live-verified on Dokploy, with no intermediate Ready transition during the roll. v0.4.0 is not affected by this intermediate Ready defect.
MySQL / PXC formation and switchover
- High — MySQL Group Replication now forms: generated group-recovery credentials fit MySQL's length limit; legacy single-member credentials are corrected before first adoption into a group. The credential-generation/adoption fix is control-plane-side. Never-formed legacy groups must be deleted and freshly created; do not force bootstrap or edit coordinator state. Formation and proxied SQL were live-verified on Zerops; a new Dokploy/Nomad Group Replication pass is not established for this release.
- High — PXC now forms on Zerops: startup durably writes required files and their parent directories without a filesystem-wide synchronization call rejected by the platform. This includes agent-side launcher/durability changes and control-plane reload/observation handling. Live-verified on Zerops with three synchronized members, ProxySQL and TLS-verified writes; fresh PXC formation also passed on Dokploy.
- High — MySQL Group Replication primary switchover and clustered restart: the native primary-election function now receives a validated literal UUID rather than an unsupported prepared parameter. Agent-side, live-verified on Zerops: clustered restart completed with three ONLINE members and 367 acknowledged writes read back without loss. Dokploy/Nomad re-proof is not established.
- Group startup failures retain the newest useful diagnostic and omit credential-bearing records; clone failures report a safe stage and, when available, the MySQL error number instead of an undifferentiated failure. These are agent-side diagnostics, not permission to expose raw SQL or error messages. Sanitized startup diagnostics were exercised on Zerops; clone-stage diagnostics identified the follower recovery failure there. Redaction and diagnostic-boundary cases are covered by tests.
MySQL / PXC backups, restore and PITR
- High — full and incremental recovery chains: the control plane now recognizes XtraBackup's native full-backup checkpoint type, unblocking full backups and dependent incremental chains, restore/clone baselines and PITR baselines. Control-plane-side; take and verify a new full backup rather than relying on a previously failed one. Full plus incremental backups passed on Zerops for MySQL Group Replication and PXC, and on Dokploy for PXC.
- High — in-place restore completes re-formation: the control plane releases durable server-stop holds so restored seeds and followers can start. MySQL Group Replication followers fence clients while native re-clone replaces old data, and retry a rejected native start only when copying never began. Stop-hold release is control-plane-side; Group Replication follower re-clone is agent-side. Live-verified on Zerops for Group Replication restore from an incremental chain and PXC full restore, with per-member data checks and writes after recovery. PXC uses its own Galera recovery path. PXC PITR restore also passed natively on Dokploy. See the still-open Zerops PXC post-restore issue below.
- High — PITR binlog archiving: configuration refresh no longer continually postpones the archive timer; status uses completed archives and real lag rather than a missing archive index or refresh timing. Agent-side archiving plus control-plane status fixes, live-verified on Zerops for MySQL Group Replication and PXC: binlog/archive-index pairs appeared every 300 seconds and PITR recovered data to a point after the last full backup. PXC archive cadence and PITR data also passed on Dokploy. Nomad re-proof is not established.
Clones and create-from-backup
- OpenBao, MySQL/PXC, Neo4j and Weaviate now publish completed initial-restore receipts to the readiness gate, allowing a successful clone to report Ready and finish creation. Control-plane-side. Live-verified on Dokploy for OpenBao (the held operation completed without replacing containers/volumes), and on Zerops for MySQL restored from a full-plus-incremental chain and Neo4j with TLS-verified source-marker readback. PXC and Weaviate completion-gate fixes have tests; completed live clone acceptance for those engines is not established in the release evidence.
- Temporal now rejects unsupported initial data sources at admission instead of accepting a create-from-backup request it cannot complete. Control-plane-side, verified by tests; no live admission re-proof is claimed. Its separate restore functionality is unchanged.
Dokploy and Keycloak
- Dokploy initial-build retry, including Neo4j: failed first builds remain observable so ownership checks can reach the retry path. A durable maximum of four build attempts per rendered content/retry epoch survives control-plane restarts; exhaustion reports
DeploymentRetriesExhaustedinstead of retrying indefinitely. Control-plane-side, affecting any engine on Dokploy. Live-verified on Dokploy with the previously blocked Neo4j deployment converging after one new build and native query/marker checks passing; the original terminally failed create operation remains historical. Budget exhaustion/restart persistence is test-covered, not a live fault-injection pass. - Keycloak omitted resource defaults: each omitted CPU, memory or disk field receives its catalog default independently, including requests supplying disk only. Control-plane-side, live dry-run verified on Dokploy; other provider paths are covered by tests, not a new live defaulting pass.
ProxySQL
- Unchanged MySQL/PXC proxy configuration no longer reloads into ProxySQL every minute, avoiding brief replica routing and failed writes. Agent-side, live-verified on Zerops PXC with 975/975 writes acknowledged and read back, zero failures and no runtime LOAD/SAVE events over 15 agent-apply cycles. Group Replication and Dokploy/Nomad live re-proof of this particular fix is not established. Identical configuration-file rewrites/log messages may still occur; they do not imply a runtime reload.
Upgrade notes v0.4.1 → v0.4.2
Follow the upgrade guide. Take a consistent control-plane state backup and preserve the original master key separately before deploying:
git fetch origin --tags && git checkout v0.4.2
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs- No new SQLite migrations:
internal/store/migrationsstill ends at017_backup_drills.sql. Retain pre-upgrade state and the old executable; reverting the control plane does not undo engine or infrastructure changes. - No new resource minimums or dependency/native-artifact lock changes. The v0.4.1 floors still apply: Group Replication and PXC need at least 2 GiB RAM per data member; single-member MySQL retains its 1 GiB floor. No automatic resize is performed.
- Changed behavior: Dokploy has the bounded build-attempt budget and explicit exhaustion condition described above; Temporal rejects unsupported initial data sources; Keycloak defaults partial resource requests; closed-window recovery restores only admitted exposure, leaving deferred changes pending.
- AGENT ROLLOUT — MySQL/PXC:
karactl clusters upgrade-agentscovers PostgreSQL (including Citus), Supabase, MongoDB, RabbitMQ and Elasticsearch only. It does not cover MySQL/PXC. Their agent-side switchover/clustered-restart, Group Replication restore-follower re-clone, PITR binlog archiving, ProxySQL reload and clone/startup diagnostics fixes reach existing clusters only by re-creating them: verified backup → clone/restore into a new v0.4.2 cluster → verify member builds, native d...
Karakoram v0.4.1
Karakoram v0.4.1 is a fix release for v0.4.0, based on live checks on Zerops, Dokploy and Nomad. It corrects database provisioning and recovery, maintenance-window enforcement, repeated resource updates and misleading health reports. There are no new engines or features. Released October 28, 2026.
Fixed
PostgreSQL, profiles, bindings and Citus
- High — silent minor-version upgrades: installing extension profiles no longer lets APT replace an exact-pinned PostgreSQL server with a newer minor release. Running SQL versions are reported instead of assigned versions; persistent
VersionDriftdiagnostics block accidental downgrades and explain corrective upgrades, including upstream-only version observations. - High — restores stuck at access: ordinary managed-role password recovery no longer goes through binding-only mutation checks. Affected non-Citus restores with ordinary roles can resume after a control-plane update, without opening client access prematurely. Existing binding-role and Citus failures have stricter recovery requirements; follow the upgrade guide rather than editing HBA rules or submitting another restore.
- High — IPv6/dual-stack connectivity: PostgreSQL data listeners, ProxySQL writer/reader frontends and HBA source families now agree with IPv6-capable provider layouts. On Zerops, IPv4 clients no longer hit an IPv6-only proxy listener while dual-stack DNS advertises both families. TLS, authentication and restore access fences remain enforced.
- High — Citus creation, backup, scale-out and recovery: preserve coordinator voters and valid worker DCS endpoints; initialize archive stanzas for added worker groups; accept retained coordinated backups after source deletion; recover passwords node-locally across group primaries; and remap restored node registrations and outbound worker authentication to the new target. Readiness verifies worker SQL connectivity instead of reporting Ready with broken distributed queries. Aggregate backup sizes and named-point recovery status are now accurate.
- TimescaleDB extension updates use a fresh backend before the first library load. PostgreSQL status overlaps independent platform/native observations and reuses writer evidence for PITR checks. Agent rollouts refresh replication barriers after failover and prove catch-up on the authoritative primary; proxy agent identities are retained. Healthy members waiting only for configuration acknowledgements now report
ConfigurationPending, separately from binding credential publication. Replication-lag metric descriptions distinguish primary-side lag from replica-local replay backlog.
Keycloak and Temporal
- High — Keycloak repeated resource updates: compare resource values rather than pointer identity, retire redundant failed resize receipts safely, and acknowledge only fully applied generations. Idle reconciliation no longer keeps requesting unchanged platform resources.
- High — PostgreSQL proxy compatibility: new Keycloak deployments use JDBC simple-query mode, avoiding unsupported mixed result formats and named portals. Temporal uses text results even for explicitly prepared persistence/visibility queries. The PostgreSQL writer proxy and TLS verification remain in place.
- Keycloak and Temporal publish fresh positive availability evidence and observed agent identities instead of showing healthy native services as disconnected or Degraded. Keycloak hostname documentation distinguishes the configured issuer from the platform-generated HTTPS route.
OpenBao
- High — provisioning and resource maintenance: install archive-validation dependencies before preparation, and compare disk bounds separately from CPU/RAM so runtime-only resizes do not trigger unsupported disk growth or repeated rolls.
- Preserve explicit active/standby roles in public status. A formed cluster reports deferred changes and rolling updates without falling back to Initializing; independent seal/leader reads no longer add unnecessary serial observation latency.
MySQL / Percona XtraDB Cluster
- High — memory admission: MySQL Group Replication and PXC require 2 GiB minimum RAM per data member. Group Replication agents budget the message cache from member memory instead of leaving the upstream 1 GiB default alongside the buffer pool. Single-member MySQL retains its 1 GiB floor. These are minimums, not workload or total-memory guarantees; Group Replication still cannot form in this release (see Known issues).
- High — startup and routing: install the missing Group Replication RPC library dependency; observe SQL-ready pre-bootstrap nonmembers without treating an empty membership row as an outage; avoid native/container probes before provisioning; and join data backends before admitting proxy deployments. PXC prepares configuration before initial Galera bootstrap and uses numeric bind addresses separately from advertised peer identities.
- High — IPv6/dual-stack sockets: MySQL/PXC data and ProxySQL listeners follow the provider address-family policy rather than hardcoding IPv4 on Zerops.
- High — PXC access and archives: managed access avoids advisory locks prohibited by PXC strict mode, and archive admission accepts eligible synchronized Galera members instead of requiring Group Replication's
ONLINEstate.
Neo4j and Weaviate
- Neo4j in-place restore credential recovery is idempotent when the recovered password already matches. Agent configuration reload uses the engine action budget, and credential-recovery failures retain useful native failure categories instead of a misleading configuration error.
- High — Weaviate retained-store readiness: derive readiness from live Raft/database/leader evidence rather than an optional bootstrap flag after restart, allowing healthy retained stores and pending storage updates to converge without weakening quorum gates.
Operations and maintenance windows
- High — disruptive changes now respect closed windows: Keycloak, Temporal, OpenBao, MySQL/PXC, Neo4j and Weaviate honor maintenance admission, recovery ownership and rolling safety. Runtime-replacing HTTPS, hostname and public TCP route changes are gated too; deferred changes no longer replace runtimes behind a waiting operation. Recovery exemptions and previously admitted recovery remain available.
- Deferred desired-state operations remain Pending until execution starts, keep their generation-owned receipts, and emit maintenance-wait evidence. Maintenance conditions use the same policy evaluation as the public maintenance summary, with stable transition times and bounded calendar searches.
- Control-plane backup policy storage can be reset when disabled. Synthetic notification tests get distinct event identities so receivers do not deduplicate separate tests.
Status truthfulness
- High — false Degraded, false Ready and empty members: availability renewal is independent of slow reconciliation, including existing members in non-Ready lifecycle phases. Failed observations preserve known lifecycle/member inventory and terminal failures rather than publishing default phases and empty lists; missing evidence still fails closed. Citus distributed readiness and Keycloak/Temporal freshness are corrected as described above.
- Observed build identities are retained across engines and PostgreSQL proxies, so
AgentsCurrentis no longer Unknown merely because the control plane discarded reported identities. This informational condition does not imply support for agent rollout. Member autoscaling reports effective platform bounds, not scheduler reservations or stale intent.
Providers
- High — Zerops deployment admission: complete unfinished-process evidence is queried separately from indexed deployment history. Large terminal process histories no longer exhaust a fixed history cap and block unrelated deployments. Missing evidence remains a retryable admission wait, not a reason to assume deployment is safe.
- High — Zerops idle resource churn: unchanged resources and converged idle clusters no longer cause redundant platform updates/repeated resizing, notably for Keycloak. PostgreSQL and MySQL/PXC IPv6-capable listener fixes are described above.
- Dokploy deletion can clean up failed initial deployments without waiting for the original deployment failure budget. Runtime log tails select the newest retained tasks, preserve untimestamped diagnostics and redact secrets.
- Nomad resource changes preflight replacement feasibility before stopping members, retain applied bounds and terminal rejection evidence, and permit corrective resource intent. Missing individual log streams no longer discard sibling output. Unsupported Dokploy/Nomad disk growth is rejected before disruptive work, checking every existing provider volume.
CLI / API and artifacts
karactl backups createforwards all provider-advertised backup types, including dumps, incremental backups and Raft snapshots. Maintenance overrides work with request files. Identical cluster updates preserve in-flight work and explicit overrides rather than cancelling an operation at the same generation.- Explicit 1 GB disks are admitted for Keycloak, Neo4j and Weaviate instead of being rejected by a shared 5 GB request floor; engine/provider limits still apply. Agent logs come from member runtimes with provider errors classified appropriately.
- Source-build identity is separate from published artifact-release identity, avoiding attempts to fetch nonexistent native bundles named after development builds. An explicit published artifact release can be selected without replacing the running control-plane executable.
Upgrade notes
Follow the v0.4.0 → v0.4.1 upgrade guide. Take a consistent control-plane state backup and preserve the original master key separately before deploying:
git fetch origin -...Karakoram v0.4.0
Karakoram v0.4.0 expands the self-hosted control plane with a PostgreSQL platform, seven new engine providers and safer fleet operations. PostgreSQL gains PostGIS, pgvector and TimescaleDB profiles, owned service bindings and distributed Citus formations. OpenBao, Temporal, Percona Server for MySQL, Percona XtraDB Cluster, Neo4j Community, Weaviate and Keycloak join the engine catalog. Maintenance windows, explicit agent rollouts, control-plane backups, notifications and backup drills make day-two work visible and recoverable. Zerops remains the reference backend; Dokploy and Nomad use the shared native runtime contract with their documented capability limits.
New engines
These entries describe implemented interfaces, not production acceptance; each item's status applies across backends.
- OpenBao — EXPERIMENTAL until tested against live backends. Native OpenBao 2.7.1 with 1/3/5 members, integrated Raft or bound PostgreSQL storage, mandatory TLS, Shamir or immutable transit/AWS KMS auto-unseal, S3 Raft snapshots and leased database credentials for PostgreSQL, MongoDB and RabbitMQ. PostgreSQL storage delegates durable data recovery to its source cluster. See OpenBao.
- Temporal — EXPERIMENTAL until tested against live backends. Temporal 1.32.1 (also 1.31.4) and UI server 2.56.0, with combined or split stateless roles, PostgreSQL persistence/visibility through service bindings, namespace administration and coordinated source-backup manifests. Supported on Dokploy and Nomad; Zerops is not supported in v0.4.0, pending IPv6 private networking because membership currently requires a private IPv4 route to the bound PostgreSQL writer. Public UI access requires configured Dokploy HTTPS and OIDC; Nomad has private UI access only. See Temporal.
- MySQL / Percona Server — EXPERIMENTAL until tested against live backends. Percona Server for MySQL 8.4 and 8.0, single-member or 3/5-member single-primary Group Replication, ProxySQL routing, native users/grants, TLS, XtraBackup full/incremental backups and GTID/binlog PITR. See MySQL.
- Percona XtraDB Cluster (PXC) — EXPERIMENTAL until tested against live backends. A separate
pxcprovider for 3/5-member Galera clusters with one ProxySQL-routed writer, separate backup provenance, SST/recovery safety checks and explicit quorum recovery. It is not an in-place switch from the MySQL provider. See PXC operations. - Neo4j Community — EXPERIMENTAL until tested against live backends. Native Neo4j 2026.10.0 (also 2026.09.0) with Java 21, one persistent member, native administrative users, Bolt TLS, and offline system/graph dumps and restores. Community limits mean no clustering, granular RBAC or PITR; backups interrupt service. See Neo4j.
- Weaviate — EXPERIMENTAL until tested against live backends. Official 1.40.1 and 1.39.12 native runtimes, single or 3–9-member clusters with three fixed Raft voters, native access management, REST/gRPC TLS and full S3 recovery. Internal transports require isolated private networking; scale-out does not rebalance existing shards. Only community features are enabled. The upstream executable is mixed-license and is not redistributed as a Karakoram release asset. See Weaviate.
- Keycloak — EXPERIMENTAL until tested against live backends. Official Keycloak 26.8.1 with OpenJDK 21, one runtime, owned PostgreSQL persistence, managed bootstrap administrator recovery and optional backend-managed public HTTPS. Maintenance causes downtime; Nomad supports private access, not managed public HTTPS. See Keycloak.
Except for Temporal's Dokploy/Nomad-only runtime, the new providers have native runtime paths for Zerops, Dokploy and Nomad. Feature availability is capability-gated: public HTTPS needs a supported engine/backend combination with managed HTTPS; horizontal scaling gates replicated stateless services; public TCP filtering on Dokploy/Nomad requires external enforcement and the documented all-address acknowledgement. See engine/backend availability.
PostgreSQL platform
- Extension profiles: select compatible PostGIS, pgvector and Apache-only TimescaleDB 2.31.0 workloads on PostgreSQL majors 16–18. Profiles are additive, bootstrap through curated SQL with durable receipts, travel with backup provenance and gate restore/upgrade compatibility. Catalog metadata reports availability rather than advertising missing artifacts. See profiles.
- Owned service bindings: a source PostgreSQL cluster supplies a dedicated
kb_login role and owned databases to OpenBao storage, Temporal or Keycloak. The shared service manages sealed credentials, rotation, readiness, source-incarnation checks, dependency guards and release retention. Binding databases are created closed, revoke PUBLICCONNECT/TEMPORARY, grant access only to the owning role (explicit grants such as Data MCP readers are kept) and disconnect sessions that no longer have access; reconcile restores the restriction if PUBLIC access reappears. See bindings. - Distributed Citus — EXPERIMENTAL until tested against live backends. One coordinator and 1–8 worker groups, each with one or three data members, matching locked Citus 14.2.0 artifacts for PostgreSQL 16–18 and coordinator-facing ProxySQL routing. Applications explicitly create distributed/reference tables; ordinary tables remain local. No in-place conversion to/from distributed topology or distributed major upgrade. See Citus.
Ops — verified offline
These features are verified offline through tests without a live backend; they do not establish backend recovery-time or availability guarantees.
- Maintenance windows: time-zone-aware cron windows gate new member disruptions while preserving health/quorum checks and recovery progress. Explicit overrides are operation/generation-scoped, not blanket permission to bypass safety. Healing, restore and other documented exemptions continue outside windows.
- Agent inventory and rollout: observe target/current/unknown agent builds and use
karactl clusters upgrade-agentsto roll the current executable without changing engine versions. Maintenance and provider safety gates still apply. - Member moves: explicit native replacement with placement constraints for supported MongoDB, RabbitMQ, Elasticsearch and PostgreSQL/Supabase members. Other new-engine moves are rejected before queueing. Zerops replacement is best-effort, not a host-separation guarantee.
- Control-plane backup and recovery: policy-driven S3 snapshots, run history, local
karakoram admin backup-stateexports and offlineadmin restore-statefrom local files or S3. Snapshots include coordinator records and sealed secrets, not engine volumes or the original master key. Restore checks integrity, schema compatibility and key fingerprint; fence the old coordinator before recovery. - Notifications: durable at-least-once webhook, Slack and TLS email delivery for operational events, with signed generic webhook payloads, retries and guarded outbound destinations. Duplicate delivery remains possible.
- Backup drills: manual or scheduled disposable restores in the source namespace, durable verification/cleanup reports, ownership-fenced cleanup and bounded concurrency. Drills consume real compute, storage and backup bandwidth; a successful report is not application correctness.
See operations, control-plane recovery, notifications and backup drills.
Documentation
The Pricing page is renamed Operating costs: Karakoram is free and open source, and the page estimates the infrastructure cost of running each engine on Zerops, Hetzner, AWS, DigitalOcean and OVHcloud, now including the new engines. The old site URL redirects to the new page.
Upgrade notes
Follow the v0.3.0 → v0.4.0 upgrade guide. Take a consistent pre-upgrade state backup and preserve the master key separately before deploying:
git fetch origin --tags && git checkout v0.4.0
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs- Automatic SQLite migrations 013–017: PostgreSQL bindings, operational cluster fields, notifications, control-plane backups and backup drills. Startup applies unrecorded migrations in order. Migrations are forward-only; downgrading the upgraded store is unsupported. Rollback requires the pre-upgrade state, original master key and matching old binary; it does not undo subsequent engine/infrastructure changes.
- Source builds require Go 1.26+.
go.moddeclaresgo 1.26.0without a separ...