Skip to content

kividb-operator v0.7.1

Latest

Choose a tag to compare

@github-actions github-actions released this 12 Nov 11:44
· 116 commits to master since this release
23bd1bf

A patch release of 0.7.0. The default engine is kividb 1.2.1. Upgrade
now if you run engine 1.2.1, or will: since 1.2.1 the engine refuses a
write pause it cannot take at once, and operator 0.7.0 then switched over
without the pause, so a write the old master acknowledged could be lost
(#81). 0.7.1 also fences a force-deleted master's last writes before a
failover (#53), keeps a restore from mixing the snapshot with the earlier
cluster's append-only file (#58), fixes the images' CVEs (#76), adds an
OLM bundle for OperatorHub.io, and fixes three install paths.

Read before upgrading: every cluster restarts, and a master with no
replica restarts in place.
The default agent image moves to 0.7.1, and
the default engine to 1.2.1, so every KividbCluster that leaves
spec.agentImage or spec.image unset gets a rolling restart when the
operator is upgraded. A master with replicas hands its role to an in-sync
replica first, as in every rollout. A master with none is deleted and
restarted on its volume, with no handover:

  • replicas: 0 (one pod): the cluster serves nothing until the pod is
    back.
  • A sharded cluster with replicas: 0: each shard's master restarts
    in turn. Its slots are unavailable while it does, and if it is down for
    longer than the node timeout (5 s by default) with requireFullCoverage
    on (the default), the whole cluster refuses commands until it is back.
  • Data: with the append-only file on (the default since 0.7.0) the
    master reloads everything it acknowledged. With appendonly: "no" it
    comes back with its shutdown snapshot only, and with nothing saved since
    its last snapshot if that save does not finish within
    terminationGracePeriodSeconds.

Before upgrading, give such clusters or shards at least one replica and
let it sync, or accept the restart: with the append-only file on, or with
the shutdown snapshot only. (Turning the append-only file on is a config
change, so it restarts such a master in place too.) To put the restart off, pin
both spec.image: docker.io/kividb/kividb:v1.2.0 and
spec.agentImage: docker.io/kividb/kividb-operator-agent:0.7.0 first:
nothing else in the pod template changes in 0.7.1. The 0.7.0 agent works
with the 0.7.1 operator, without the agent's fixes: it cuts every pause at
5 s, which aborts a switchover that needs longer (#81), and its restore
lacks the #58 fix. Unpin once the cluster has a replica or its
append-only file.

Engine 1.2.1 has upgrade notes of its own. The rollout restarts
replicas before their master, as 1.2.1 requires (a 1.2.0 replica cannot
full-sync from a 1.2.1 master that holds a key 1.2.0 cannot read). Read
the engine's "Before you upgrade"
first: among others, ACL categories are Redis's from the moment a pod
runs 1.2.1, GEORADIUS is refused on replicas, and there is no going
back to 1.2.0 once 1.2.1 has saved such a key. Do not change
spec.shards during the rollout: a slot move from a 1.2.0 node loses
the moved time series' compaction rules.

The CRD change is additive, and upgrading from 0.7.0 is safe. The
kividbclusters status gains pods[].podIP and fencedMaster (name,
podIP, runID), and new conditions (Switchover); nothing is removed
or renamed, and the other CRDs do not change. Apply the new
kividbclusters CRD before helm upgrade:
Helm does not upgrade it.
Until it is applied, the API server drops the new fields and failovers do
not hold the old master's writes (FailoverUnfenced Warning). The CRDs
are attached to the GitHub Release. Coming from 0.6.x, read 0.7.0's
"Read before upgrading" first.

Security

  • The images had golang.org/x/net v0.60.0 (CVE-2026-78661, high;
    CVE-2026-97045, medium) and an out-of-date Debian 12 base
    (#76).
    golang.org/x/net is now v0.62.0, the other golang.org/x modules are
    current, the binaries are built with Go 1.26.11, and the images run on
    gcr.io/distroless/static-debian13:nonroot (still no shell, still UID
    65532). CI and the release workflow scan every image with Trivy and fail
    on a fixable HIGH or CRITICAL.

Changed

  • Default engine docker.io/kividb/kividb:v1.2.1 (it was v1.2.0):
    in the controller, the CRD, the chart, the GUI, the samples and the docs.
    See "Read before upgrading" above.
  • Default agent image docker.io/kividb/kividb-operator-agent:0.7.1.
    A cluster that leaves spec.agentImage unset rolls to it in the same
    rollout. The switchover and restore fixes below need it.
  • The e2e suite tests kividb 1.2.1 by default.

Added

  • An OLM bundle for OperatorHub.io. make bundle VERSION=… IMG=… builds
    the bundle from config/manifests (operator-sdk v1.36.0, kustomize v5.4.3),
    and make scorecard runs the scorecard tests. The bundle validates for
    Kubernetes 1.27, 1.32 and 1.33 and installs through OLM 0.28.0 in
    AllNamespaces mode.

Fixed

  • The plain-YAML install and kubectl apply -k config/default made the
    manager Deployment with different selectors (#86).
    Applying one over the
    other failed with spec.selector: ... field is immutable.
    config/manager/manager.yaml now selects on app.kubernetes.io/managed-by
    too, as config/default does; what config/default and the OLM bundle
    render is unchanged. A Deployment made before this from config/manager
    alone, by the plain-YAML install or kubectl apply -k config/manager, has
    the old selector: delete it once (kubectl delete deploy -n kividb-operator-system kividb-operator-manager) before applying the new
    manifests.
    (#88, by SaiPisey2)

  • docs/INSTALL.md's install without kustomize exited 1 (#67).
    kubectl apply -f config/rbac/ applied the RBAC and then failed on
    config/rbac/kustomization.yaml, and on a new cluster the namespace did
    not exist. The steps now create the namespace and name the RBAC files.
    (#85, by SaiPisey2)

  • A switchover no longer hands over without holding writes when the pause
    could not take hold (#81).
    Since kividb 1.2.1 (kividbio/kividb#629),
    CLIENT PAUSE ... WRITE waits up to 5 s for the writes already in progress
    and refuses, pausing nothing, if one has not finished (a big FLUSHALL, a
    DEL of a huge key, a long script). The switchover took that refusal, and
    a pause that got no answer within the agent's 5 s, for an engine before
    kividb 1.1: it promoted the target without waiting for it to catch up,
    and raised SwitchoverWithoutPause telling the user to upgrade an engine
    that was already new. A write the old master acknowledged then could be
    lost. Now such a switchover is called off (SwitchoverAborted) and tried
    again; a pause that got no answer is lifted, in case it landed. Only an
    engine that answers that it does not know the command is switched over
    without the pause: every kividb has it, so any other refusal (LOADING,
    BUSY) aborts too. The pause call is given 7 s, past the engine's wait
    (or --agent-timeout, if longer), and the agent is asked to wait 6 s:
    its exec request takes a timeoutMs (capped at 15 s), which the
    operator sends for the pause only. An agent before this release ignores
    it and cuts the pause at 5 s, which now aborts the switchover rather than
    dropping the pause. The engine counts the pause from the request, so it
    now lasts the call, the catch-up and a margin (17 s), and the old master
    is a replica before it runs out. Aborted switchovers back off: 30 s,
    doubling with each abort in a row up to 10 min, and 2 s between the tries
    for a master pod being deleted; a Switchover condition (False,
    Aborted) says why and when the next try is, until one goes through.
    The failover fence (#53) waits for a pause the same way once the old
    master has answered, and tries again after a refusal: every try is given
    the whole call, and new ones start for 3 s after the first, so the fence
    takes at most two pause calls and that window against a live master that
    cannot pause (17 s with the default --agent-timeout, more with a longer
    one). A pause that got no answer may have landed, so that
    master is recorded in status.fencedMaster and demoted after the
    promotion like a fenced one. The Switchover condition and the count of
    aborts go away once no switchover is wanted any more, or the master is
    another pod.
    (#82, by lofoneh)

  • kubectl apply -k config/default installed no KividbDbOps CRD.
    config/crd/kustomization.yaml did not list it, so the documented install
    could not run KividbDbOps operations. After applying it again, restart
    the manager (kubectl rollout restart): it looks for the CRD once, at
    start-up.

  • The full KividbConfig sample was rejected by its own CRD. It set
    tls: {enabled: false}, and the CRD requires certSecretRef whenever tls
    is present. The sample now leaves tls out.

  • A restore over a kept volume could mix the snapshot with the earlier
    cluster's append-only file
    (#58). The restore Job removes the earlier
    cluster's dump.kdb.gen, and without it engine 1.2.0 and 1.2.1 both
    replay appendonly.aof over dump.kdb. The Job removed the earlier
    cluster's appendonly.aof only after it had renamed the restored
    dump.kdb in, so a Job stopped between the two (killed, evicted, or a
    removal that failed) left the restored dump with the earlier cluster's
    log. A cluster started on that volume, once spec.bootstrapFromSnapshot
    was removed after a failed bootstrap, came up with the writes of both.
    Every file of the earlier cluster's engine now goes before a restored
    file takes its name: appendonly.aof and every appendonly.aof.*,
    every dump.kdb.* (its state and temp files), dump.rdb, and the
    superseded.* logs that engine 1.2.1 keeps aside. The engine never
    removes those: they held the volume's space, and every start named
    them with how to apply their writes. A Job stopped part-way now leaves
    what is left of the earlier cluster's files, or part of the snapshot,
    never both, and its retry ends with the snapshot's files only. The
    manual restore in docs/BACKUP_RESTORE.md removes the same files.

  • A force-deleted master's last writes were lost (#53). Its pod object
    is gone at once and the operator fails over within the second, but its
    container runs on until its replicas have left it, acknowledging the
    writes that come in on connections clients already had to it: 176 lost in
    the AKS test on engine 1.2.1, 689 to 954 per force-delete on kind. Before
    promoting a replica, the operator now holds the old master's writes
    (CLIENT PAUSE WRITE through its agent, for the pod's termination grace
    period -- 10 s at least -- and 5 s more) at the address and engine run
    status.pods[] last recorded for it (podIP, new, and runID), waits
    for the replica to catch up with it (5 s at most, less once the old
    master stops answering), records it in status.fencedMaster (new), and
    once the replica is promoted makes the old master its replica and
    disconnects its clients, as a switchover does. Should that fail, or the
    operator restart, every pass holds its writes again and retries, every
    2 s, until it is done or the engine there is gone (another engine
    answers, or none does twice in a row); the first failure raises a
    FencedMasterNotDemoted Warning. An address that does not answer is
    asked again for about 1 s (one that refuses the connection, once); a
    pause that goes unanswered is sent again for up to 3 s. A failover that
    goes on without the fence, or before the replica caught up, raises a
    FailoverUnfenced Warning. A failed promotion no longer lifts the old
    master's pause: it may have gone through. The agent's preStop hook looks at a stopping master every 10 ms
    instead of 250 ms.
    hack/e2e/10-force-delete-fence.sh force-deletes the master under load
    and counts the writes lost (skipped in CI: SKIP_FENCE=1).
    (#77, by lofoneh)