Skip to content

kividb-operator v0.7.0

Choose a tag to compare

@github-actions github-actions released this 04 Nov 03:19
· 156 commits to master since this release
d2d8ea6

Sharded clusters (spec.shards). The default engine is kividb 1.2.0.

Read before upgrading: the agent's port (8081) is no longer on the
<cluster>-master and <cluster>-replicas Services, and the agent's API
needs a token. Anything that called an agent through those Services, other
than the operator's own backup Jobs, has to change. The pods roll once.
New backups go to <pathPrefix>/<namespace>.<cluster>/ instead of
<pathPrefix>/<cluster>/: widen any IAM or bucket policy that limits the
backup credentials to the old folder, and adjust scripts and lifecycle
rules that match the old keys (see Fixed).

Apply the new CRDs before helm upgrade: Helm does not upgrade them,
and the KividbCluster, KividbConfig, KividbDbOps and KividbSnapshot
CRDs changed. Until they are applied, the API server drops the new fields
(spec.shards among them) and the checks that read them never fire. The
CRDs are attached to the GitHub Release.

Read before upgrading: 0.7.0 targets kividb engine 1.2.0.

  • A cluster without spec.image gets an engine upgrade to the new
    default image, docker.io/kividb/kividb:v1.2.0, as a rolling restart:
    replicas first, masters last, each master only after an in-sync replica
    has taken over. Pin spec.image to stay on 1.1.x for now.
  • No going back to 1.1.x once a 1.2 pod has saved a snapshot that
    holds a time series with compaction rules: 1.1.x cannot read it. Take
    a backup before the upgrade if you may need to downgrade.
  • During the rollout, 1.1.x replicas of a 1.2 master log errors. A 1.2
    master sends a transaction, and a script that writes more than once, to
    its replicas between MULTI and EXEC; a 1.1.x replica still applies
    every write, but logs Replicated command error (non-fatal): ERR unknown command 'MULTI' (and 'EXEC') for each, until it is upgraded too.
  • The append-only file is on by default (appendonly yes,
    appendfsync everysec), for every cluster whose KividbConfig does not
    set appendonly itself. It comes with the rolling restart above. Writes
    go through the file, so expect fewer writes a second at high
    concurrency, and size the volumes for the log too. appendonly: "no" in
    a KividbConfig keeps the old behaviour, with a Persistence condition
    and a Warning Event that say what it risks: without it, a killed engine
    comes back with only its last shutdown or full-sync snapshot, often
    nothing (see Fixed).
  • A KividbConfig directive the engine refuses stops the server. 1.2
    accepts Redis's own settings with a warning and applies some it used to
    ignore, but a name that is neither KiviDB's nor Redis's, or a bad value,
    is still fatal. A rolling change stalls safely after the first pod; a new
    cluster or a cold restart has every pod down. See
    CONFIGURATION.md.
  • A KividbSnapshotConfig whose spec.schedule has a TZ= or
    CRON_TZ= prefix is now refused
    as InvalidSpec (#47): a CronJob
    does not accept one. Every cluster that references it holds back its
    apply steps until it is fixed, though failover still runs. Remove the
    prefix before upgrading.
  • Reshard only under light write load. Engine 1.2.0 can lose writes
    while slots move between shards under heavy writes
    (kividbio/kividb#331). That covers changing spec.shards (scaling in
    or out, which uses CLUSTER MIGRATION IMPORT) and redis-cli --cluster reshard. It was seen only under heavy CPU contention. Until engine
    1.2.1, change spec.shards while writes are light, or pause writes.
    Engine 1.2.1 fixes it: set spec.image to it once it is released.

Security

  • The agent's API needs a token. In 0.6.0 any client that could reach
    port 8081 of a kividb pod could promote it, repoint its replication, or
    run any command through /exec; and the port was on the master and
    replica Services, which can be LoadBalancers. Now every endpoint but
    /healthz, /readyz and /metrics requires Authorization: Bearer <token>, and an agent without a token refuses them. The operator keeps
    the token in a Secret in its namespace (created with a random value at
    its first start; --agent-token-secret, the chart's
    <release>-agent-token) and copies it into each cluster's auth Secret.
    The operator, the backup Jobs and the GUI send it; the GUI reads it from
    the Secret mounted as a file. A new token rolls the pods, as a new
    password does.
  • The agent port is on the headless Service only, not on the master
    and replica Services.

Added

  • spec.shards: sharded clusters in kividb's cluster mode, one
    StatefulSet per shard (<cluster>-<shard>), a PodDisruptionBudget per
    shard, and ClusterIP Services only: one per pod, one per shard,
    <cluster>-replicas, <cluster> for the whole cluster, and a headless
    Service per shard. spec.cluster sets the node timeout, full coverage,
    min-replicas-to-write and the bus secret (generated when unset). The
    operator owns every cluster-mode directive. A cluster is sharded or not
    from its creation; up to 24 nodes for now.
  • Sharded clusters form and look after themselves. The operator
    introduces the nodes (CLUSTER MEET), assigns the slots evenly across the
    shards once every pod is up, puts each shard's replicas behind its
    master, and has every node announce its pod's DNS name. Failover is the
    engine's: the operator reports it and moves the role labels, and never
    promotes in cluster mode. It re-introduces nodes after IP changes (a
    cold restart), and forgets the old ID of a pod that rejoined with a new
    one, never a node that owns slots. The agent gains /cluster/status,
    /meet, /addslots, /replicate, /forget and /announce, all behind
    the agent token.
  • Rolling restarts of sharded clusters, for a template change or a
    KividbDbOps restart: one pod at a time across the cluster, replicas
    first, and each shard's master only after its most caught-up replica
    has taken over with CLUSTER FAILOVER. The agent has a
    /cluster/failover endpoint, and reports whether a replica's link to
    its master is up; status.pods[].synced is set for sharded clusters.
  • Changing a sharded cluster's shard count. Edit spec.shards and
    the operator adds shards, or drains and removes them, moving only the
    slots it has to: one move at a time, each an atomic CLUSTER MIGRATION IMPORT on the destination master (no ASK for clients). Phase
    Rebalancing, with the plan, the move under way and the last error in
    status.cluster.rebalance. status.cluster.shards is the count the
    slots are spread over. Removed shards' PVCs are deleted once their
    slots have all moved out. The agent has
    /cluster/migration/import and /cluster/migration/status.
  • Backups of sharded clusters. A run backs up every shard at once,
    each from a pod with the configured role, under
    <namespace>.<cluster>/<run>/shard-<i>-<pod>.tar.gz. A manifest.json
    names each shard's object, source, master node ID and slots, and is
    written only when every shard succeeded. Retention counts complete runs
    and prunes whole runs. KividbSnapshot.status.shards[] lists the
    shards, and objectKey is the manifest. The agent has
    /backup/manifest, and /backup takes a run and shard. Restoring a
    sharded cluster is not supported yet.
  • Access to a sharded cluster from outside Kubernetes. Expose each pod
    with a Service of your own (a LoadBalancer, a NodePort) labelled
    kividb.io/announce: "true", and the pod's node announces that
    Service's address (its load-balancer address, or the
    kividb.io/announce-address annotation) instead of its DNS name, so
    cluster clients outside follow MOVED to an address they can reach.
    The operator only reads these Services. The ExternallyAccessible
    condition says whether every pod is exposed (PartiallyExposed when
    only some are) and what is wrong with a Service that cannot be used.
  • The console shows sharded clusters.
    • Shards: one card per shard, with its slots, master and replicas, and each replica's lag against its own shard's master.
    • Slot map: the slots coloured by shard; the ones moving are striped.
    • Rebalance: progress while the shard count changes.
    • Tiles: slots and cluster_state.
    • Actions: +/− Shard. Promote runs CLUSTER FAILOVER on the replica. Take snapshot starts a backup run of every shard, as a Job from the backup CronJob, so the GUI's ClusterRole gains create on jobs.
  • spec.cluster.replicaValidityFactor: cluster-replica-validity-factor,
    which engine 1.2 enforces: a replica whose link to its master has been
    down for longer than nodeTimeoutMs × replicaValidityFactor does not
    stand for election. Unset, the operator writes nothing and the engine's
    default, 10, applies (with the default 5 s node timeout, a replica cut
    off for more than about a minute does not take over); 0 lets every
    replica stand. Needs engine 1.2: older engines do not start with the
    directive. A KividbConfig may not set it.
  • Handovers that do not happen are reported. Engine 1.2 pauses a
    shard master's writes for up to 10 s during CLUSTER FAILOVER, and the
    replica gives the failover up after 5 s (Manual failover abandoned).
    The ShardSwitchover Event now marks only the first request of a
    handover; a Warning ShardSwitchoverAbandoned follows when the master
    has not moved 15 s after a request. The operator asks at most 5 times
    in a row (then again after 10 minutes), and the ShardSwitchover
    condition shows the shard, the pods and the attempt while a handover is
    pending (InProgress, Retrying, Stalled).
  • A KividbDbOps restart no longer waits forever for a master to hand
    over.
    After 5 minutes it fails, says where to look, and withdraws its
    step-down request. status.restart.handoverStartTime records when it
    started waiting.

Changed

  • Default engine docker.io/kividb/kividb:v1.2.0 (it was v1.1.3):
    in the controller, the CRD, the chart, the GUI, the samples and the docs.
    The GUI's create form still adds -tls, -lua or -full for a variant.
    See "Read before upgrading" above.
  • Default agent image docker.io/kividb/kividb-operator-agent:0.7.0.
    A cluster that leaves spec.agentImage unset rolls to it in the same
    rollout. Several fixes below need it.
  • The e2e suite tests kividb 1.2.0 by default.
  • CI runs the core e2e suites on a kind cluster for every pull request
    (#20, by nightmare-tech).
  • The docs say what engine 1.2 adds to a pod's stop and memory: a master
    with replicas waits up to shutdown-timeout (10 s) for them before its
    final save, inside terminationGracePeriodSeconds; and the 64 MB
    repl-backlog-size counts in maxmemory, so a maxmemory of tens of
    megabytes needs repl-backlog-size: "1mb".
  • The rendered kividb.conf is a Secret (<cluster>-config), not a
    ConfigMap, for every cluster: a sharded cluster's carries the bus secret
    and masterauth. The old ConfigMap is removed once no pod mounts it;
    the pods move to the Secret as they roll.
  • The operator's ClusterRole adds PodDisruptionBudgets.
  • The backup Job finds the pod to back up by asking every pod's agent for
    its role, through the headless Service's per-pod names, instead of going
    through the master or replica Service.
  • A KividbConfig that sets cluster-enabled to anything but no is
    refused (the cluster's Ready condition says why). Cluster mode would
    fight the operator's own failover; sharded clusters come as
    spec.shards.
  • In a sharded cluster, a KividbConfig may set the slot migration
    settings, cluster-slot-migration-handoff-max-lag-bytes and
    cluster-slot-migration-write-pause-timeout (applied at runtime since
    engine kividbio/kividb#111). Every other cluster-* directive is still
    the operator's.
  • The console's Promote of a shard replica reports a request, not a
    promotion: CLUSTER FAILOVER answering OK only means the failover
    started. The endpoint answers 202 {"requested": pod} and the page
    refreshes a few seconds later to show whether the role moved.
  • The docs say what engine 1.2 fixes: a failed import's keys are deleted
    at restart (#108), cluster-announce-client-port exists but is not used
    yet (#110), heartbeats keep one connection per peer though the 24-node
    cap stays (#112), replicas resync in part after a restart since 1.0.5
    and after a failover since 1.2 (#109), elections favour the most
    caught-up replica (#107), and certificates can be reloaded with CONFIG SET, not used yet. KIVIDB_ENGINE_ISSUES.md is marked frozen at 1.0.5,
    with a table of the later engine issues and the version that fixes each.
  • The e2e harness documents KIVIDB_IMAGE_BASE=ghcr.io/kividbio/kividb
    for engine release candidates, which are on GHCR only.

Fixed

  • A cluster re-created at once with a deleted one's name and
    bootstrapFromSnapshot came up on the deleted cluster's data, saying
    Completed / skipped
    (#43, new in this release). Right after saving
    BootstrapWaiting, the pass read the cluster back from a cache that
    had not seen it yet and updated the earlier StatefulSet; that moved
    its owner reference to the new cluster, so it was never
    garbage-collected. While the bootstrap runs, the cluster now never
    updates a StatefulSet it does not control by UID, and only one of its
    own with pods (above 0 replicas), and no restore Job of its own,
    counts as running. A <cluster>-bootstrap Job from the earlier
    cluster is waited for too, not taken for this cluster's restore, and a
    restore onto a pod-0 volume older than the cluster says so in
    status.bootstrap and with a BootstrapOverExistingVolume Warning.
    (#46, by SaiPisey2)
    A pod-0 PVC that is being deleted is waited for too: the restore Job
    starts once it is gone, on a new one. The wait shows in
    status.bootstrap.message ("waiting for PVC data--0 to finish
    deleting") and in a Normal BootstrapWaitingForVolume Event, not only in
    the operator's log (#55). A new condition, Bootstrapped, says where a
    bootstrap is: False with reason WaitingForEarlierCluster,
    WaitingForVolumeDeletion, Restoring, RestoreRetrying or
    RestoreFailed, and True once restored (or skipped).
  • status.bootstrap said Failed while the restore Job still had a
    retry left
    (#54, also in 0.6.x). A failed pod was taken for a failed
    Job, and the retry then went on to complete. The bootstrap now fails
    only once the Job's JobFailed condition is true (its backoffLimit
    used up or its deadline passed), with a BootstrapFailed Warning Event.
    Until then it stays InProgress, and the message says "restore Job
    attempt N failed, retrying" with the failed pod's reason, with a
    BootstrapRetrying Warning Event per attempt. The restore container's
    last log lines are its termination message, so the reason includes them.
  • A bootstrap onto a kept PVC that still held the engine's dump.kdb
    failed and destroyed that file
    (#44, also in 0.6.x). The restore Job
    runs as a different UID from the engine: it overwrote the file, then
    could not chmod it. It now writes each file to a temp file next to it
    and renames it in only once the whole archive has been read, so a
    failed restore leaves the old files whole. The volume needs room for
    the old and the restored files at once; a full one fails with an error
    that says so. A dump.kdb or appendonly.aof that the snapshot does
    not have is removed, so a log left from the earlier cluster is no
    longer replayed on top of the restored data.
    (#45, by SaiPisey2)
    The engine's other files from the earlier cluster are removed too, before
    the restored files go in: dump.kdb.gen, its save number, made pod-0
    refuse to start or skip the snapshot's log and lose its writes, and
    appendonly.aof.prev (with .prev.len and .rewrite.tmp), from a
    checkpoint that did not finish, was replayed on top of the restored dump.
    dump.kdb.repl and dump.kdb.clean go as well. A restore that fails
    before then still leaves them all in place, with the old dump and log.
  • A master that restarted with the append-only file on was failed over
    to a replica behind it
    (new in this release). A killed engine starts a
    new stream, which was taken as lost writes. On engine 1.2.0 or later
    with aof-ack-after-write yes (its default), which acknowledges a write
    only once it is in the file, a master holding at least as many keys as
    the replica now stays the master, and the replicas resync from it. Key
    counts miss a lost overwrite or delete; the replicas resync without it.
    Any other master is still failed over, as is one whose agent does not
    report the engine's version and that setting (the agent's /status now
    carries both): engine 1.1.x acknowledges a write before it is in the
    file, so a killed master can lack writes its replica holds.
    (#42, by SaiPisey2)
  • A KividbDbOps restart deleted a master in place when its replica was
    not in sync at that moment, and did not wait for the other pods or ops

    (also in 0.6.x), so two restarts of one cluster could take two pods
    down. A master with a replica now always hands over first, and each pod
    is restarted only while every other pod is Ready, in sync and not being
    deleted, as in a rolling update (a replica that is the only unhealthy
    pod goes first; a wait of more than 5 minutes for another pod, or 60 for
    a restarted one, fails the op, naming the pod). The ops of one cluster
    run one at a time, oldest first; the others stay Pending with a
    Queued condition.
    (#40, by SaiPisey2)
  • spec.bootstrapFromSnapshot added to a running cluster scaled it to 0
    and restored the snapshot over pod-0's data
    (also in 0.6.x). The CRD
    now refuses the field after creation, and a cluster whose StatefulSet
    already exists skips the bootstrap with a BootstrapSkipped Event. A
    StatefulSet and its pods left from an earlier cluster with the same name
    are waited for (BootstrapWaiting), not skipped and not touched;
    removing the field cancels the wait. Helm does not upgrade CRDs: until
    the new kividbclusters CRD is applied, the API server still accepts the
    field, and only the skip protects the cluster.
    (#37, by SaiPisey2)
  • FailoverBlocked held for one pass, then an empty replica was
    promoted
    (also in 0.6.x). The blocked pass emptied status.pods, so
    the next one no longer knew the master had held keys, and the old
    master later resynced from the empty replica. A failed pass now keeps
    the last status.pods (with no pod Ready), and a newly promoted
    master keeps its key count.
    (#35, by SaiPisey2)
  • A failed reconcile step turned off failover, with the cluster still
    Ready
    (also in 0.6.x). A refused object or a missing referenced
    KividbConfig, KividbAclConfig, KividbSnapshotConfig or Secret now
    holds back the writes after it, not failover, role labels or status:
    Ready is False (ReconcileError or InvalidSpec), in sharded
    clusters too, with a Warning Event of that reason (#28). The Event
    comes once per change of the error, not every pass, and not for a
    conflict, which the next pass retries. Port clashes (spec.port,
    tls.port, the agent's 8081, the exporter's 9121) and backup
    schedules a CronJob would refuse (not valid cron, or with a TZ= or
    CRON_TZ= prefix) are refused before anything is written. The
    referenced kinds are watched, so a fix takes effect at once.
    (#36 and #47, by SaiPisey2)
  • Removing a shard could stay in Rebalancing for good (new in this
    release). On a status conflict, a pass that had read the cluster a
    version behind wrote its status over the latest one, and could bring
    back a change that was already finished. The removed shard's
    StatefulSet was gone by then, so later passes waited for pods that never
    came back. Now a status that changed since the pass read it is not
    written over, a removed shard whose StatefulSet is gone is not waited
    for, and a shard made again with no slots, when spec.shards was
    raised back meanwhile, gets its master and its share.
    (#38, by SaiPisey2)
  • Backup retention deleted other clusters' backups (also in 0.6.x). A
    cluster pruned everything under <pathPrefix>/<cluster>/, which also
    held the backups of a same-named cluster in another namespace, or of a
    cluster nested under it by pathPrefix; two such clusters could even
    write the same key. Backups now go to
    <pathPrefix>/<namespace>.<cluster>/ (the agent gets POD_NAMESPACE),
    and retention deletes only this cluster's own archives or runs there.
    Older backups still restore but are no longer pruned (see
    BACKUP_RESTORE.md before deleting them), and a
    0.6.x agent, until upgraded, still prunes all of its
    <pathPrefix>/<cluster>/, which holds new keys only of a cluster whose
    pathPrefix is that folder or lies under it.
    (#41, by SaiPisey2)
  • A new agent token stalled the rollout of every cluster with 2+ pods
    (new in this release). The agent read AGENT_TOKEN once, at start, so
    the running pods answered the operator's new token with 401 and the
    rollout that was to give it to them waited for good. The agent now reads
    agent-token from the auth Secret it already mounts, on each request,
    falling back to AGENT_TOKEN; the pods take a new token once the kubelet
    refreshes the mount (a minute or two), so a rollout may wait that long.
    (#39, by SaiPisey2)

Found by a third end-to-end test on AKS (operator master 853cd01 on an
engine 1.2.0 candidate):

  • A force-killed shard master lost its whole shard's data (new in
    this release: sharded clusters). The engine never snapshots on a
    schedule, and the operator wrote no appendonly (for any cluster), so
    there was no persistence at all. The master, deleted with
    --grace-period=0 --force, was back in about a second -- inside the 5 s
    node timeout, so no failover -- with its node ID, its slots and 0 keys.
    Its replica refused to full-sync from a master with no keys, but asked
    again every 10 s, and once the master had taken a few writes it went
    through: "Dropped the 25939 local keys before loading 987". The cluster
    stayed Ready, with no Event; a second kill lost 53,271 more. With
    appendonly yes the same kill lost nothing. Now:
    • The append-only file is on by default, for sharded and replicated
      clusters alike: the operator writes appendonly yes and appendfsync everysec unless the KividbConfig sets appendonly (or aof)
      itself. appendonly: "no" opts out; the new Persistence condition
      is then False (AppendOnlyDisabled) and a Warning Event
      PersistenceDisabled is recorded when it turns so. See
      CONFIGURATION.md
      for the cost (write throughput, disk) and the trade-off.
    • A shard master that restarted with less than its replica holds is
      failed away from.
      When a shard master's engine started less than 2
      minutes ago and a replica of its shard holds more than twice its keys,
      the operator holds the master's writes (CLIENT PAUSE WRITE, which
      also keeps an empty master empty, so the replica's full-sync guard
      keeps refusing it) and has the replica take over with CLUSTER FAILOVER FORCE, then TAKEOVER (at once with a single shard). It
      asks a restarted master's agent before the pod is Ready. Events
      MasterRestartedEmpty, MasterRestartedEmptyResolved and, when the
      replica had resynced first or no replica held more, ShardDataLost;
      condition MasterRestartedEmpty (FailingOver, Stalled,
      FailedOver, DataLost) for 10 minutes. This is a safety net, not a
      fix: a client that kept the pod's address (a container restarted in
      place) writes to it at once, and the replica's first resync, about a
      second after the restart, then goes through before any controller can
      act. In a kind test without persistence, a forced delete lost 54 and
      65 writes instead of the shard (takeover 0.7-1.0 s after the engine
      started); an in-place kill still lost the shard (the replica resynced
      0.16-0.43 s after the engine started), reported as ShardDataLost. The agent's /cluster/status reports the engine's run_id,
      uptime, key count, aof_enabled and loading, and
      /cluster/failover takes {"mode": "force"|"takeover"}; a sharded
      cluster's status.pods[] record keys, runID and incarnation.
      Needs the 0.7.0 agent image.
  • Every planned handover of a shard master was also reported as a
    Warning ShardFailover
    (new in this release). The move was taken as
    planned only if the old master's pod still had its step-down annotation
    or deletion timestamp when the move was seen, and a deleted pod was by
    then gone or back without them. The operator now remembers the
    handovers it asked for (2 minutes), and the console's Promote notes the
    replica it promoted on the cluster (kividb.io/shard-promote) before
    asking for it: both are Normal ShardSwitchover.
  • A sharded password change recorded no PasswordChanged Event, and
    other Events went missing after a rollout. client-go drops an object's
    Events past 25, then lets one through every 5 minutes; one rollout of a
    sharded cluster used that up. The operator's limit is now 100, then one
    every 5 seconds.
  • Removing a shard made its StatefulSet again (new in this release).
    On 4 → 3 the operator deleted the removed shard's StatefulSet and
    volumes, and a pass that still read the change as under way (its status
    a moment behind) created the StatefulSet again: two new pods on two new
    disks, removed again 30 s later. A removed shard's StatefulSet (and
    disruption budget) is now only updated while it exists, never created,
    and it is deleted in the foreground, so it stays until its pods are
    gone.
  • A replicated cluster bootstrapped from a sharded cluster's backup
    ran its restore Job on the run's manifest.json ("gzip: invalid
    header"), failed, and kept saying "restore Job running". It is refused
    before any Job runs (BootstrapRefused), and a failed restore Job's
    message says so.

Found by a second end-to-end test on AKS (operator master on engine
v1.2.0-rc2):

  • A false MasterRestartedBehind failover right after a planned
    switchover lost 314 acknowledged writes
    (new in this release). The
    check compared the new master's offset with one recorded for it a moment
    earlier, readings taken at different times, and failed over from a
    master that had never restarted; the master it failed away from kept
    taking writes until its resync from the replica promoted in its place
    discarded them. Now the check needs evidence that the master's engine
    restarted (a new engine run_id, or, from an older agent, a new pod or
    a container restart), compares offsets only within the stream the pod
    reported writing as the master (status.pods[].replID, cleared when the
    role moves), and skips replicas already resynced from the restarted
    master. When it does fail over, the restarted master's writes are held
    (CLIENT PAUSE WRITE) until it follows the new master, as in a
    switchover. The agent reports the engine's run_id and, on a master,
    its master_replid; status.pods[] gains replID, runID and
    incarnation. Helm does not upgrade CRDs: apply the new
    kividbclusters CRD, or these fields are dropped and the check never
    fires.
  • A rollout waiting on a replica that was not in sync gave no sign
    (also in 0.6.0 for replicated clusters). The rollout rightly stops, in
    a sharded cluster cluster-wide, but the cluster went on reporting
    Running and Ready with no Event and no log line; a frozen replica
    was found only because nothing moved. A new condition,
    RolloutProgressing, says what the rollout is doing: ReplacingPod,
    or WaitingForReplicaSync / WaitingForPodReady naming each pod it
    waits for and why (link up or down, bytes behind the master, last I/O
    from it). After 2 minutes on the same pods it turns False, with a
    RolloutWaiting Warning Event and a log line, repeated every 10
    minutes while the wait lasts. The rollout still waits as before. The
    agent reports master_last_io_seconds_ago in its cluster status too.
  • GUI: a cluster's operations list was empty for a KividbDbOps
    created with kubectl.
    It was filtered by a kividb.io/cluster label
    that only the GUI's own DbOps carry; it now goes by spec.clusterRef.
  • GUI: promoting a shard's replica left no Event. The GUI asks the
    engine for CLUSTER FAILOVER itself, so the operator never saw it; the
    GUI now records a ShardSwitchover Event on the cluster, as the
    operator does for its own handovers (and as a replicated cluster's
    promote gets a Switchover Event). The GUI's ClusterRole gains
    create on events.

Found by an end-to-end test on AKS (replicated: 1 master + 2 replicas
under 800 writes/s; sharded: 3 × (1 + 2)), which checked every
acknowledged write afterwards.

  • A planned switchover lost the writes acknowledged during it (also in
    0.6.0 and earlier). A rollout, a KividbDbOps restart, a password or
    config change, or the GUI's promote moved the master role by relabelling
    the old master, promoting the replica and repointing the old master,
    with nothing to stop the old master acknowledging writes in between, nor
    any wait for the replica to have the writes it had already acknowledged;
    and the old master was deleted on the next pass. Each switchover in the
    test lost the writes of that second: 19, 25, 23 and 64. Now the old
    master's writes are held (CLIENT PAUSE ... WRITE) until the replica has
    applied everything up to the master's offset, at most 5 s, and only then
    is the replica promoted, the old master made its replica, the label
    moved and the pause lifted; if the replica does not catch up, the pause
    is lifted and the old master stays (SwitchoverAborted). The label goes
    onto the new master before it comes off the old one: the master Service
    was without an endpoint for about 1.5 s (connection refused), it no
    longer is. The old master's clients are disconnected, so they reconnect
    to the new master rather than get READONLY. The GUI's promote now asks
    the operator for this (the kividb.io/switchover-to annotation on the
    cluster) instead of promoting the pod itself.
  • A replica whose link to the master was down counted as in sync, and
    was promoted
    (also in 0.6.0). In a replicated cluster "in sync" looked
    only at the offset, allowing 8 MiB of lag -- minutes of writes at a
    modest rate -- so a restarted replica that could not authenticate to its
    master, its offset frozen, was taken as in sync, and a switchover
    promoted it 541 KB behind: 13,455 and 13,919 acknowledged writes lost.
    In sync now needs master_link_status:up, something heard from the
    master in the last 20 s, and at most 1 MiB of lag (the agent reports the
    link and master_last_io_seconds_ago); a switchover target must also be
    following the master with its link up, and catch up exactly.
  • Removing the password broke replication during the rollout (also in
    0.6.0): the restarted pods had no password while the old master still
    required one, so their replication links stayed down -- what made the
    13,455 writes above. And a new password was refused until the master
    was restarted
    (11 s of authentication errors in the test): the master
    kept the old one until it was replaced. Now the pods still on the
    previous password are given the new credential live (ACL SETUSER default #<hash>, or nopass for a removal) before any pod restarts,
    pods started during the rollout accept both, and the previous password is
    dropped, by an ACL reload on every pod, once every pod runs with the new
    one. See CONFIGURATION.md.
  • Deleting, draining or evicting the master pod lost writes and took the
    cluster down for about 14 s
    (also in 0.6.0). Nothing handed the role
    over: the pod came back as the master from its shutdown snapshot, and the
    replicas, which held every acknowledged write, were resynced from it (92
    lost). Now both containers have a preStop hook (the agent's /prestop)
    that holds a master with replicas attached -- for at most a third of the
    grace period, 5 to 30 s -- while the operator, seeing the pod's deletion,
    does the switchover above; the engine then stops as a replica. The same
    holds for a shard master, which hands over with CLUSTER FAILOVER. A
    replicated cluster now has a PodDisruptionBudget (maxUnavailable: 1),
    as each shard has.
  • A master that restarted behind its replicas had them resync from its
    older data
    (also in 0.6.0). When the master comes back with a lower
    replication offset than the operator last recorded for it while a
    replica that was in sync holds more, that replica is promoted instead,
    and the restarted pod becomes its replica (MasterRestartedBehind). The
    status keeps a pod's last offset and key count while it cannot be asked.
    An in-place container restart, which keeps the pod's address, is still
    resynced by the engine before the operator can act.
  • A hung engine was never restarted (also in 0.6.0). The kividb
    container's liveness probe was a TCP check, which a deadlocked engine
    passes: it accepts connections and answers nothing. It is now the
    agent's /livez, which passes on any reply to PING (LOADING and
    BUSY too) and fails after about 90 s of none, after the failover of a
    hung master; a startup probe gives an engine up to 30 minutes to load
    its data first. Needs the 0.7.0 agent image.
  • The operator forgot nodes of sharded pods that were only unresponsive.
    A pod whose agent did not answer lost its node ID from the status at
    once, and its node, marked failed by the others and owning no slots, was
    then taken for "a node that is none of the pods" and forgotten
    (CLUSTER FORGET) on every node; the pod could only rejoin once the
    forget ban expired. The status now keeps such a pod's node ID, a node is
    also recognised as a pod's by its address and announced name, and a node
    that is no pod's is forgotten only after 2 minutes. A node of an existing
    pod is forgotten only when that pod answers with a new node ID.
  • /readyz, /status and /metrics failed while a password was being
    removed
    on engine 1.2 (also in 0.6.0). The agent logged in with
    AUTH <password>, which engine 1.2 refuses while the default user has
    no password (kividbio/kividb#135), as when the password is gone from the
    server and a pod still has the old Secret. The agent now always sends
    AUTH default <password>, which every engine takes; it sets
    masteruser default with masterauth, so replicas log in to their
    master the same way; and the exporter gets REDIS_USER=default. The
    new exporter variable rolls the pods of clusters with a password.
  • The chart's ServiceMonitor scraped each pod twice, through the headless
    Service and through its master or replica Service. It selects the
    headless Service only (kividb.io/service: headless).