Repository navigation
kividb-operator v0.7.0
Sharded clusters (spec.shards). The default engine is kividb 1.2.0.
Read before upgrading: the agent's port (8081) is no longer on the
<cluster>-master and <cluster>-replicas Services, and the agent's API
needs a token. Anything that called an agent through those Services, other
than the operator's own backup Jobs, has to change. The pods roll once.
New backups go to <pathPrefix>/<namespace>.<cluster>/ instead of
<pathPrefix>/<cluster>/: widen any IAM or bucket policy that limits the
backup credentials to the old folder, and adjust scripts and lifecycle
rules that match the old keys (see Fixed).
Apply the new CRDs before helm upgrade: Helm does not upgrade them,
and the KividbCluster, KividbConfig, KividbDbOps and KividbSnapshot
CRDs changed. Until they are applied, the API server drops the new fields
(spec.shards among them) and the checks that read them never fire. The
CRDs are attached to the GitHub Release.
Read before upgrading: 0.7.0 targets kividb engine 1.2.0.
- A cluster without
spec.imagegets an engine upgrade to the new
default image,docker.io/kividb/kividb:v1.2.0, as a rolling restart:
replicas first, masters last, each master only after an in-sync replica
has taken over. Pinspec.imageto stay on 1.1.x for now. - No going back to 1.1.x once a 1.2 pod has saved a snapshot that
holds a time series with compaction rules: 1.1.x cannot read it. Take
a backup before the upgrade if you may need to downgrade. - During the rollout, 1.1.x replicas of a 1.2 master log errors. A 1.2
master sends a transaction, and a script that writes more than once, to
its replicas betweenMULTIandEXEC; a 1.1.x replica still applies
every write, but logsReplicated command error (non-fatal): ERR unknown command 'MULTI'(and'EXEC') for each, until it is upgraded too. - The append-only file is on by default (
appendonly yes,
appendfsync everysec), for every cluster whoseKividbConfigdoes not
setappendonlyitself. It comes with the rolling restart above. Writes
go through the file, so expect fewer writes a second at high
concurrency, and size the volumes for the log too.appendonly: "no"in
aKividbConfigkeeps the old behaviour, with aPersistencecondition
and a Warning Event that say what it risks: without it, a killed engine
comes back with only its last shutdown or full-sync snapshot, often
nothing (see Fixed). - A
KividbConfigdirective the engine refuses stops the server. 1.2
accepts Redis's own settings with a warning and applies some it used to
ignore, but a name that is neither KiviDB's nor Redis's, or a bad value,
is still fatal. A rolling change stalls safely after the first pod; a new
cluster or a cold restart has every pod down. See
CONFIGURATION.md. - A
KividbSnapshotConfigwhosespec.schedulehas aTZ=or
CRON_TZ=prefix is now refused asInvalidSpec(#47): a CronJob
does not accept one. Every cluster that references it holds back its
apply steps until it is fixed, though failover still runs. Remove the
prefix before upgrading. - Reshard only under light write load. Engine 1.2.0 can lose writes
while slots move between shards under heavy writes
(kividbio/kividb#331). That covers changingspec.shards(scaling in
or out, which usesCLUSTER MIGRATION IMPORT) andredis-cli --cluster reshard. It was seen only under heavy CPU contention. Until engine
1.2.1, changespec.shardswhile writes are light, or pause writes.
Engine 1.2.1 fixes it: setspec.imageto it once it is released.
Security
- The agent's API needs a token. In 0.6.0 any client that could reach
port 8081 of a kividb pod could promote it, repoint its replication, or
run any command through/exec; and the port was on the master and
replica Services, which can be LoadBalancers. Now every endpoint but
/healthz,/readyzand/metricsrequiresAuthorization: Bearer <token>, and an agent without a token refuses them. The operator keeps
the token in a Secret in its namespace (created with a random value at
its first start;--agent-token-secret, the chart's
<release>-agent-token) and copies it into each cluster's auth Secret.
The operator, the backup Jobs and the GUI send it; the GUI reads it from
the Secret mounted as a file. A new token rolls the pods, as a new
password does. - The agent port is on the headless Service only, not on the master
and replica Services.
Added
spec.shards: sharded clusters in kividb's cluster mode, one
StatefulSet per shard (<cluster>-<shard>), a PodDisruptionBudget per
shard, and ClusterIP Services only: one per pod, one per shard,
<cluster>-replicas,<cluster>for the whole cluster, and a headless
Service per shard.spec.clustersets the node timeout, full coverage,
min-replicas-to-writeand the bus secret (generated when unset). The
operator owns every cluster-mode directive. A cluster is sharded or not
from its creation; up to 24 nodes for now.- Sharded clusters form and look after themselves. The operator
introduces the nodes (CLUSTER MEET), assigns the slots evenly across the
shards once every pod is up, puts each shard's replicas behind its
master, and has every node announce its pod's DNS name. Failover is the
engine's: the operator reports it and moves the role labels, and never
promotes in cluster mode. It re-introduces nodes after IP changes (a
cold restart), and forgets the old ID of a pod that rejoined with a new
one, never a node that owns slots. The agent gains/cluster/status,
/meet,/addslots,/replicate,/forgetand/announce, all behind
the agent token. - Rolling restarts of sharded clusters, for a template change or a
KividbDbOpsrestart: one pod at a time across the cluster, replicas
first, and each shard's master only after its most caught-up replica
has taken over withCLUSTER FAILOVER. The agent has a
/cluster/failoverendpoint, and reports whether a replica's link to
its master is up;status.pods[].syncedis set for sharded clusters. - Changing a sharded cluster's shard count. Edit
spec.shardsand
the operator adds shards, or drains and removes them, moving only the
slots it has to: one move at a time, each an atomicCLUSTER MIGRATION IMPORTon the destination master (noASKfor clients). Phase
Rebalancing, with the plan, the move under way and the last error in
status.cluster.rebalance.status.cluster.shardsis the count the
slots are spread over. Removed shards' PVCs are deleted once their
slots have all moved out. The agent has
/cluster/migration/importand/cluster/migration/status. - Backups of sharded clusters. A run backs up every shard at once,
each from a pod with the configured role, under
<namespace>.<cluster>/<run>/shard-<i>-<pod>.tar.gz. Amanifest.json
names each shard's object, source, master node ID and slots, and is
written only when every shard succeeded. Retention counts complete runs
and prunes whole runs.KividbSnapshot.status.shards[]lists the
shards, andobjectKeyis the manifest. The agent has
/backup/manifest, and/backuptakes a run and shard. Restoring a
sharded cluster is not supported yet. - Access to a sharded cluster from outside Kubernetes. Expose each pod
with a Service of your own (a LoadBalancer, a NodePort) labelled
kividb.io/announce: "true", and the pod's node announces that
Service's address (its load-balancer address, or the
kividb.io/announce-addressannotation) instead of its DNS name, so
cluster clients outside followMOVEDto an address they can reach.
The operator only reads these Services. TheExternallyAccessible
condition says whether every pod is exposed (PartiallyExposedwhen
only some are) and what is wrong with a Service that cannot be used. - The console shows sharded clusters.
- Shards: one card per shard, with its slots, master and replicas, and each replica's lag against its own shard's master.
- Slot map: the slots coloured by shard; the ones moving are striped.
- Rebalance: progress while the shard count changes.
- Tiles: slots and
cluster_state. - Actions: +/− Shard. Promote runs
CLUSTER FAILOVERon the replica. Take snapshot starts a backup run of every shard, as a Job from the backup CronJob, so the GUI's ClusterRole gainscreateonjobs.
spec.cluster.replicaValidityFactor:cluster-replica-validity-factor,
which engine 1.2 enforces: a replica whose link to its master has been
down for longer thannodeTimeoutMs × replicaValidityFactordoes not
stand for election. Unset, the operator writes nothing and the engine's
default, 10, applies (with the default 5 s node timeout, a replica cut
off for more than about a minute does not take over);0lets every
replica stand. Needs engine 1.2: older engines do not start with the
directive. AKividbConfigmay not set it.- Handovers that do not happen are reported. Engine 1.2 pauses a
shard master's writes for up to 10 s duringCLUSTER FAILOVER, and the
replica gives the failover up after 5 s (Manual failover abandoned).
TheShardSwitchoverEvent now marks only the first request of a
handover; aWarningShardSwitchoverAbandonedfollows when the master
has not moved 15 s after a request. The operator asks at most 5 times
in a row (then again after 10 minutes), and theShardSwitchover
condition shows the shard, the pods and the attempt while a handover is
pending (InProgress,Retrying,Stalled). - A
KividbDbOpsrestart no longer waits forever for a master to hand
over. After 5 minutes it fails, says where to look, and withdraws its
step-down request.status.restart.handoverStartTimerecords when it
started waiting.
Changed
- Default engine
docker.io/kividb/kividb:v1.2.0(it wasv1.1.3):
in the controller, the CRD, the chart, the GUI, the samples and the docs.
The GUI's create form still adds-tls,-luaor-fullfor a variant.
See "Read before upgrading" above. - Default agent image
docker.io/kividb/kividb-operator-agent:0.7.0.
A cluster that leavesspec.agentImageunset rolls to it in the same
rollout. Several fixes below need it. - The e2e suite tests kividb 1.2.0 by default.
- CI runs the core e2e suites on a kind cluster for every pull request
(#20, by nightmare-tech). - The docs say what engine 1.2 adds to a pod's stop and memory: a master
with replicas waits up toshutdown-timeout(10 s) for them before its
final save, insideterminationGracePeriodSeconds; and the 64 MB
repl-backlog-sizecounts inmaxmemory, so amaxmemoryof tens of
megabytes needsrepl-backlog-size: "1mb". - The rendered
kividb.confis a Secret (<cluster>-config), not a
ConfigMap, for every cluster: a sharded cluster's carries the bus secret
andmasterauth. The old ConfigMap is removed once no pod mounts it;
the pods move to the Secret as they roll. - The operator's ClusterRole adds PodDisruptionBudgets.
- The backup Job finds the pod to back up by asking every pod's agent for
its role, through the headless Service's per-pod names, instead of going
through the master or replica Service. - A
KividbConfigthat setscluster-enabledto anything butnois
refused (the cluster'sReadycondition says why). Cluster mode would
fight the operator's own failover; sharded clusters come as
spec.shards. - In a sharded cluster, a
KividbConfigmay set the slot migration
settings,cluster-slot-migration-handoff-max-lag-bytesand
cluster-slot-migration-write-pause-timeout(applied at runtime since
engine kividbio/kividb#111). Every othercluster-*directive is still
the operator's. - The console's Promote of a shard replica reports a request, not a
promotion:CLUSTER FAILOVERansweringOKonly means the failover
started. The endpoint answers202 {"requested": pod}and the page
refreshes a few seconds later to show whether the role moved. - The docs say what engine 1.2 fixes: a failed import's keys are deleted
at restart (#108),cluster-announce-client-portexists but is not used
yet (#110), heartbeats keep one connection per peer though the 24-node
cap stays (#112), replicas resync in part after a restart since 1.0.5
and after a failover since 1.2 (#109), elections favour the most
caught-up replica (#107), and certificates can be reloaded withCONFIG SET, not used yet.KIVIDB_ENGINE_ISSUES.mdis marked frozen at 1.0.5,
with a table of the later engine issues and the version that fixes each. - The e2e harness documents
KIVIDB_IMAGE_BASE=ghcr.io/kividbio/kividb
for engine release candidates, which are on GHCR only.
Fixed
- A cluster re-created at once with a deleted one's name and
bootstrapFromSnapshotcame up on the deleted cluster's data, saying
Completed/ skipped (#43, new in this release). Right after saving
BootstrapWaiting, the pass read the cluster back from a cache that
had not seen it yet and updated the earlier StatefulSet; that moved
its owner reference to the new cluster, so it was never
garbage-collected. While the bootstrap runs, the cluster now never
updates a StatefulSet it does not control by UID, and only one of its
own with pods (above 0 replicas), and no restore Job of its own,
counts as running. A<cluster>-bootstrapJob from the earlier
cluster is waited for too, not taken for this cluster's restore, and a
restore onto a pod-0 volume older than the cluster says so in
status.bootstrapand with aBootstrapOverExistingVolumeWarning.
(#46, by SaiPisey2)
A pod-0 PVC that is being deleted is waited for too: the restore Job
starts once it is gone, on a new one. The wait shows in
status.bootstrap.message("waiting for PVC data--0 to finish
deleting") and in a NormalBootstrapWaitingForVolumeEvent, not only in
the operator's log (#55). A new condition,Bootstrapped, says where a
bootstrap is:Falsewith reasonWaitingForEarlierCluster,
WaitingForVolumeDeletion,Restoring,RestoreRetryingor
RestoreFailed, andTrueonce restored (or skipped). status.bootstrapsaidFailedwhile the restore Job still had a
retry left (#54, also in 0.6.x). A failed pod was taken for a failed
Job, and the retry then went on to complete. The bootstrap now fails
only once the Job'sJobFailedcondition is true (itsbackoffLimit
used up or its deadline passed), with aBootstrapFailedWarning Event.
Until then it staysInProgress, and the message says "restore Job
attempt N failed, retrying" with the failed pod's reason, with a
BootstrapRetryingWarning Event per attempt. The restore container's
last log lines are its termination message, so the reason includes them.- A bootstrap onto a kept PVC that still held the engine's
dump.kdb
failed and destroyed that file (#44, also in 0.6.x). The restore Job
runs as a different UID from the engine: it overwrote the file, then
could not chmod it. It now writes each file to a temp file next to it
and renames it in only once the whole archive has been read, so a
failed restore leaves the old files whole. The volume needs room for
the old and the restored files at once; a full one fails with an error
that says so. Adump.kdborappendonly.aofthat the snapshot does
not have is removed, so a log left from the earlier cluster is no
longer replayed on top of the restored data.
(#45, by SaiPisey2)
The engine's other files from the earlier cluster are removed too, before
the restored files go in:dump.kdb.gen, its save number, made pod-0
refuse to start or skip the snapshot's log and lose its writes, and
appendonly.aof.prev(with.prev.lenand.rewrite.tmp), from a
checkpoint that did not finish, was replayed on top of the restored dump.
dump.kdb.replanddump.kdb.cleango as well. A restore that fails
before then still leaves them all in place, with the old dump and log. - A master that restarted with the append-only file on was failed over
to a replica behind it (new in this release). A killed engine starts a
new stream, which was taken as lost writes. On engine 1.2.0 or later
withaof-ack-after-write yes(its default), which acknowledges a write
only once it is in the file, a master holding at least as many keys as
the replica now stays the master, and the replicas resync from it. Key
counts miss a lost overwrite or delete; the replicas resync without it.
Any other master is still failed over, as is one whose agent does not
report the engine's version and that setting (the agent's/statusnow
carries both): engine 1.1.x acknowledges a write before it is in the
file, so a killed master can lack writes its replica holds.
(#42, by SaiPisey2) - A
KividbDbOpsrestart deleted a master in place when its replica was
not in sync at that moment, and did not wait for the other pods or ops
(also in 0.6.x), so two restarts of one cluster could take two pods
down. A master with a replica now always hands over first, and each pod
is restarted only while every other pod is Ready, in sync and not being
deleted, as in a rolling update (a replica that is the only unhealthy
pod goes first; a wait of more than 5 minutes for another pod, or 60 for
a restarted one, fails the op, naming the pod). The ops of one cluster
run one at a time, oldest first; the others stayPendingwith a
Queuedcondition.
(#40, by SaiPisey2) spec.bootstrapFromSnapshotadded to a running cluster scaled it to 0
and restored the snapshot over pod-0's data (also in 0.6.x). The CRD
now refuses the field after creation, and a cluster whose StatefulSet
already exists skips the bootstrap with aBootstrapSkippedEvent. A
StatefulSet and its pods left from an earlier cluster with the same name
are waited for (BootstrapWaiting), not skipped and not touched;
removing the field cancels the wait. Helm does not upgrade CRDs: until
the newkividbclustersCRD is applied, the API server still accepts the
field, and only the skip protects the cluster.
(#37, by SaiPisey2)FailoverBlockedheld for one pass, then an empty replica was
promoted (also in 0.6.x). The blocked pass emptiedstatus.pods, so
the next one no longer knew the master had held keys, and the old
master later resynced from the empty replica. A failed pass now keeps
the laststatus.pods(with no podReady), and a newly promoted
master keeps its key count.
(#35, by SaiPisey2)- A failed reconcile step turned off failover, with the cluster still
Ready(also in 0.6.x). A refused object or a missing referenced
KividbConfig, KividbAclConfig, KividbSnapshotConfig or Secret now
holds back the writes after it, not failover, role labels or status:
ReadyisFalse(ReconcileErrororInvalidSpec), in sharded
clusters too, with a Warning Event of that reason (#28). The Event
comes once per change of the error, not every pass, and not for a
conflict, which the next pass retries. Port clashes (spec.port,
tls.port, the agent's 8081, the exporter's 9121) and backup
schedules a CronJob would refuse (not valid cron, or with aTZ=or
CRON_TZ=prefix) are refused before anything is written. The
referenced kinds are watched, so a fix takes effect at once.
(#36 and #47, by SaiPisey2) - Removing a shard could stay in
Rebalancingfor good (new in this
release). On a status conflict, a pass that had read the cluster a
version behind wrote its status over the latest one, and could bring
back a change that was already finished. The removed shard's
StatefulSet was gone by then, so later passes waited for pods that never
came back. Now a status that changed since the pass read it is not
written over, a removed shard whose StatefulSet is gone is not waited
for, and a shard made again with no slots, whenspec.shardswas
raised back meanwhile, gets its master and its share.
(#38, by SaiPisey2) - Backup retention deleted other clusters' backups (also in 0.6.x). A
cluster pruned everything under<pathPrefix>/<cluster>/, which also
held the backups of a same-named cluster in another namespace, or of a
cluster nested under it bypathPrefix; two such clusters could even
write the same key. Backups now go to
<pathPrefix>/<namespace>.<cluster>/(the agent getsPOD_NAMESPACE),
and retention deletes only this cluster's own archives or runs there.
Older backups still restore but are no longer pruned (see
BACKUP_RESTORE.md before deleting them), and a
0.6.x agent, until upgraded, still prunes all of its
<pathPrefix>/<cluster>/, which holds new keys only of a cluster whose
pathPrefixis that folder or lies under it.
(#41, by SaiPisey2) - A new agent token stalled the rollout of every cluster with 2+ pods
(new in this release). The agent readAGENT_TOKENonce, at start, so
the running pods answered the operator's new token with 401 and the
rollout that was to give it to them waited for good. The agent now reads
agent-tokenfrom the auth Secret it already mounts, on each request,
falling back toAGENT_TOKEN; the pods take a new token once the kubelet
refreshes the mount (a minute or two), so a rollout may wait that long.
(#39, by SaiPisey2)
Found by a third end-to-end test on AKS (operator master 853cd01 on an
engine 1.2.0 candidate):
- A force-killed shard master lost its whole shard's data (new in
this release: sharded clusters). The engine never snapshots on a
schedule, and the operator wrote noappendonly(for any cluster), so
there was no persistence at all. The master, deleted with
--grace-period=0 --force, was back in about a second -- inside the 5 s
node timeout, so no failover -- with its node ID, its slots and 0 keys.
Its replica refused to full-sync from a master with no keys, but asked
again every 10 s, and once the master had taken a few writes it went
through: "Dropped the 25939 local keys before loading 987". The cluster
stayedReady, with no Event; a second kill lost 53,271 more. With
appendonly yesthe same kill lost nothing. Now:- The append-only file is on by default, for sharded and replicated
clusters alike: the operator writesappendonly yesandappendfsync everysecunless theKividbConfigsetsappendonly(oraof)
itself.appendonly: "no"opts out; the newPersistencecondition
is thenFalse(AppendOnlyDisabled) and aWarningEvent
PersistenceDisabledis recorded when it turns so. See
CONFIGURATION.md
for the cost (write throughput, disk) and the trade-off. - A shard master that restarted with less than its replica holds is
failed away from. When a shard master's engine started less than 2
minutes ago and a replica of its shard holds more than twice its keys,
the operator holds the master's writes (CLIENT PAUSE WRITE, which
also keeps an empty master empty, so the replica's full-sync guard
keeps refusing it) and has the replica take over withCLUSTER FAILOVER FORCE, thenTAKEOVER(at once with a single shard). It
asks a restarted master's agent before the pod is Ready. Events
MasterRestartedEmpty,MasterRestartedEmptyResolvedand, when the
replica had resynced first or no replica held more,ShardDataLost;
conditionMasterRestartedEmpty(FailingOver,Stalled,
FailedOver,DataLost) for 10 minutes. This is a safety net, not a
fix: a client that kept the pod's address (a container restarted in
place) writes to it at once, and the replica's first resync, about a
second after the restart, then goes through before any controller can
act. In a kind test without persistence, a forced delete lost 54 and
65 writes instead of the shard (takeover 0.7-1.0 s after the engine
started); an in-place kill still lost the shard (the replica resynced
0.16-0.43 s after the engine started), reported asShardDataLost. The agent's/cluster/statusreports the engine'srun_id,
uptime, key count,aof_enabledandloading, and
/cluster/failovertakes{"mode": "force"|"takeover"}; a sharded
cluster'sstatus.pods[]recordkeys,runIDandincarnation.
Needs the 0.7.0 agent image.
- The append-only file is on by default, for sharded and replicated
- Every planned handover of a shard master was also reported as a
WarningShardFailover(new in this release). The move was taken as
planned only if the old master's pod still had its step-down annotation
or deletion timestamp when the move was seen, and a deleted pod was by
then gone or back without them. The operator now remembers the
handovers it asked for (2 minutes), and the console's Promote notes the
replica it promoted on the cluster (kividb.io/shard-promote) before
asking for it: both areNormalShardSwitchover. - A sharded password change recorded no
PasswordChangedEvent, and
other Events went missing after a rollout. client-go drops an object's
Events past 25, then lets one through every 5 minutes; one rollout of a
sharded cluster used that up. The operator's limit is now 100, then one
every 5 seconds. - Removing a shard made its StatefulSet again (new in this release).
On 4 → 3 the operator deleted the removed shard's StatefulSet and
volumes, and a pass that still read the change as under way (its status
a moment behind) created the StatefulSet again: two new pods on two new
disks, removed again 30 s later. A removed shard's StatefulSet (and
disruption budget) is now only updated while it exists, never created,
and it is deleted in the foreground, so it stays until its pods are
gone. - A replicated cluster bootstrapped from a sharded cluster's backup
ran its restore Job on the run'smanifest.json("gzip: invalid
header"), failed, and kept saying "restore Job running". It is refused
before any Job runs (BootstrapRefused), and a failed restore Job's
message says so.
Found by a second end-to-end test on AKS (operator master on engine
v1.2.0-rc2):
- A false
MasterRestartedBehindfailover right after a planned
switchover lost 314 acknowledged writes (new in this release). The
check compared the new master's offset with one recorded for it a moment
earlier, readings taken at different times, and failed over from a
master that had never restarted; the master it failed away from kept
taking writes until its resync from the replica promoted in its place
discarded them. Now the check needs evidence that the master's engine
restarted (a new enginerun_id, or, from an older agent, a new pod or
a container restart), compares offsets only within the stream the pod
reported writing as the master (status.pods[].replID, cleared when the
role moves), and skips replicas already resynced from the restarted
master. When it does fail over, the restarted master's writes are held
(CLIENT PAUSE WRITE) until it follows the new master, as in a
switchover. The agent reports the engine'srun_idand, on a master,
itsmaster_replid;status.pods[]gainsreplID,runIDand
incarnation. Helm does not upgrade CRDs: apply the new
kividbclustersCRD, or these fields are dropped and the check never
fires. - A rollout waiting on a replica that was not in sync gave no sign
(also in 0.6.0 for replicated clusters). The rollout rightly stops, in
a sharded cluster cluster-wide, but the cluster went on reporting
RunningandReadywith no Event and no log line; a frozen replica
was found only because nothing moved. A new condition,
RolloutProgressing, says what the rollout is doing:ReplacingPod,
orWaitingForReplicaSync/WaitingForPodReadynaming each pod it
waits for and why (link up or down, bytes behind the master, last I/O
from it). After 2 minutes on the same pods it turnsFalse, with a
RolloutWaitingWarning Event and a log line, repeated every 10
minutes while the wait lasts. The rollout still waits as before. The
agent reportsmaster_last_io_seconds_agoin its cluster status too. - GUI: a cluster's operations list was empty for a
KividbDbOps
created with kubectl. It was filtered by akividb.io/clusterlabel
that only the GUI's own DbOps carry; it now goes byspec.clusterRef. - GUI: promoting a shard's replica left no Event. The GUI asks the
engine forCLUSTER FAILOVERitself, so the operator never saw it; the
GUI now records aShardSwitchoverEvent on the cluster, as the
operator does for its own handovers (and as a replicated cluster's
promote gets aSwitchoverEvent). The GUI's ClusterRole gains
createonevents.
Found by an end-to-end test on AKS (replicated: 1 master + 2 replicas
under 800 writes/s; sharded: 3 × (1 + 2)), which checked every
acknowledged write afterwards.
- A planned switchover lost the writes acknowledged during it (also in
0.6.0 and earlier). A rollout, aKividbDbOpsrestart, a password or
config change, or the GUI's promote moved the master role by relabelling
the old master, promoting the replica and repointing the old master,
with nothing to stop the old master acknowledging writes in between, nor
any wait for the replica to have the writes it had already acknowledged;
and the old master was deleted on the next pass. Each switchover in the
test lost the writes of that second: 19, 25, 23 and 64. Now the old
master's writes are held (CLIENT PAUSE ... WRITE) until the replica has
applied everything up to the master's offset, at most 5 s, and only then
is the replica promoted, the old master made its replica, the label
moved and the pause lifted; if the replica does not catch up, the pause
is lifted and the old master stays (SwitchoverAborted). The label goes
onto the new master before it comes off the old one: the master Service
was without an endpoint for about 1.5 s (connection refused), it no
longer is. The old master's clients are disconnected, so they reconnect
to the new master rather than getREADONLY. The GUI's promote now asks
the operator for this (thekividb.io/switchover-toannotation on the
cluster) instead of promoting the pod itself. - A replica whose link to the master was down counted as in sync, and
was promoted (also in 0.6.0). In a replicated cluster "in sync" looked
only at the offset, allowing 8 MiB of lag -- minutes of writes at a
modest rate -- so a restarted replica that could not authenticate to its
master, its offset frozen, was taken as in sync, and a switchover
promoted it 541 KB behind: 13,455 and 13,919 acknowledged writes lost.
In sync now needsmaster_link_status:up, something heard from the
master in the last 20 s, and at most 1 MiB of lag (the agent reports the
link andmaster_last_io_seconds_ago); a switchover target must also be
following the master with its link up, and catch up exactly. - Removing the password broke replication during the rollout (also in
0.6.0): the restarted pods had no password while the old master still
required one, so their replication links stayed down -- what made the
13,455 writes above. And a new password was refused until the master
was restarted (11 s of authentication errors in the test): the master
kept the old one until it was replaced. Now the pods still on the
previous password are given the new credential live (ACL SETUSER default #<hash>, ornopassfor a removal) before any pod restarts,
pods started during the rollout accept both, and the previous password is
dropped, by an ACL reload on every pod, once every pod runs with the new
one. See CONFIGURATION.md. - Deleting, draining or evicting the master pod lost writes and took the
cluster down for about 14 s (also in 0.6.0). Nothing handed the role
over: the pod came back as the master from its shutdown snapshot, and the
replicas, which held every acknowledged write, were resynced from it (92
lost). Now both containers have apreStophook (the agent's/prestop)
that holds a master with replicas attached -- for at most a third of the
grace period, 5 to 30 s -- while the operator, seeing the pod's deletion,
does the switchover above; the engine then stops as a replica. The same
holds for a shard master, which hands over withCLUSTER FAILOVER. A
replicated cluster now has a PodDisruptionBudget (maxUnavailable: 1),
as each shard has. - A master that restarted behind its replicas had them resync from its
older data (also in 0.6.0). When the master comes back with a lower
replication offset than the operator last recorded for it while a
replica that was in sync holds more, that replica is promoted instead,
and the restarted pod becomes its replica (MasterRestartedBehind). The
status keeps a pod's last offset and key count while it cannot be asked.
An in-place container restart, which keeps the pod's address, is still
resynced by the engine before the operator can act. - A hung engine was never restarted (also in 0.6.0). The kividb
container's liveness probe was a TCP check, which a deadlocked engine
passes: it accepts connections and answers nothing. It is now the
agent's/livez, which passes on any reply toPING(LOADINGand
BUSYtoo) and fails after about 90 s of none, after the failover of a
hung master; a startup probe gives an engine up to 30 minutes to load
its data first. Needs the 0.7.0 agent image. - The operator forgot nodes of sharded pods that were only unresponsive.
A pod whose agent did not answer lost its node ID from the status at
once, and its node, marked failed by the others and owning no slots, was
then taken for "a node that is none of the pods" and forgotten
(CLUSTER FORGET) on every node; the pod could only rejoin once the
forget ban expired. The status now keeps such a pod's node ID, a node is
also recognised as a pod's by its address and announced name, and a node
that is no pod's is forgotten only after 2 minutes. A node of an existing
pod is forgotten only when that pod answers with a new node ID. /readyz,/statusand/metricsfailed while a password was being
removed on engine 1.2 (also in 0.6.0). The agent logged in with
AUTH <password>, which engine 1.2 refuses while thedefaultuser has
no password (kividbio/kividb#135), as when the password is gone from the
server and a pod still has the old Secret. The agent now always sends
AUTH default <password>, which every engine takes; it sets
masteruser defaultwithmasterauth, so replicas log in to their
master the same way; and the exporter getsREDIS_USER=default. The
new exporter variable rolls the pods of clusters with a password.- The chart's ServiceMonitor scraped each pod twice, through the headless
Service and through its master or replica Service. It selects the
headless Service only (kividb.io/service: headless).