You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
KividbDbOps CRD with op: restart / method: InPlace rolling
restart (replicas first, then master). GUI can create/list these.
spec.bootstrapFromSnapshot on KividbCluster: seed a new
cluster's pod-0 PVC from a Succeeded KividbSnapshot before the
StatefulSet starts (supported restore path; see docs/BACKUP_RESTORE.md).
Agent POST /exec for authenticated RESP commands (used by the GUI
explorer).
Agent restore-from-s3 subcommand used by the bootstrap Job.
Management GUI: Basic auth (GUI_AUTH_* / chart gui.auth.existingSecret),
live status gauges, pod logs, RESP explorer, DbOps restart actions,
24h local metrics scraper (PVC-backed) with sparkline charts.
e2e suite 08-resp-acl-select.sh: RESP3 HELLO 3, ACL deny smoke, SELECT on replica, failover single-master re-point. Override KIVIDB_VERSION (e.g. v1.0.4-rc2) for pre-GA engine testing.
Changed
Default engine pin quay.io/kividbio/kividb:v1.0.5 (and docs/
samples). Default agent image …-agent:0.4.0. v1.0.5 fixes all eleven
engine issues recorded in docs/KIVIDB_ENGINE_ISSUES.md, among them
unauthenticated access as the default user and the data-loss paths
around failed resyncs and skipped shutdown snapshots.
The agent sets masterauth to the default user's password before
every REPLICAOF. kividb v1.0.5 refuses an unauthenticated replication
handshake, so without it replicas of a password-protected cluster never
sync. It is set at runtime, so images of older engines keep working.
GUI ClusterRole gains create on kividbdbops and get on pods/log
(still never Secrets to the browser).
Rolling updates are now done by the operator. The StatefulSet uses
the OnDelete update strategy and the operator replaces pods itself:
replicas first, the master last, one at a time, and only while every
pod is Ready and every replica has finished syncing. A pod that is
unready on an outdated template is replaced without waiting.
The master is never restarted in place. A rolling update or KividbDbOps restart first hands the master role to an in-sync replica
(event Switchover) and then replaces the old master. Restarting it in
place made every replica resync from whatever the master had last saved,
which on EKS lost 55 seconds of acknowledged writes during a config
change. Measured after the change: about one second of failed writes
and none lost.
Failover will not promote an empty replica when the master was last
seen holding data: kividb drops a replica's dataset when a full resync
fails part-way, which can empty every replica at once if the master is
crash-looping. The failover waits for the master instead (event FailoverBlocked); set the kividb.io/allow-empty-failover: "true"
annotation on the KividbCluster to fail over regardless. status.pods[].keys reports each pod's key count.
status.pods[].synced reports whether a replica has completed its full
sync from the current master. KividbDbOps restarts wait for it, and
failover prefers replicas that were in sync.
Fixed
Failover could hand the cluster back to the old master and lose
writes. The pod being failed away from kept its role=master label;
when it came back it rejoined the master Service, was picked as the
master again, and the pod that had been promoted was made its replica.
The old master is now relabeled before its replacement is promoted and
rejoins as a replica. status.phase also no longer sticks at FailingOver with lastFailoverTime advancing every reconcile.
A master that had been turned into a replica was never repaired
(cluster reported Running, every write failed with READONLY). The
operator now follows the pod it is replicating from if that is a healthy
master of the same cluster, and otherwise promotes it again.
bootstrapFromSnapshot produced an empty cluster. The restored
files were readable only by the restore Job's own user, so kividb could
not load them and started empty while the bootstrap reported success.
KividbDbOps restart deleted every pod at once. A pod counted as
restarted while the deleted one was still terminating. Each pod must now
be replaced, Ready and back in the cluster before the next is deleted.
Backups failed with archive/tar: write too long whenever kividb
appended to appendonly.aof during the upload.
ACL changes were never applied to running pods. The operator now
has each pod run ACL LOAD once the updated file has reached it.
KividbConfig changes were never applied to running pods. They now
roll the pods, as does a change of the default user's password.
GUI: write endpoints (RESP explorer, restart, scale, promote, snapshot,
delete) were open when no auth Secret was configured, and accepted
cross-site requests. Promote sent REPLICAOF to backup Job pods, failed
halfway and could leave the cluster without a master.
The operator crash-looped if the KividbDbOps CRD was not installed,
which is the case after a helm upgrade from 0.3.0.
An unparseable spec.storage.size made the reconciler panic in a loop;
it is now rejected by the CRD schema and reported on the cluster's Ready condition.
The snapshot-restore pod was listed in status.pods as a cluster
member, and pod-0's volume was the only one deleted with the cluster.
A rolling update required the pod being replaced to be in sync as well
as every other pod. Upgrading the engine from v1.0.4 to v1.0.5 stalled on
the last old pod: once the master role was on a v1.0.5 pod, the v1.0.4
pod could not authenticate to it and never became "synced".
A crash-looping pod blocked every rollout, including the template
change meant to fix it, and rollouts moved on to the master while the
replaced replica was still resyncing. See "Changed" below.
The GUI pod stayed Pending on clusters without a default StorageClass
(EKS): its metrics PVC was on by default and could not be provisioned. gui.metrics.persistence.enabled now defaults to false.
Upgrading the operator while pods still ran kividb v1.0.3 never started
the rollout, because that engine does not report a replica's master.
The agent reported every replica's master port as 0, which made the
operator re-send REPLICAOF to every replica on every reconcile.
Upgrade notes
Apply the CRDs before upgrading the chart
(kubectl apply -f charts/kividb-operator/crds/): KividbDbOps is new
and KividbCluster gained validation. Helm does not do this for you.
Every existing cluster's pods are rolled once after the operator
is upgraded, one at a time: the pod template gains the agent's ACL
mount and the kividb.io/config-hash / kividb.io/auth-generation
annotations.
The GUI refuses all writes until gui.auth.existingSecret is set.
Clusters without spec.image move to kividb v1.0.5 in that same
rolling restart (from v1.0.3 if they were created by 0.3.0, or v1.0.4).
Pin spec.image first to upgrade the engine separately. v1.0.5 changes
behaviour that worked before:
The default user's password is now enforced. Clients that never
sent AUTH (which worked, wrongly, on older engines) now get NOAUTH. Check client configuration before upgrading.
Unknown or malformed directives in a KividbConfig are fatal: the
pod will not start. Older engines ignored them silently.
maxmemory is enforced, with noeviction by default: writes over
the limit are refused.
Clusters pinned to an engine older than v1.0.5 keep the old behaviour,
including unauthenticated access as the default user; restrict
network access to their Services.
Notes
Known engine issue in v1.0.5: a replica whose CPU limit is changed in
place (for example by the Vertical Pod Autoscaler) panics on its next
full resync and stops replicating until restarted. The operator itself
never resizes pods in place. See docs/KIVIDB_ENGINE_ISSUES.md, issue 12.
Out of scope: ReducedImpact restart, in-place restore of a live
cluster.