Skip to content

Upgrading k4all (v2.x)

Giorgio Pillon edited this page Nov 18, 2026 · 3 revisions

Upgrading k4all (v2.x)

This page is for the bootc line (v2.x). On v1 (Fedora CoreOS) see Kubernetes updates.

k4all 2.x nodes are bootc images. An upgrade switches the node to a newer image and reboots it. Kubernetes, CRI-O and the kubelet live inside the image, so an upgrade can also move Kubernetes. Doing that safely needs kubeadm, and k4all-upgrade takes care of it.

This page is the short version. The full reference is docs/upgrades.md in the repo.

Don't run bootc upgrade by hand: it skips the kubeadm step. For the same reason k4all masks bootc's own bootc-fetch-apply-updates.timer. Don't unmask it.

Step 0: the bridge (v2.0 – v2.2.3 only)

Nodes on v2.0 – v2.2.3 don't have k4all-upgrade yet. Move them to the v2.2.4 bridge first:

curl -fsSL https://github.com/gpillon/k4all/releases/download/v2.2.4/k4all-bridge.sh | sudo bash -s -- --now
  • HA cluster with kube-vip? Read Known issue: kube-vip on HA clusters (v2.0–v2.2.3) first.
  • One node at a time: the bootstrap node, then the other control planes, then the workers. Each one Ready again before the next.
  • The bridge reboots the node without a drain. On a cluster with more nodes, drain the node first and uncordon it once it is Ready on v2.2.4:
    kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
    # on the node: run the bridge command above
    kubectl uncordon <node>
  • Without --now the bridge is staged and applied at the next reboot.
  • The script checks the node first. If a check fails, it stops and changes nothing.
  • After the bridge the node has the default policy (mode: auto). On a cluster with more nodes, change it right away: see Policy and modes.

The release ladder

Every minor release (v2.Y.0) is a mandatory stop: no skipping. Patch releases inside the same line can be skipped, but before a node leaves a line it always takes that line's latest patch. Kubernetes never moves more than one minor per release.

Release What changes
v2.2.4 The bridge: brings k4all-upgrade, masks the bootc timer, fixes scripts. Kubernetes 1.35, CRI-O 1.32 and Fedora 43 stay the same.
v2.3.0 CRI-O 1.35, to match Kubernetes 1.35, plus the latest 1.35 patch. Image names without a registry (nginx:latest) now resolve on docker.io only.
v2.4.0 First sync of the cluster components: k4all-operator v0.5.0 (with ClusterUpgrade), the Headlamp plugin, every component pinned and updated. Traefik replaces ingress-nginx: see Moving from ingress-nginx to Traefik (v2.4.0). kube-vip now belongs to the host.
v2.5.0 Fedora 45.
v2.6.0 Kubernetes 1.36 (and CRI-O 1.36). ingress-nginx leaves the release manifest.
v2.6.1 Patch: k4all-operator v0.5.1, which also removes the empty ingress-nginx namespace that v2.4.0 left behind. Fixes how k4all-upgrade matches a staged image when the upgrade graph pins images by digest.

So a node on v2.2.4 goes v2.3.0 → v2.4.0 → v2.5.0 → v2.6.0 → v2.6.1: five hops, one reboot each. v2.6.1 is a patch: nodes in mode: auto take it by themselves in their window.

The notes of each release are on the Releases page.

The k4all-upgrade command

On v2.2.4 or newer:

sudo k4all-upgrade status      # where am I, what is the policy, any HOLD
sudo k4all-upgrade plan        # which releases would be visited
sudo k4all-upgrade run         # do it now (one reboot per release)

plan and run go to the latest release. Add --to <release> to stop earlier, for example sudo k4all-upgrade run --to v2.4.0.

After each reboot the upgrade goes on by itself (k4all-upgrade-resume.service).

What one hop does, in short:

  1. Preflight: enough disk space, the apiserver and all nodes Ready, the right order on multi-node clusters.
  2. Backup: k4all-backup.sh (etcd snapshot, PKI, configs).
  3. Stage the new image.
  4. kubeadm, only when Kubernetes changes: the control plane is upgraded before the reboot.
  5. Drain and reboot.
  6. Post-boot checks: new kubelet, node Ready, apiserver answers.
  7. Sync, only for releases that update the cluster components (like v2.4.0).
  8. Commit, then the next hop starts.

Policy and modes

The policy is /etc/k4all-upgrade.yaml. When that file is missing, this default applies:

mode: auto          # auto | manual | managed
autoAllow: patch    # patch | minor
window:
  days: [Sat]
  start: "02:30"    # node local time
  durationMinutes: 120
  • mode: auto: a timer checks every 15 minutes. Inside the window it runs what autoAllow allows. With patch (the default) patch releases install by themselves, and minor releases wait for sudo k4all-upgrade run. With autoAllow: minor everything runs by itself.
  • mode: manual: nothing happens until you run sudo k4all-upgrade run.
  • mode: managed: the timer does nothing. Hops start only when the k4all-operator asks for one (a ClusterUpgrade, see below), or with sudo k4all-upgrade run.

In every mode the node also follows the k4all-operator's requests.

Every release after v2.2.4 so far is a minor release, so with the default policy they all wait for you.

On a cluster with more nodes, mode: auto upgrades each node on its own, in no order. Right after the bridge, on each node:

printf 'mode: manual\n' | sudo tee /etc/k4all-upgrade.yaml     # hops from the command line
printf 'mode: managed\n' | sudo tee /etc/k4all-upgrade.yaml    # from v2.4.0, ClusterUpgrade

From Headlamp, with a ClusterUpgrade (v2.4.0 and later)

From v2.4.0 the k4all-operator can upgrade the whole cluster: you set the target release on the ClusterUpgrade named k4all. The easy way is the Headlamp plugin: k4all > Upgrades.

  1. Choose a release. The page lists the releases the cluster can move to, with their notes and the hops to get there.
  2. Review. The path hop by hop, the node order, how nodes are drained, how many reboots and how much downtime to expect. You can change the drain timeout (30 minutes by default). Tick the box and start.
  3. Follow it. Each node's phase, what k4all-upgrade is doing on it, and how long it took.

The operator moves one node at a time: the bootstrap node, then the other control planes, then the workers. It cordons and drains each node, asks it for the hop, and uncordons it once it is back. Set mode: managed on these nodes, so they never start a hop on their own.

Pause stops before the next node. Resume goes on. Stop clears the target: the node in progress finishes its hop, and the others stay where they are.

Clusters before v2.4.0 can't do this yet. Bring each node to v2.4.0 from the command line first (sudo k4all-upgrade run --to v2.4.0, bootstrap node first). Nodes on v2.0 – v2.2.3 take the bridge before that.

What to expect

  • One reboot per hop, on every node.
  • Single node: it is cordoned, not drained. While it reboots, your workloads, the Kubernetes API and the dashboard are down.
  • More nodes: one node at a time, drained first. See How to Scale your Cluster.
  • Two control planes: the API goes down at every control plane reboot. Use one control plane or three.
  • v2.4.0: ingress-nginx is removed and Traefik installed. The routes are down until Traefik runs: plan for some minutes.
  • Rollback: if the node isn't healthy after the reboot, k4all-upgrade goes back to the previous image once (bootc rollback) and stops in HOLD. A failure in the component sync also stops in HOLD, but never rolls the OS back. bootc rollback only changes the node's image: it can't undo CRD upgrades or the first component sync. That's one reason those come in minor releases.

When something goes wrong (HOLD)

sudo k4all-upgrade status shows the reason. Fix the cause, then:

sudo k4all-upgrade clear-hold
sudo k4all-upgrade resume

To give up the remaining hops instead: sudo k4all-upgrade clear-hold --abandon.

A wrong order or a missing drain is not a HOLD: the hop just doesn't start yet, and status says why (Waiting:).

More in docs/upgrades.md.

Clone this wiki locally