battery-operator v0.4.1: Pools deleted with battery's draining

Pools are now deleted with battery v0.4.0's own draining, and a claim no longer binds to a Pool that is being deleted.

Deleting a Pool

• The Pool Controller calls battery's DeletePool. battery drains the Pool's available, provisioning and quarantined MicroVMs itself, and removes the Pool.
• While a MicroVM of the Pool is leased, battery refuses. The Pool keeps its finalizer and shows Ready false with reason DeletionBlocked and the leased count (PO-043). Its spec is set to a drained spec, so battery does not replace MicroVMs meanwhile (PO-042). The Pool goes once the last claim is released.
• The Operator's own drain from v0.2, which claimed and released each MicroVM, is gone (PO-030 to PO-034 withdrawn, replaced by PO-042 to PO-044).

Claims on a deleting Pool

A claim whose Pool is being deleted does not call ClaimVM. It stays Pending with reason PoolDeleting and retries with backoff; once the Pool is gone it shows PoolNotFound (CL-063). A Bound claim is not affected. One race remains: a Pool deleted between the claim's read of it and ClaimVM can still give the claim a MicroVM, which then holds the deletion like any Bound claim.

Known issue

battery v0.4.0 still leaks a MicroVM whose create hook fails while flintlockd is down (liquidmetal-dev/battery#131). The fix is liquidmetal-dev/battery#132, not yet released.

Artifacts

All for linux/amd64 and linux/arm64:

ghcr.io/phoban01/battery-operator:v0.4.1@sha256:089b90e3ca1bf75610c0f71f11015bb36cd0c4b0addb1121970bbfc7d2a43dc3
ghcr.io/phoban01/battery-operator/exec-agent:v0.4.1@sha256:f103d75102fa909bdbd8f74d6a95d671ce0a123d5ba440776f5ad1cb83df0690
ghcr.io/phoban01/battery-operator/host-image:v0.4.1@sha256:4dbba809ef5e5610bf50100fc70edcea5bb37da36a1d14154bca50df660fd0be
ghcr.io/phoban01/battery-operator/guest-kernel:v0.4.1@sha256:0eb901c1f9ce9000f3c8d3e166aec8e06a64a8d233b140aa79062882cede4751
ghcr.io/phoban01/battery-operator/guest-rootfs:v0.4.1@sha256:74b6f247e8d27d7cbb34db8aaeb444d889f46a5fa555c459885bc5e5e96a1ba9

Manifests artifact:

ghcr.io/phoban01/battery-operator/manifests:v0.4.1@sha256:ec872c133c5ed02e924a30bd63ba39182695499e6d7cc4df0818c2c7ff5efd7a
flux pull artifact oci://ghcr.io/phoban01/battery-operator/manifests:v0.4.1 --output ./battery-operator

battery-operator v0.4.0: battery v0.4.0, idempotent claims and arm64 AMIs

battery-operator now runs battery v0.4.0, claims MicroVMs idempotently, and can publish arm64 AMIs.

battery v0.4.0

• The battery sidecar is poolmgrd v0.4.0, and the Operator's battery client follows v0.4.0's API.
• Every ClaimVM carries the claim's UID as its request id (CL-060). A retry after a lost answer, or after the Operator restarts before writing the claim's status, gets the same Lease back instead of a second MicroVM (CL-061).
• While the Lease's MicroVM is being deleted, battery answers ABORTED. The claim stays Pending with reason ClaimAborted and retries with backoff (CL-062).
• The claim model checks that a retried bind yields the same Lease, never a second MicroVM.

Upgrading from v0.3.x

battery migrates its database from v0.3.3's schema when it starts. A new e2e test starts the Operator with poolmgrd 0.3.3, binds a claim, upgrades the sidecar, and checks that the claim keeps its Lease, renews it, and that a new claim binds. No manual step is needed.

arm64 AMIs

hostimage/publish-ami.sh --arch amd64|arm64 (or make host-image-ami ARCH=arm64) makes an AMI of either architecture, named with it and tagged battery.liquidmetal-x.dev/architecture. bootc-image-builder runs on a machine of the target architecture. An arm64 disk made this way booted, joined a kubeadm cluster, and was reported ready by the Exec Agent. The AMI registration itself has not yet run against AWS. Graviton VM sizes have no nested virtualization, so an arm64 Host is a metal size such as c7gd.metal or c8gd.metal-24xl; config/capi shows both.

Known issue

battery v0.4.0 still leaks a MicroVM whose create hook fails while flintlockd is down (liquidmetal-dev/battery#131). The fix is liquidmetal-dev/battery#132, not yet released.

Artifacts

All for linux/amd64 and linux/arm64:

ghcr.io/phoban01/battery-operator:v0.4.0@sha256:799da2c1eff7d62a8bb6835104874cfb99759d43ad81d32680cf761529069319
ghcr.io/phoban01/battery-operator/exec-agent:v0.4.0@sha256:6383e4c29a7d14724efd473d4c76a4b8b3893efc5cc121e41dec236b2d6364e7
ghcr.io/phoban01/battery-operator/host-image:v0.4.0@sha256:507affac2a6191b6e9637aa49817614893f6cbe3511564c5a98045ef3aaf8cf7
ghcr.io/phoban01/battery-operator/guest-kernel:v0.4.0@sha256:2ace715defd6e7bf5c47164d19a16c6ab836ab49ab40537a48d04a816edd10bf
ghcr.io/phoban01/battery-operator/guest-rootfs:v0.4.0@sha256:f81ce8f9879a4c9ef4dd24f25a15417a278f0748decc73c598ec828bac1dffc2

Manifests artifact:

ghcr.io/phoban01/battery-operator/manifests:v0.4.0@sha256:5330107d0d46d9640da93eed92a6bd1189ca1f9ff1ec7f507cb5aedd062e2c77
flux pull artifact oci://ghcr.io/phoban01/battery-operator/manifests:v0.4.0 --output ./battery-operator

battery-operator v0.3.3: guest kernel and root filesystem images

battery-operator now publishes guest images: a kernel and a root filesystem that a Pool's MicroVMs can boot from.

• ghcr.io/phoban01/battery-operator/guest-kernel:v0.3.3 holds Firecracker's CI kernel 6.1.128 at boot/vmlinux: an ELF on x86_64 and an arm64 Image on aarch64.
• ghcr.io/phoban01/battery-operator/guest-rootfs:v0.3.3 holds Ubuntu 24.04 with systemd and flintlock's guest agent v0.4.0, enabled at boot. systemd-networkd leaves eth0 to flintlock's metadata service and configures eth1 and later interfaces by DHCP.
• Both are multi-architecture images for linux/amd64 and linux/arm64. Their base images are pinned by digest, every download is checked against a pinned checksum, and a check stage runs in every build (GI-001 to GI-032, docs/requirements/13-guest-image.md).
• The kernel has no virtio-pci, so it boots on Firecracker and not on Cloud Hypervisor (#202).
• The demo, the sample Pool and the Install page name these images.

ghcr.io/phoban01/battery-operator/guest-kernel:v0.3.3@sha256:bb802de5df51bbb06775e3d640152293bb275c55315adabfbdffa081c1d19285
ghcr.io/phoban01/battery-operator/guest-rootfs:v0.3.3@sha256:615ed92fbeb3bd57dc96373b7dc903a9495acab8be0790f8322fc1ab59a225d4

The Operator, the Exec Agent and the Host Image have not changed since v0.3.2.

Manifests artifact:

ghcr.io/phoban01/battery-operator/manifests:v0.3.3@sha256:e8507649cb875ecce374b50534ae901e173fe96eafb3f7235ee95df3c4bf9edc
flux pull artifact oci://ghcr.io/phoban01/battery-operator/manifests:v0.3.3 --output ./battery-operator

Images, for linux/amd64 and linux/arm64:

• ghcr.io/phoban01/battery-operator:v0.3.3
• ghcr.io/phoban01/battery-operator/exec-agent:v0.3.3
• ghcr.io/phoban01/battery-operator/host-image:v0.3.3
• ghcr.io/phoban01/battery-operator/guest-kernel:v0.3.3
• ghcr.io/phoban01/battery-operator/guest-rootfs:v0.3.3

battery-operator v0.3.2: the guest bridge is virbr-battery, so Calico's autodetection skips it

The Host Image's guest bridge is now virbr-battery, so Calico's IP autodetection skips it.

Calico's default autodetection method, first-found, takes the first interface whose name is not on its exclude list. The old name, flbr0, was not on it. So Calico could take the guest bridge's address as the Node's, and the CNI failed on the Node. Calico v3.31.2 and v3.32.2 both exclude ^virbr.*. The list is DEFAULT_INTERFACES_TO_EXCLUDE in node/pkg/lifecycle/startup/autodetection/autodetection_linux.go.

• The bridge follows libvirt's model (TAP devices for VMs, DHCP and DNS on the gateway, NAT out), and takes libvirt's prefix. libvirt names its own bridges virbr0, virbr1 and so on, so there is no clash.
• HI-083 requires a bridge name on Calico's exclude list, and the image's checks enforce it.
• The name is set in one place, /usr/libexec/battery/lib.sh. flintlockd's --bridge-name and the firewall rules take it from there.
• A Host booted from the arm64 image ran Calico v3.31.2 with first-found and no IP_AUTODETECTION_METHOD. calico-node took the Host's own address, and every smoke step passed: pod networking, a Pool, a claim, the guest network, guest isolation, a flintlockd restart and Pool deletion.
• The Hosts page applies Calico v3.31.2 and no longer suggests IP_AUTODETECTION_METHOD.

Upgrade: a Host that boots this image has virbr-battery and no flbr0. Rebooting into the new image ends the Host's MicroVMs, as any reboot does.

flintlock-runner's fleet scripts for the battery stack still name flbr0 (phoban01/flintlock-runner#124).

The Operator and the Exec Agent have not changed since v0.2.1.

Manifests artifact:

ghcr.io/phoban01/battery-operator/manifests:v0.3.2@sha256:dbd8584f2d071dc10a76aea68ffd6919953d3a1636cdeb837cedad7325f76c19
flux pull artifact oci://ghcr.io/phoban01/battery-operator/manifests:v0.3.2 --output ./battery-operator

Images, for linux/amd64 and linux/arm64:

• ghcr.io/phoban01/battery-operator:v0.3.2
• ghcr.io/phoban01/battery-operator/exec-agent:v0.3.2
• ghcr.io/phoban01/battery-operator/host-image:v0.3.2

battery-operator v0.3.1: links into /var where Kubernetes writes, and Calico on a Host

The Host Image now gives Kubernetes somewhere to write, and pod networking with Calico is proven on a Host.

• /opt, /usr/local and /usr/libexec/kubernetes are links into /var (HI-082): /var/opt, /var/usrlocal and /var/libexec/kubernetes. On bootc these paths are read-only, so charts that write under /opt (such as local-path-provisioner), DaemonSets that install into /usr/local/bin, and the kubelet's flex-volume and credential-provider plugins failed before. Each target carries the label SELinux gives the path that links to it. /usr/lib/modules stays read-only. See "Where Kubernetes writes" in hostimage/README.md.
• Calico v3.32.2 runs on a Host with SELinux enforcing and no changes to the image. A normal pod on a Host gets a pod IP, reaches pods and Services on other Nodes, and resolves cluster DNS. MicroVMs still reach the internet, and still cannot reach pod, Service or API server addresses.
• hack/kubeadm-host now runs Calico and checks pod networking and guest isolation in its smoke steps.
• The Hosts page covers Calico on AWS: CAPA's default security group admits Calico's BGP and IP-in-IP, and IP_AUTODETECTION_METHOD=kubernetes-internal-ip pins a Host's address, since a Host also has the guest bridge flbr0.

Not yet proven: Calico on a real AWS Host, and MicroVMs on Cloud Hypervisor (#202).

The Operator and the Exec Agent have not changed since v0.2.1. The Host Image keeps Kubernetes v1.36.3, containerd v2.2.9, flintlock v0.16.0, Firecracker v1.16.1 and Cloud Hypervisor v48.0.

Manifests artifact:

ghcr.io/phoban01/battery-operator/manifests:v0.3.1@sha256:440a7b08a05ebc234aa4d6db13078caba6dc4500484e60dec68943990631edb3
flux pull artifact oci://ghcr.io/phoban01/battery-operator/manifests:v0.3.1 --output ./battery-operator

Images, for linux/amd64 and linux/arm64:

• ghcr.io/phoban01/battery-operator:v0.3.1
• ghcr.io/phoban01/battery-operator/exec-agent:v0.3.1
• ghcr.io/phoban01/battery-operator/host-image:v0.3.1

battery-operator v0.3.0: Kubernetes v1.36, containerd v2 and flintlock v0.16 on the Host Image

The Host Image moves to Kubernetes v1.36 and containerd v2, with flintlock v0.16.0.

Component	v0.2.x	v0.3.0
Kubernetes (kubelet, kubeadm)	v1.35.8	v1.36.3
containerd	v1.7.22	v2.2.9
flintlock	v0.15.2	v0.16.0
Firecracker	v1.10.1	v1.16.1
Cloud Hypervisor	v41.0	v48.0

• containerd's configuration is now in v2's format (version = 3). The thin pool snapshotter for flintlock is unchanged.
• flintlockd and the kubelet's image pulls read registry configuration from /etc/containerd/certs.d (HI-081). The image ships none, so pulls work as before. hostimage/README.md says how to add a mirror, a CA or a credential.
• The example Host pool in config/capi uses Kubernetes v1.36.3. Pools that boot a v0.2.x AMI keep v1.35.8. With the AWS cloud controller manager, use v1.36.1 for v1.36.
• A Host booted from the arm64 image joined a v1.36.3 kubeadm cluster and ran a Pool, claims, the guest network and a flintlockd restart (hack/kubeadm-host).

Not proven yet: MicroVMs on Cloud Hypervisor (#202). Cloud Hypervisor starts, but the trial's guest kernel lacks virtio-pci.

The Operator and the Exec Agent have not changed since v0.2.1.

Manifests artifact:

ghcr.io/phoban01/battery-operator/manifests:v0.3.0@sha256:f714b72f094d8bde1a20410e8178d997d25d5acab46cf9bf46cb6f6ea46647f1
flux pull artifact oci://ghcr.io/phoban01/battery-operator/manifests:v0.3.0 --output ./battery-operator

Images, for linux/amd64 and linux/arm64:

• ghcr.io/phoban01/battery-operator:v0.3.0
• ghcr.io/phoban01/battery-operator/exec-agent:v0.3.0
• ghcr.io/phoban01/battery-operator/host-image:v0.3.0

To make an AMI: make host-image-ami HOST_IMG=ghcr.io/phoban01/battery-operator/host-image:v0.3.0 HOST_IMAGE_AMI_ARGS="--bucket ... --region ...". AMIs are x86_64 only.

battery-operator v0.2.1: the Manifests as an OCI artifact for Flux

A release now publishes the Manifests as an OCI artifact, so Flux can install battery-operator with an OCIRepository and a Kustomization.

• ghcr.io/phoban01/battery-operator/manifests:v0.2.1 holds one file, install.yaml: the Operator with battery as its sidecar, and the Exec Agent. The images in it are pinned to this release by digest. See the install guide.
• The docs site has two new pages: Install, with Flux or kubectl, and Hosts, from the Host Image to Hosts made with Cluster API on AWS. The Hosts page has not run against AWS yet.
• make build-installer writes the same install.yaml to dist/.

Manifests artifact:

ghcr.io/phoban01/battery-operator/manifests:v0.2.1@sha256:96ce039319c88798b14f2a5f114ed19edfa9d5c8511b5035295a6b27a3c7a332
flux pull artifact oci://ghcr.io/phoban01/battery-operator/manifests:v0.2.1 --output ./battery-operator

Images, for linux/amd64 and linux/arm64:

• ghcr.io/phoban01/battery-operator:v0.2.1
• ghcr.io/phoban01/battery-operator/exec-agent:v0.2.1
• ghcr.io/phoban01/battery-operator/host-image:v0.2.1

The Operator, the Exec Agent and the Host Image have not changed since v0.2.0.

battery-operator v0.2.0: a reference Host Image and Cluster API host pools

battery-operator now provides a reference Host Image, and Cluster API templates that make Hosts from it (ADR 0007).

• The Host Image is a bootc image on Fedora bootc 44. It carries containerd with its thin pool, Firecracker, Cloud Hypervisor, flintlockd over mutual TLS, the kubelet and kubeadm, a guest bridge with DHCP, NAT and isolation from the cluster, and SELinux enforcing. It is built for linux/amd64 and linux/arm64. See hostimage/README.md.
• The Cluster API host pool templates in config/capi/ boot the Host Image on AWS and join each Host with kubeadm, with one settings object per pool: instance type, AMI, subnet, security groups, root volume size and the Host's settings. See config/capi/README.md.
• hack/kubeadm-host/ proves the two together on a Mac: a Host booted from the arm64 image joins a kubeadm cluster, and runs a Pool, a claim and the guest network.
• A Bound claim now has a HostReady condition, taken from the Exec Agent's report on its Host. The Client Library reports a Host that goes not ready to the Consumer (Claim.HostNotReady(), CL-050 to CL-053, CC-013).
• The EXPIRES column of kubectl get microvmclaims shows a timestamp (#161).

Images:

• ghcr.io/phoban01/battery-operator:v0.2.0 and ghcr.io/phoban01/battery-operator/exec-agent:v0.2.0, for linux/amd64 and linux/arm64.
• ghcr.io/phoban01/battery-operator/host-image:v0.2.0, for linux/amd64 and linux/arm64. To make an AMI from it, see hostimage/README.md.

Docs: https://phoban01.github.io/battery-operator/

v0.1.0

The first release. battery-operator puts battery v0.3.3 behind two Kubernetes resources, Pool and MicroVMClaim (battery.liquidmetal-x.dev/v1alpha1).

• The Operator runs battery's poolmgrd as a sidecar and turns Pools and claims into battery calls. It approves and signs the Hosts' certificates, and tells battery which Nodes are Hosts.
• The Exec Agent on each Host lets the holder of a claim, and only the holder, run commands in its MicroVM.
• The Client Library, pkg/claimclient, claims a MicroVM, renews its Lease, dials the Exec Agent, and releases it.

Images: ghcr.io/phoban01/battery-operator:v0.1.0 and ghcr.io/phoban01/battery-operator/exec-agent:v0.1.0, for linux/amd64 and linux/arm64.

Docs and a recorded demo on a real Host: https://phoban01.github.io/battery-operator/

Known issue: the EXPIRES column of kubectl get microvmclaims shows <invalid> for a Bound claim (#161).