RFC: Baked Image Strategy — pre-built VM images for DBInstance provisioning (feature-imagebaking)

[tgcjananga]

Problem

Today every DBInstance provisions a PostgreSQL database by running apt-get install postgresql
at first VM boot. This creates four compounding issues:

1. Boot-to-ready is ~3 minutes — apt-get install postgresql is the dominant cost. Customers wait.
2. Network required at boot — if archive.ubuntu.com or apt.postgresql.org is slow during a burst of deployments, VMs hang silently and DBInstances enter Failed.
3. engineVersion is completely ignored — the field is omitempty and never validated. spec.engineVersion: "17" silently gives whatever version apt resolves that day.
4. COW storage savings are erased — Harvester stores OS disks as Longhorn BackingImage deltas (~200 MB per VM by design). apt-get install postgresql writes 1–2 GB of packages into every VM's delta at first boot. At 1000 VMs that is ~1–2 TB in deltas vs ~200 GB with baked images.

---

Proposed Solution — Baked Image + Repave

Pre-build a qcow2 image containing all supported PostgreSQL versions (15, 16, 17) using Packer.
Upload to Harvester as a BackingImage. All VMs share this base via Longhorn COW — each VM holds
only a ~200 MB config delta.

Current:   base ~1 GB + delta ~1–2 GB per VM  (apt writes packages at boot)
Baked:     base ~5 GB stored once per node    + delta ~200 MB per VM  (config only)

Boot-to-ready drops from ~3 min to 30–60 seconds. No network calls at boot.

---

How Packer Builds the Image

%%{init: {'themeVariables': {'fontSize': '18px'}}}%%
flowchart LR
    A(["Ubuntu 22.04\ncloud image"])

    subgraph PACKER["Packer Build  runs once per patch cycle"]
        direction LR
        B["Boot temporary\nQEMU VM"]

        subgraph PROVISION["provision.sh  runs inside VM over SSH"]
            direction TB
            P1["apt-get upgrade\nsecurity patches"]
            P2["Install PG 15 / 16 / 17\ndisable all PG clusters"]
            P3["Disable unattended-upgrades\nand apt daily timers"]
            P4["cloud-init clean\ntruncate machine-id\nremove Packer SSH key"]
            P1 --> P2 --> P3 --> P4
        end

        D["Shut down VM\nexport disk as qcow2"]
        B --> P1
        P4 --> D
    end

    E(["ubuntu-2204-postgres-vYYYYMMDD.qcow2\nready to upload to Harvester"])

    A --> B
    D --> E

Loading

Why seal the image?
cloud-init clean wipes all first-boot state so every VM booted from this qcow2 runs
bootstrap.sh fresh. Without it VMs look like the already-booted Packer VM and skip
initialization entirely — PostgreSQL never starts.

truncate -s 0 /etc/machine-id forces systemd to generate a unique UUID per VM at boot,
preventing DHCP IP conflicts when many VMs come up from the same base image.

Why disable unattended-upgrades?
Ubuntu's background apt updater would silently pull packages into each VM's COW delta after
provisioning — erasing the storage savings and causing PostgreSQL minor-version drift between
VMs provisioned at different times. Security patches are instead delivered via a controlled
image rebuild and repave .

---

End-to-End Lifecycle

%%{init: {'themeVariables': {'fontSize': '18px'}}}%%
flowchart TD
    subgraph OPS["Ops Team  once per patch cycle"]
        B1["Build\n./build.sh YYYYMMDD\nPacker produces qcow2"]
        B2["Publish\nUpload to Harvester\nBackingImage + StorageClass auto-created"]
        B3["Catalog update + redeploy\nLatestBakedImages bumped\noperator redeployed"]
        B1 --> B2 --> B3
    end

    B3 --> NEW_PATH
    B3 --> EXIST_PATH

    subgraph NEW_PATH["New DBInstance  customer applies CR"]
        N1{"engineVersion\nin catalog?"}
        N_REJECT["Rejected\nclear error returned"]
        N2["OS disk provisioned\nCOW delta on new BackingImage\n30-60 sec ready, no apt at boot"]
        N3["status.phase = Available\nimageRevision = latest"]
        N1 -- No --> N_REJECT
        N1 -- Yes --> N2 --> N3
    end

    subgraph EXIST_PATH["Existing Running VMs  operator reconcile"]
        E1["imageRevision != LatestBakedImage\nOSUpdateAvailable = True"]
        E1 --> REPAVE_TYPE
        REPAVE_TYPE{"What changed?"}
    end

    subgraph PATCH["Security Patch or new PG minor\ne.g. 22.04.5 or PG 16.4"]
        R1["Repave same OS stream\nclear pgdata ownerRef\nstop VM, swap OS disk\nstart VM, cloud-init re-runs"]
        R2["pgdata untouched\nPG version unchanged\n30-60 sec downtime"]
        R1 --> R2
    end

    subgraph OS_EOL["Ubuntu LTS EOL  e.g. 22.04 reaches Apr 2027"]
        OE1["DefaultOSVersion flipped to 24.04\nnew VMs get 24.04 automatically"]
        OE2["Repave each 22.04 VM to 24.04 stream\nOS disk replaced, pgdata untouched\nPG version stays the same"]
        OE1 --> OE2
    end

    subgraph PG_EOL["PG Major Version EOL  e.g. PG 15 reaches Nov 2027"]
        PE1["Removed from new image builds\nnew DBInstances with engineVersion 15 rejected"]
        PE2["Existing PG 15 VMs keep running\nPGVersionEOL = True flagged on CR"]
        PE3["Customer must migrate\npg_dump and restore to PG 16+\nno in-place major upgrade path"]
        PE1 --> PE2 --> PE3
    end

    REPAVE_TYPE -->|"OS or PG patch"| PATCH
    REPAVE_TYPE -->|"Ubuntu EOL migration"| OS_EOL
    REPAVE_TYPE -->|"PG version EOL"| PG_EOL

    R2 --> E1
    OE2 -.->|"next patch cycle"| B1

Loading

---

Release Cadence

Signal	Frequency	Action
Ubuntu 22.04.x point release	~6 months	Build + staged repave all VMs
PostgreSQL minor release	1–3 months	Build + staged repave all VMs
Critical CVE (CVSS >= 7)	As needed	Same-day build + immediate repave
New PG major version	Yearly	Add to PG_VERSIONS, new enum value
New Ubuntu LTS	Every 2 years	New parallel stream in catalog

Security patches are not applied by monthly scheduled builds — that would rebuild without
any new patches half the time. The build is triggered by an actual release event.

---

Key Design Decisions — Input Wanted

1. Image catalog structure

One image per OS version containing all PG versions vs one image per OS+PG pair.

• All-in-one (proposed): cloud-init activates only the requested PG version at boot. Catalog is simple — one entry per build date.
• Per-pair: Smaller images but N×M catalog entries, more upload and storage overhead.

2. Multi-OS stream

Two parallel streams keyed by OS version:

var LatestBakedImages = map[string]string{
    "22.04": "v20260615",
    "24.04": "v20261201",   // starts in parallel when ready
}

DefaultOSVersion = "22.04" until the 24.04 stream is validated.

3. Repave trigger

Currently scoped to manual kubectl annotate dbinstance ... repave=true. Should we support:

• a) Manual only — operator sets OSUpdateAvailable=True, human triggers repave
• b) Automatic rolling repave with a maintenance window annotation
• c) Operator-scheduled rolling repave with maxUnavailable

4. engineVersion validation

Make engineVersion required with a kubebuilder enum ("15", "16", "17"). This is a
breaking API change — existing CRs without a valid value will fail admission. Migration path needed.

5. BackingImage replica count — numberOfReplicas on upload

Drawback without attention here: Our prod cluster has 4 nodes but harvester-longhorn template
StorageClass is set to numberOfReplicas: "2". A VM scheduled onto a node that has no local BackingImage
copy triggers an on-demand sync before the OS disk can be used — this can add significant latency for very first vm in that node (similar to the original ~3 min apt boot problem) and defeats the purpose of image baking for fast provisioning.

Option 1 — Set numberOfReplicas: "4" on VirtualMachineImage upload

spec:
  storageClassParameters:
    numberOfReplicas: "4"

• BackingImage is distributed proactively to all 4 nodes at upload time
• Any VM on any node gets instant COW — zero sync latency at provision time
• Overrides the harvester-longhorn template "2" via mergeStorageClassParams() in Harvester's mutator
• Auto-generated longhorn-ubuntu-postgres-vX StorageClass inherits numberOfReplicas: "4"
• Optimal for rapid provisioning objective (~30–60 sec boot)

Option 2 — Keep "2" (inherit template StorageClass)

• Do not set storageClassParameters.numberOfReplicas in VirtualMachineImage spec
• BackingImage lands on 2 of 4 nodes → 50% chance of on-demand sync on first VM boot on a new node
• Sync adds minutes of latency on affected nodes, matching the old apt-install boot time
• Effectively defeats the fast-provisioning goal half the time

Question: Should we enforce numberOfReplicas: "4" in the upload spec (Option 1) and document it
as a required step in the image publishing playbook?

---

Major Operator Changes

File	Change
api/v1alpha1/dbinstance_types.go	BakedImage type + BakedImages catalog + LatestBakedImages map; engineVersion required+enum; ImageRevision in AppliedSpec;
internal/harvester/client.go	Replace p.OSImage lookup with catalog entry
internal/harvester/cloudinit.go	Remove apt-get install block; add ENGINE_VERSION env; add PG cluster activation
internal/controller/dbinstance_controller.go	Catalog lookup + engineVersion validation; ImageRevision snapshot; drift detection; phaseRepave()
images/packer/scripts/provision.sh	Already done — disable unattended-upgrades + apt timers before sealing

---

Open Questions

• Should osVersion be exposed in the DBInstance spec, or kept as an operator-internal concern?
• PG major version EOL requires customer-driven data migration (pg_dump + restore). What notice period should we mentioned to?
• Is 30–60 sec repave downtime acceptable (standby + failover before repave)?

[Akini98]

Overall, this is a very solid idea to have baked images.

Regarding the design decisions, please find my suggestions below.

1. Image catalog structure
All-in-one image is the better option. N×M catalog entries for per-OS+PG pair images would create operational complexity that outweighs the size savings. One issue with the all-in-one image is multiple PG versions can accumulate overtime. So there should be a guard against this by enforcing atleast a strict EOL removal policy in the build script.

2. Multi-OS stream
Yes, it is better to have two parallel streams, but it must be ensure that a new version is not set as the default version until it is fully tested. So it is better if it possible to have a validation gate to that map so the default can't be flipped to an untested stream by accident.

var LatestBakedImages = map[string]BakedImage{
"22.04": {Revision: "v20260615", Validated: true},
"24.04": {Revision: "v20261201", Validated: false}, // still being tested
}

3. Repave trigger
For initial release it is better to begin with manual trigger, but it needs to be built such that in future it can support scheduled or automated repave triggers. Annotations can be designed like repave-trigger=now. and in next release it could additionally handle repave-trigger=scheduled.

4. engineVersion validation
Since existing CRs without a valid value will fail admission during next update, all existing CRs should be patched with the current engineVersion. Also it is better if a defaulting webhook can be added, such that it sets default to a value like "16" or "17". But this webhook should check against the actual LatestBakedImages catalog so that an EOL version gets rejected at admission time, not silently at VM boot.

5. BackingImage replica count
For backingImage replica count, it is better to have the backing images on each node. (i.e. replica-count = 4). Would it be possible to have a separate storage class with 4 replicas for just the backing images created for these baked images. Option 2 defeats the purpose of reducing time for boot up in my opinion. It is better if the BackingImage replica count can be a configurable parameter as the number of nodes in the harvester setup can change.

In my opinion, osVersion does not need to be exposed

[tgcjananga]

Thanks for the detailed feedbacks ,

---

• Image catalog stays all-in-one (one revision per build, all PG versions inside, cloud-init activates only the requested one). As a related improvement I added an EOL guard directly in build.sh — it validates every PG version in PG_VERSIONS against a PG_EOL_DATES table before Packer even starts, so a past-EOL version can never be accidentally baked in.

• Multi-OS streams are now modelled with a BakedImageStream struct that carries both a revision and a Validated bool. Each qcow2 image uploaded to Harvester gets its own revision entry in BakedImages and is never deleted — old VMs reference their creation-time revision for drift detection. LatestBakedImages just tracks the current tip per OS stream:

// All revisions ever built — never deleted
var BakedImages = map[string]BakedImageEntry{
    "v20260615":      { OSVersion: "22.04", PGVersions: []string{"15","16","17"} },
    "v20261201-2404": { OSVersion: "24.04", PGVersions: []string{"15","16","17"} },      // initial 24.04 test image
    "v20270301-2404": { OSVersion: "24.04", PGVersions: []string{"15","16","17","18"} }, // PG 18 added
}

// Current tip per OS stream — only this changes when a new image is released
var LatestBakedImages = map[string]BakedImageStream{
    "22.04": {Revision: "v20260615",      Validated: true},
    "24.04": {Revision: "v20261201-2404", Validated: false}, // still in testing
}

Validated: false means the controller will refuse to use that stream even if BACKING_IMAGE_OS_VERSION is accidentally pointed at it — an untested image can never reach production VMs without an explicit sign-off in the catalog. Once the 24.04 stream passes testing, set Validated: true and update the env var to "24.04" — new DBInstances get 24.04 automatically, existing 22.04 VMs get OSUpdateAvailable=True and are migrated via repave.

DefaultOSVersion is an operator env var (BACKING_IMAGE_OS_VERSION) rather than a Go constant — flipping the OS default is an env var update + rolling restart, no code change or PR needed.

Release Cadence also update to align with M1,

Signal	Frequency	Action
Ubuntu point release or PG minor release	~6 months / 1–3 months	./build.sh YYYYMMDD — apt-get upgrade picks up both Ubuntu and PG patches in one build; bump catalog, redeploy operator; operator sets OSUpdateAvailable=True on existing VMs; repave triggered manually per VM (M1)
Critical CVE (CVSS >= 7)	As needed	Same-day build + immediate repave
New PG major version	Yearly	Add to PG_VERSIONS arg, update BakedImages catalog, redeploy — no enum change needed, catalog is source of truth
New Ubuntu LTS	Every 2 years	New parallel stream in LatestBakedImages with Validated: false; flip BACKING_IMAGE_OS_VERSION env var when tested and ready

PG minor versions are never tracked or exposed — apt-get upgrade picks up whatever is current at build time. Customers only ever specify a major version (engineVersion: "16").

M1 scope: repave is manually triggered per VM via kubectl annotate ... repave-trigger=now. Automated rolling repave is M2.

---

• On the osVersion exposure question: keeping it internal. Customers specify only engineVersion (the PG major version they want). The OS stream is an infrastructure concern( for M1) they shouldn't need to know about, and hiding it means OS migrations (22.04 → 24.04) are fully transparent — same connection string, same data, just a repave.

• Repave trigger changed from repave=true to repave-trigger=now. The value-based annotation means M2 can add repave-trigger=scheduled without any API breaking change — the controller just adds a new case to its switch. One thing I hadn't considered in the original proposal: the annotation must be cleared by the controller after the repave completes (Step 7), otherwise the next reconcile sees repave-trigger=now still set and triggers another repave. That's now explicit in the phaseRepave sequence.

• engineVersion stays optional (omitempty). Rather than making it required (breaking change — existing CRs without the field would fail admission), a defaulting webhook sets it to the latest PG version from the live catalog if the customer leaves it empty. A validating webhook then checks the resolved version against the live catalog and rejects EOL versions with a clear error.Then No pre-rollout migration script needed — the field was already optional, so all existing CRs remain valid.

• BackingImage replica count is now driven by a BACKING_IMAGE_REPLICAS operator env var rather than hardcoded "4". The operator passes it as storageClassParameters.numberOfReplicas when managing VirtualMachineImage resources. Setting replicas equal to node count at upload time is the solution now.

---

none of these decisions affect the boot identity flow.

[tgcjananga]

EOL handling — two separate paths

Version EOL splits into two completely different flows depending on whether it's a PostgreSQL major version or an Ubuntu LTS. The key difference: PG stores data in a version-specific on-disk format — a repave alone cannot change the PG major version without the data directory becoming unreadable. Ubuntu has no such constraint.

%%{init: {'themeVariables': {'fontSize': '12px'}}}%%
flowchart TD
    EOL(["Version reaches EOL"])
    EOL --> PG_EOL["PG Major Version EOL\ne.g. PG 15 — Nov 2027"]
    EOL --> OS_EOL["Ubuntu LTS EOL\ne.g. 22.04 — Apr 2027"]

    subgraph PG_PATH["PG EOL — Customer action required"]
        direction TB
        PG1["Ops: remove from build\nOperator: PGVersionEOL = True"]
        PG2["Customer: pg_dump old VM\nrestore to new DBInstance\nwith engineVersion: 16"]
        PG3["Cutover + delete old VM\nor force-deleted at T+3 months"]
        PG1 --> PG2 --> PG3
    end

    subgraph OS_PATH["Ubuntu EOL — Transparent to customer"]
        direction TB
        OS1["Ops: flip BACKING_IMAGE_OS_VERSION\nto 24.04 stream"]
        OS2["Operator: OSUpdateAvailable = True\non existing 22.04 VMs"]
        OS3["Repave: OS disk swapped\npgdata untouched\nsame IP, same data"]
        OS1 --> OS2 --> OS3
    end

    PG_EOL --> PG1
    OS_EOL --> OS1

Loading

[tgcjananga]

---

Update — Implementation & Testing Findings

1. Overview of Image Baking Feature M1

2. High Level Architecture - VM Provisioning and Teardown Flow

flowchart LR
    subgraph Provision
        P1["VM annotation:<br/>volumeClaimTemplates"]
        P2["Harvester reads annotation"]
        P3[("OS PVC<br/>no owner")]
        P4[("pgdata PVC<br/>no owner")]
        P1 --> P2
        P2 --> P3
        P2 --> P4
    end

    subgraph Repave
        R1["New image"]
        R2["New PVC name"]
        R3["Patch annotation"]
        R4[("New OS PVC")]
        R5["Delete old PVC"]
        R1 --> R2 --> R3 --> R4 --> R5
    end

    subgraph Teardown
        T1["Delete instance"]
        T2["Delete VM"]
        T3[("PVCs remain<br/>no cascade")]
        T4["Explicit PVC delete"]
        T1 --> T2 --> T3 --> T4
    end

    P3 --> R1
    R5 --> T1

Loading

Attached for the discussion — this is the visual counterpart to point 1 above: the actual Harvester-native volume path (annotation-declared PVC, no DataVolume/dataVolumeTemplates involved) that the operator's repave and teardown logic had to be built against, as opposed to the KubeVirt-native DataVolume path the original plan assumed. Worth keeping alongside the prose explanation above and the harvesterhci.io/volumeClaimTemplates mechanics as the reference diagram for anyone ramping up on why repave/teardown work the way they do.

3. Ownerless PVCs

Each repave gives the OS disk a revision-suffixed name (pg-<id>-os-v20260815, never a reused pg-<id>-os). SwapVMOSDisk edits the volumeClaimTemplates annotation and the volume's claimName to the new name; Harvester provisions a fresh PVC from the new image. The old PVC — a distinct name — is deleted afterward. Old and new never share a name, so creation and deletion can't race over the same object — this makes both bugs structurally impossible rather than just less likely.

Harvester creates annotation-declared PVCs with no ownerReference. Deleting the VM does not cascade-delete its disks. TeardownAll (in typed_client.go) now explicitly collects and deletes every OS + pgdata PVC. Without this, repeated create/delete cycles on the same instance name silently reattached the previous instance's old disk — a customer deleting and recreating a DBInstance would get someone else's old data back.

4. Blocked-repave failures needed an unplanned annotation-clearing fix

phaseRepave's Step-0 pre-flight guards (InvalidOSStream, and the new RepaveBlockedPGVersionEOL — see EOL section below) call fail(), which sets ProvisioningPhase=Failed but never clears repave-trigger. The immediate error-triggered re-reconcile then hit the other annotation guard (annotation still "now", phase now Failed) and overwrote the real failure reason with a generic "requires ProvisioningPhase=Available" message — within under a second, and forever, since nothing ever cleared the annotation. A new failBlockedRepave() clears the annotation specifically for these two pre-flight guards (the request itself is invalid; re-annotating won't fix it). Every other phaseRepave failure (stop/swap/start) still leaves the annotation in place on purpose, so transient infra errors keep retrying automatically without operator intervention.

5. Engine version validation

engineVersion defaulting/validation against the live catalog — run at reconcile time in phaseVM() (UnsupportedEngineVersion), not at admission. Trade-off: a bad request is accepted into etcd and fails a moment later instead of being rejected instantly at kubectl apply. This was an implementation choice made along the way, not caught as a deviation until writing this update — flagging it here so it's an explicit, known gap rather than something someone discovers by grep later. Worth a real webhook in M2 if instant admission feedback becomes a requirement.

6. PG major-version EOL guards

Drift detection ended up needing two mutually-exclusive conditions, not one: OSUpdateAvailable=True when the running engineVersion is still supported by the new revision (safe to repave), PGVersionEOL=True instead when it's been dropped (repave would be destructive — must not offer it as routine maintenance). New-instance creation also independently rejects an EOL engineVersion