Skip to content

v0.16.1

Choose a tag to compare

@github-actions github-actions released this 02 May 16:02
v0.16.1
aeb2275

Breaking

  • Remove additional_nics[*].addresses and additional_nics[*].gateway — these fields shipped in v0.16.0 but are fundamentally unsafe on a shared MachineClass. Every worker in a MachineSet renders the same class, so a static IP in addresses would be claimed by N workers and collide; a static default-route gateway would duplicate across all workers and steer traffic through whichever interface the kernel picked first. There is no safe way to encode per-worker static addressing in a class-shared config. The sanctioned path for pinning specific IPs is an upstream DHCP reservation keyed off the deterministic MAC the provider logs at VM creation — those reservations survive reprovision because the MAC is derived from the machine request ID.
  • DHCP *bool on AdditionalNIC stays. The tri-state simplifies: nil / unset → DHCP enabled (golden path), explicit true → same as default, explicit false → link attached but left unconfigured for advanced users (bond slave, VLAN parent, manually-applied per-node patch). The v0.16.0 address-based default heuristic ("nil + addresses set → dhcp=false") is gone because addresses is gone.
  • Schema: additional_nics.items loses addresses and gateway properties; the remaining {network_interface, type, mtu, dhcp} shape is stable going forward.
  • Patch builder (buildAdditionalNICInterfacesPatch) no longer emits addresses or routes keys — the output is now strictly {deviceSelector.hardwareAddr, dhcp} per NIC. Any MachineRequest that reconciled under v0.16.0 with static fields set keeps the old patch in Omni state until the VM is replaced; v0.16.1 does not retroactively mutate old patches.
  • MaxAddressesPerNIC constant removed. MaxAdditionalNICs = 16 stays.
  • Validation drops all address/gateway branches (CIDR parse, multicast/loopback/zero-mask reject, gateway family match, on-link check, single-gateway-per-MachineClass check) and the config_invalid alert category loses those specific error-message fragments. The category itself stays and still fires on disk-size and duplicate-NIC typos.

Migration

Skip v0.16.0. Upgrade from v0.15.5 straight to v0.16.1. If you already deployed v0.16.0:

  1. Edit any MachineClass that sets additional_nics[*].addresses or .gateway — remove those fields. v0.16.1 rejects them at JSON-schema validation; MachineRequests against such classes will never reconcile until the class is edited.
  2. If you need a specific worker pinned to a specific IP on a secondary segment, add a DHCP reservation on the upstream router for the NIC's MAC (visible in provider logs: attached additional NIC … mac=02:…).
  3. VMs provisioned under v0.16.0 with static patches keep running; replace them against the v0.16.1 class when convenient.

Tests

  • multinic_test.go: removed 13 address/gateway tests (static valid, CIDR junk, unspecified, multicast, loopback, gateway invalid-IP, non-unicast, family mismatch, not-on-link, without-addresses, multiple-gateways, NICs-exceed-max-with-addresses, addresses-per-NIC-exceed-max). 3 new tests cover the simplified surface: DHCPTrue_Allowed, DHCPFalse_Allowed, AdditionalNICs_ExceedMax.
  • config_patch_test.go: removed StaticAddress, StaticWithGateway, DHCPPlusStatic patch-shape tests and the nil-defaults-to-false-when-addresses-set resolver case. Kept and pinned: SingleDHCPNIC now asserts patch MUST NOT carry addresses / routes keys.
  • schema_drift_test.go: field-type map loses addresses and gateway entries.
  • error_categorization_test.go: config_invalid test vectors swap address/gateway error strings for duplicate-NIC + NIC-exceeds-max strings.

Fixes — host-OOM surfacing and memory ballooning

  • Surface "TrueNAS host out of memory" instead of an endless uploadISO 2/4 UI freeze. Before this change, when vm.start returned the libvirt-relayed truenas api error (code 12): [ENOMEM] Cannot guarantee memory for guest …, the provisioner returned the raw error and Omni's step-progress UI stayed pinned on the previously-completed step (uploadISO) while the controller retried every ~60 s indefinitely. Operators saw "stuck on step 2 for an hour" with no visible cause unless they pulled provider logs. The error path now: (1) categorizes ENOMEM into a new host_oom provision-error bucket distinct from the existing memory bucket (oversized MachineClass) so dashboards and alerts can route them to different operator responses; (2) translates the wire error into a leading "TrueNAS host out of memory: cannot start VM N (name) requesting M MiB" message that names the diagnosis up front; (3) tracks consecutive ENOMEM retries per VM and returns a permanent error after MaxStartOOMAttempts (default 5) so MachineRequestStatus.Conditions shows the failure instead of the controller silently spinning. New client.IsNoMemory(err) helper, client.ErrCodeNoMemory = 12 constant, and UserFriendlyError switch case route the wire error consistently across the provisioner. Three call sites updated: stepCreateVM's vm.start, handleExistingVM's vm.start retry, and the post-NVRAM-reset start.
  • Pre-flight memory check now subtracts the running-guest commitment before deciding whether a new VM fits. Previously the check only compared memory against 80% of total physmem, which let a request through whenever it was small relative to the box even if every byte of free RAM was already locked by other VMs — exactly the v0.16.0 incident pattern (talos-home-workers-f9xkk2 failed step 3 because two earlier VMs had committed 28 GiB of a 32 GiB host before this one ran). The new check sums memory of every RUNNING guest via a single vm.query and rejects requests where the actual reservation (min_memory if set, otherwise memory) exceeds 90% of remaining free MiB, with a hint pointing at min_memory when not configured. The single-VM 80%-of-total ceiling stays as a second guard against ZFS ARC starvation. The aggregate query is best-effort: a vm.query failure logs at debug and falls back to the original ceiling rather than blocking provisioning on an observability call.
  • Add min_memory to MachineClass — soft floor for memory ballooning. New optional min_memory field (MiB, ≥ 1024 when set, ≤ memory) maps to TrueNAS's existing vm.create min_memory parameter. When set, the VM launches with min_memory reserved and balloons up to memory as host RAM is available — letting operators over-commit on tight hosts without hitting ENOMEM at start. When unset (the default), behavior is unchanged: memory is fully reserved. The memory field's schema description was rewritten to call out that it's the maximum / hard limit and that min_memory is the soft-floor escape hatch. The pre-flight check above compares against min_memory when set, so a balloon config that legitimately oversubscribes the ceiling no longer fails validation. Caveat documented in the schema description, the new docs section, and the sizing guide: the Talos kernel does not auto-load virtio-balloon, so until balloon is explicitly enabled in-guest the VM will sit at min_memory and memory becomes a ceiling that's never reached — in practice, size min_memory to what Talos actually needs.

Tests — host-OOM and balloon coverage

  • internal/client/vm_test.go: TestRunningGuestsMemoryMiB_OnlyCountsRunning (RUNNING-only summation; STOPPED guests excluded), TestRunningGuestsMemoryMiB_EmptyHost, and TestIsNoMemory (code 12, message-fallback [ENOMEM] and Cannot guarantee memory, code 28 ENOSPC negative case, non-API error, nil).
  • internal/provisioner/error_categorization_test.go: 5 new host_oom test vectors covering the raw libvirt string, the translated leading message, the permanent-failure suffix, the pre-flight rejection wording, and the UserFriendlyError output.
  • internal/provisioner/steps_test.go: TestHandleExistingVM_Stopped_StartFails_ENOMEM (operator-actionable wording on the existing-VM start path), TestTranslateStartError_PermanentAfterMaxAttempts (fail-fast pinned at the configured budget), TestTranslateStartError_NonOOMPassesThrough (counter doesn't advance on non-OOM errors), TestClearOOMAttempts_ResetsCounter. The legacy TestHandleExistingVM_Stopped_StartFails was updated to assert the new failed to start VM <id> (<name>) wording instead of the deprecated failed to start existing VM substring.
  • internal/provisioner/data_test.go: 7 new validation cases for min_memory (zero accepted, negative rejected, below MinMemoryMiB rejected, above memory rejected with field-named error, equal-to-memory accepted, between-floor-and-memory accepted, and the no-balloon path).

Docs — host-OOM and balloon coverage

  • docs/troubleshooting.md: new "VM creation succeeds but VM won't start: host out of memory" section with the symptom (frozen uploadISO 2/4 UI), the root cause (KVM-level guard, not bypassable), midclt diagnostic commands, and four prioritized fixes (stop another VM, set min_memory, manual vm.update override on the stuck VM, reduce memory, add RAM).
  • docs/sizing.md: new bullet under "Rules of thumb worth knowing" calling out memory as a hard reservation by default and min_memory as the balloon escape hatch with the Talos virtio-balloon caveat.
  • docs/quickstart.md: memory row description rewritten to flag it as the hard / max limit; new min_memory row with the soft-floor explanation and a link to the troubleshooting section.

Full Changelog: v0.16.1-rc.6...v0.16.1