v0.16.1
Breaking
- Remove
additional_nics[*].addressesandadditional_nics[*].gateway— these fields shipped in v0.16.0 but are fundamentally unsafe on a shared MachineClass. Every worker in a MachineSet renders the same class, so a static IP inaddresseswould be claimed by N workers and collide; a static default-route gateway would duplicate across all workers and steer traffic through whichever interface the kernel picked first. There is no safe way to encode per-worker static addressing in a class-shared config. The sanctioned path for pinning specific IPs is an upstream DHCP reservation keyed off the deterministic MAC the provider logs at VM creation — those reservations survive reprovision because the MAC is derived from the machine request ID. DHCP *boolonAdditionalNICstays. The tri-state simplifies: nil / unset → DHCP enabled (golden path), explicittrue→ same as default, explicitfalse→ link attached but left unconfigured for advanced users (bond slave, VLAN parent, manually-applied per-node patch). Thev0.16.0address-based default heuristic ("nil + addresses set → dhcp=false") is gone becauseaddressesis gone.- Schema:
additional_nics.itemslosesaddressesandgatewayproperties; the remaining{network_interface, type, mtu, dhcp}shape is stable going forward. - Patch builder (
buildAdditionalNICInterfacesPatch) no longer emitsaddressesorrouteskeys — the output is now strictly{deviceSelector.hardwareAddr, dhcp}per NIC. Any MachineRequest that reconciled under v0.16.0 with static fields set keeps the old patch in Omni state until the VM is replaced; v0.16.1 does not retroactively mutate old patches. MaxAddressesPerNICconstant removed.MaxAdditionalNICs = 16stays.- Validation drops all address/gateway branches (CIDR parse, multicast/loopback/zero-mask reject, gateway family match, on-link check, single-gateway-per-MachineClass check) and the
config_invalidalert category loses those specific error-message fragments. The category itself stays and still fires on disk-size and duplicate-NIC typos.
Migration
Skip v0.16.0. Upgrade from v0.15.5 straight to v0.16.1. If you already deployed v0.16.0:
- Edit any MachineClass that sets
additional_nics[*].addressesor.gateway— remove those fields. v0.16.1 rejects them at JSON-schema validation; MachineRequests against such classes will never reconcile until the class is edited. - If you need a specific worker pinned to a specific IP on a secondary segment, add a DHCP reservation on the upstream router for the NIC's MAC (visible in provider logs:
attached additional NIC … mac=02:…). - VMs provisioned under v0.16.0 with static patches keep running; replace them against the v0.16.1 class when convenient.
Tests
multinic_test.go: removed 13 address/gateway tests (static valid, CIDR junk, unspecified, multicast, loopback, gateway invalid-IP, non-unicast, family mismatch, not-on-link, without-addresses, multiple-gateways, NICs-exceed-max-with-addresses, addresses-per-NIC-exceed-max). 3 new tests cover the simplified surface:DHCPTrue_Allowed,DHCPFalse_Allowed,AdditionalNICs_ExceedMax.config_patch_test.go: removedStaticAddress,StaticWithGateway,DHCPPlusStaticpatch-shape tests and thenil-defaults-to-false-when-addresses-setresolver case. Kept and pinned:SingleDHCPNICnow asserts patch MUST NOT carryaddresses/routeskeys.schema_drift_test.go: field-type map losesaddressesandgatewayentries.error_categorization_test.go: config_invalid test vectors swap address/gateway error strings for duplicate-NIC + NIC-exceeds-max strings.
Fixes — host-OOM surfacing and memory ballooning
- Surface "TrueNAS host out of memory" instead of an endless
uploadISO 2/4UI freeze. Before this change, whenvm.startreturned the libvirt-relayedtruenas api error (code 12): [ENOMEM] Cannot guarantee memory for guest …, the provisioner returned the raw error and Omni's step-progress UI stayed pinned on the previously-completed step (uploadISO) while the controller retried every ~60 s indefinitely. Operators saw "stuck on step 2 for an hour" with no visible cause unless they pulled provider logs. The error path now: (1) categorizes ENOMEM into a newhost_oomprovision-error bucket distinct from the existingmemorybucket (oversized MachineClass) so dashboards and alerts can route them to different operator responses; (2) translates the wire error into a leading "TrueNAS host out of memory: cannot start VM N (name) requesting M MiB" message that names the diagnosis up front; (3) tracks consecutive ENOMEM retries per VM and returns a permanent error afterMaxStartOOMAttempts(default 5) soMachineRequestStatus.Conditionsshows the failure instead of the controller silently spinning. Newclient.IsNoMemory(err)helper,client.ErrCodeNoMemory = 12constant, andUserFriendlyErrorswitch case route the wire error consistently across the provisioner. Three call sites updated: stepCreateVM's vm.start, handleExistingVM's vm.start retry, and the post-NVRAM-reset start. - Pre-flight memory check now subtracts the running-guest commitment before deciding whether a new VM fits. Previously the check only compared
memoryagainst 80% of totalphysmem, which let a request through whenever it was small relative to the box even if every byte of free RAM was already locked by other VMs — exactly the v0.16.0 incident pattern (talos-home-workers-f9xkk2failed step 3 because two earlier VMs had committed 28 GiB of a 32 GiB host before this one ran). The new check sumsmemoryof every RUNNING guest via a singlevm.queryand rejects requests where the actual reservation (min_memoryif set, otherwisememory) exceeds 90% of remaining free MiB, with a hint pointing atmin_memorywhen not configured. The single-VM 80%-of-total ceiling stays as a second guard against ZFS ARC starvation. The aggregate query is best-effort: avm.queryfailure logs at debug and falls back to the original ceiling rather than blocking provisioning on an observability call. - Add
min_memoryto MachineClass — soft floor for memory ballooning. New optionalmin_memoryfield (MiB, ≥ 1024 when set, ≤memory) maps to TrueNAS's existingvm.createmin_memoryparameter. When set, the VM launches withmin_memoryreserved and balloons up tomemoryas host RAM is available — letting operators over-commit on tight hosts without hitting ENOMEM at start. When unset (the default), behavior is unchanged:memoryis fully reserved. Thememoryfield's schema description was rewritten to call out that it's the maximum / hard limit and thatmin_memoryis the soft-floor escape hatch. The pre-flight check above compares againstmin_memorywhen set, so a balloon config that legitimately oversubscribes the ceiling no longer fails validation. Caveat documented in the schema description, the new docs section, and the sizing guide: the Talos kernel does not auto-loadvirtio-balloon, so until balloon is explicitly enabled in-guest the VM will sit atmin_memoryandmemorybecomes a ceiling that's never reached — in practice, sizemin_memoryto what Talos actually needs.
Tests — host-OOM and balloon coverage
internal/client/vm_test.go:TestRunningGuestsMemoryMiB_OnlyCountsRunning(RUNNING-only summation; STOPPED guests excluded),TestRunningGuestsMemoryMiB_EmptyHost, andTestIsNoMemory(code 12, message-fallback[ENOMEM]andCannot guarantee memory, code 28 ENOSPC negative case, non-API error, nil).internal/provisioner/error_categorization_test.go: 5 newhost_oomtest vectors covering the raw libvirt string, the translated leading message, the permanent-failure suffix, the pre-flight rejection wording, and theUserFriendlyErroroutput.internal/provisioner/steps_test.go:TestHandleExistingVM_Stopped_StartFails_ENOMEM(operator-actionable wording on the existing-VM start path),TestTranslateStartError_PermanentAfterMaxAttempts(fail-fast pinned at the configured budget),TestTranslateStartError_NonOOMPassesThrough(counter doesn't advance on non-OOM errors),TestClearOOMAttempts_ResetsCounter. The legacyTestHandleExistingVM_Stopped_StartFailswas updated to assert the newfailed to start VM <id> (<name>)wording instead of the deprecatedfailed to start existing VMsubstring.internal/provisioner/data_test.go: 7 new validation cases formin_memory(zero accepted, negative rejected, belowMinMemoryMiBrejected, abovememoryrejected with field-named error, equal-to-memory accepted, between-floor-and-memory accepted, and the no-balloon path).
Docs — host-OOM and balloon coverage
docs/troubleshooting.md: new "VM creation succeeds but VM won't start: host out of memory" section with the symptom (frozenuploadISO 2/4UI), the root cause (KVM-level guard, not bypassable),midcltdiagnostic commands, and four prioritized fixes (stop another VM, setmin_memory, manualvm.updateoverride on the stuck VM, reducememory, add RAM).docs/sizing.md: new bullet under "Rules of thumb worth knowing" calling outmemoryas a hard reservation by default andmin_memoryas the balloon escape hatch with the Talos virtio-balloon caveat.docs/quickstart.md:memoryrow description rewritten to flag it as the hard / max limit; newmin_memoryrow with the soft-floor explanation and a link to the troubleshooting section.
Full Changelog: v0.16.1-rc.6...v0.16.1