v0.16.0

Breaking

• Raise disk_size minimum from 5 GiB to 20 GiB on the root disk — the additional-disk floor stays at 5 GiB, but the primary / OS disk now fails validation below 20. Rationale lives in docs/sizing.md#why-the-root-disk-has-a-20-gib-minimum: a Talos CP node pulls kube-apiserver + kube-controller-manager + kube-scheduler + etcd + kube-proxy + CNI + CoreDNS during bootstrap, plus the Talos squashfs image and kubelet's 10% GC headroom. A 5–10 GiB root disk fills up mid-install, the kubelet evicts images mid-pull, and etcd never comes up — observed on the 5 GiB default path before this change. New MinRootDiskSizeGiB const in internal/provisioner/data.go, validator message cites the bootstrap reason, schema.json updated to "minimum": 20 with matching description. Migration: any MachineClass currently specifying disk_size < 20 will fail validation on next apply — edit the value to ≥ 20 (default 40 recommended for production) before provisioning. Existing VMs built against an older class are not retroactively resized; reprovision against the updated class if you hit DiskPressure.

Fixes

• Auto-configure every additional NIC (DHCP, static addresses, gateway) — stepCreateVM now emits a nic-interfaces ConfigPatchRequest alongside the existing nic-mtu patch whenever additional_nics are declared on the MachineClass. The patch writes machine.network.interfaces[] entries keyed by deviceSelector.hardwareAddr with dhcp, addresses, and per-interface default-route routes derived from the new AdditionalNIC fields. Talos's default platform config (nocloud, metal, …) only DHCPs the primary link, so before this fix additional NICs came up at the link layer but never acquired an IPv4 address — VMs were effectively single-homed despite the hypervisor attaching the extra vNIC correctly. Observed on talos-home workers running v0.15.5: talosctl get links showed both eth0 and eth1 UP with linkState: true, but talosctl get addresses showed eth1 with only fe80::/64 and no IPv4. MAC-based matching (not interface-name matching) is used because Talos's interface enumeration can shift between boots while the deterministic MACs the provider assigns survive reprovision. Orthogonal to advertised_subnets — that pins kubelet/etcd to a specific subnet but does not bring additional links up. Worker-safe: no cluster.* section, so no risk of the v0.15.0–v0.15.3 etcd-on-worker validation bug returning.

  New AdditionalNIC fields (all optional, backward-compatible):

  • dhcp (*bool) — tri-state. Unset → default true when no static addresses, false when addresses is set. Explicit true or false always wins, so "DHCP + static alias on the same link" and "attach the NIC but disable all autoconfig" are both expressible.
  • addresses ([]string) — static IPv4/IPv6 addresses in CIDR form. Validated with net.ParseCIDR at config-load time; bad entries are rejected with a field-indexed error before the VM is touched.
  • gateway (string) — optional default-route IP. Only meaningful alongside addresses — a gateway without addresses is rejected as a config mistake (DHCP supplies its own gateway; a static-only route with no link address can't be installed).

  New buildAdditionalNICInterfacesPatch builder + resolveNICDHCP policy helper. 20 new regression tests: 4 for the DHCP-default resolver (nil→true/false, explicit true/false), 11 for patch shape (DHCP on/off, static address, static+gateway, DHCP+static coexistence, multi-NIC mixed, empty list→nil, empty-MAC skip, all-empty→nil, JSON structure pin, no-cluster-section pin), 7 for Validate (valid CIDR, invalid CIDR, junk, valid gateway, invalid gateway IP, gateway-without-addresses rejected, DHCP-false-with-no-addresses allowed). Schema drift test guards the three new fields in cmd/omni-infra-provider-truenas/data/schema.json against silent removal.

Hardening (parallel-review follow-up, same session)

• Tighter address/gateway validation. Data.Validate now rejects interface addresses that are unspecified (0.0.0.0/0, ::/0), multicast, loopback, or zero-length-mask; rejects gateways that are unspecified, multicast, loopback, or IPv4 broadcast; enforces IPv4/IPv6 family match between gateway and at least one address; enforces that the gateway is on-link with at least one of the configured CIDRs; and rejects MachineClasses declaring a gateway on more than one additional NIC (non-deterministic default-route ambiguity). Closes the "malicious-operator-of-a-MachineClass" footguns where a bad value would either fail at Talos apply-time or silently steer worker traffic.
• Operator-input DoS caps. MaxAdditionalNICs = 16 and MaxAddressesPerNIC = 16 enforced in Validate() and as maxItems in schema.json. Prevents a misconfigured MachineClass with 10k entries from serializing a multi-MB ConfigPatchRequest that Omni stores + every reconcile re-fetches.
• Schema-side regex for fast-fail. addresses items gain pattern: ^[0-9a-fA-F:.]+/[0-9]{1,3}$ and gateway gains ^[0-9a-fA-F:.]+$ in schema.json — catches "forgot the /24" typos at MachineClass apply time (Omni-side JSON-schema gate) rather than deferring the failure to provision time.
• Route network picked by gateway family. buildAdditionalNICInterfacesPatch emits network: "::/0" when the gateway is IPv6, "0.0.0.0/0" when IPv4 — previously always IPv4 regardless of family, which Talos rejects at apply for IPv6 gateways.
• Duplicate-MAC reject in patch builder. buildAdditionalNICInterfacesPatch returns an error on duplicate deviceSelector.hardwareAddr — defense-in-depth against upstream MAC-collision-resolution bugs that would otherwise produce last-write-wins ambiguity in Talos.
• disk_size floor coverage. data_test.go pins MinRootDiskSizeGiB = 20: 19 rejected, 20 accepted, 5 rejected. Prevents a future refactor from silently restoring the undersized floor that caused control-plane DiskPressure GC loops during bootstrap.

Observability (parallel-review follow-up, same session)

• config_invalid and config_patch error-category buckets. categorizeError now routes MachineClass validation failures (wrapped via "invalid MachineClass config: %w") to config_invalid and every CreateConfigPatch failure to config_patch, so dashboards and alert routing can distinguish operator typos from hypervisor regressions and pinpoint which patch kind is failing. Previously both classes aliased into nic_invalid / unknown.
• truenas.config_patch.duration histogram. New Float64Histogram in internal/telemetry/metrics.go with a patch_kind label covers all five patch-emission RPCs (data-volumes, longhorn-ops, nic-mtu, nic-interfaces, advertised-subnets). Buckets mirror APICallDuration. Wraps every call via the new applyConfigPatch helper so timing is recorded on success and failure alike.
• Warn log on empty-MAC NIC skip. When AddNICWithConfig succeeds but TrueNAS returns no MAC attribute, the NIC is silently skipped from the patch — with this fix, a Warn log now fires (not just Debug) so SRE can correlate when a multi-homed VM comes up with fewer IPs than the operator declared.
• Aggregate breakdown in applied additional-NIC interfaces config patch Info log. Carries dhcp_nics, static_nics, gateway_nics counts. SRE can verify "is the static-address codepath actually firing?" during a rollout without enabling Debug everywhere.

Refactor (parallel-review follow-up, no behavior change)

• collectNICInterfaceConfigs pure helper. Extracts the per-NIC config+aggregate accumulation out of stepCreateVM's attach loop so it's unit-testable without a live provision.Context, TrueNAS client, or VM. Panics on len(nics) != len(attachedMACs) (caller bug, not recoverable). resolveNICDHCP now called once per NIC (was twice — hoist).
• applyConfigPatch(ctx, pctx, kind, requestID, data) helper. Centralizes patchName() + CreateConfigPatch + timing metric for all five patch kinds. AST-level static check updated: collectPatchNameKinds now recognizes both patchName(<kind>, ...) and applyConfigPatch(ctx, pctx, <kind>, ...) so the wiring registry stays accurate across the refactor.
• boolPtr test helper centralized. Moved from config_patch_test.go into testhelpers_test.go so new *_test.go files in the package don't duplicate it.

Tests (parallel-review follow-up)

• 23 new tests across config_patch_test.go, multinic_test.go, data_test.go, schema_drift_test.go, error_categorization_test.go: caller-seam wiring for collectNICInterfaceConfigs (5), extended validation (unspecified / multicast / loopback addresses, gateway non-unicast, family mismatch, not-on-link, multiple gateways, max caps), config_invalid/config_patch error categorization, schema-drift type pins, MinRootDiskSizeGiB floor pin.
• Patch-kind registry entry for nic-interfaces. TestStepCreateVM_WiresAllExpectedPatches now asserts the kind appears in a non-test source file, so a refactor that deletes the emission site (silently reverting the v0.15.5 fix) fails CI.

Documentation (parallel-review follow-up)

• Talos round-trip gap documented. docs/testing.md now carries a "Known test-coverage gaps" section explaining that Go-side unit tests pin the provider's understanding of Talos config shape but don't round-trip through config.NewFromBytes — so the v0.15.0 etcd-on-worker regression class can recur silently. Lists the two ways to close the gap (test-only dep on machinery/config, or -tags e2e cassette against talosctl apply).

Experimental

• Autoscaler Helm chart + operator docs (phase 4 of 4) — deploy/helm/omni-autoscaler/ ships a two-container chart (autoscaler subcommand + upstream cluster-autoscaler sidecar) with full values surface for Omni + TrueNAS credentials, per-cluster opt-in, experimental labels (bearbinary.com/experimental=true), and Recreate rollout strategy so no two autoscaler replicas are ever alive simultaneously. Chart renders cleanly against helm lint / helm template. docs/autoscaler.md is the operator guide: full annotation reference, deploy recipe, RBAC notes, observability, how to disable, and the known-limitations list (no scale-down, no scale-from-zero, no host-mem check until the system.mem_info wrapper lands). Feature is end-to-end deployable but still experimental — the combination of per-MachineClass opt-in, hard-gate-by-default capacity check, scale-down-disabled-at-two-layers, and single-replica rollout gives operators four independent layers to disable if something goes wrong.
• Autoscaler write path wired (phase 3d of 4) — internal/autoscaler/writer.go implements ScaleWriter.IncreaseMachineCount via safe.StateUpdateWithConflicts[*omni.MachineSet] with a live re-check of Max inside the mutator so a stale CurrentSize in the caller can't bypass the bound even if Omni's OCC would have let the write land. Server.NodeGroupIncreaseSize is the first mutating RPC: validates input (delta > 0, non-empty id) → re-runs Discover for a fresh current-size read → runs the capacity gate when one is wired → invokes the writer → logs the structured scale event. Errors map to the specific gRPC status codes cluster-autoscaler uses for scheduling decisions: ResourceExhausted on capacity breach or Max violation (CAS stops retrying), InvalidArgument on bad delta (no retry), NotFound on unknown group (prune from CAS cache), Unavailable on transient state errors (retry later). New AnnotationAutoscalePool + Config.Pool field let operators target a specific TrueNAS pool for capacity checks; falls back to Server.WithDefaultPool(…) (set from DEFAULT_POOL in the subcommand wiring). Subcommand now builds the full production dependency tree: Omni client, TrueNAS client (optional — absent means "capacity gate disabled" with a warn log, useful for dry-run deploys), Discoverer, ScaleWriter, and passes all into NewServer. TestServer_NodeGroupIncreaseSize_* covers happy path + the reject-with-status matrix (invalid delta, above max, unknown group); TestScaleWriter_* covers the write semantics including the live-recheck race guard. Correctness does not depend on a singleton lease — Omni's optimistic concurrency rejects stale writes on its own — but the chart still pins replicas: 1 to avoid wasted API calls.
• Autoscaler subcommand skeleton (phase 1 of 4) — omni-infra-provider-truenas autoscaler is the new experimental entry point for a Kubernetes cluster-autoscaler external-gRPC cloud provider, vendored from Justin Rothgar's omni-node-autoscaler PoC. Opt-in per MachineClass via bearbinary.com/autoscale-min / bearbinary.com/autoscale-max annotations on Omni MachineClass.omni.sidero.dev resources; a class without the annotations is not discovered. Optional bearbinary.com/autoscale-capacity-gate (hard or soft) controls whether TrueNAS pool/host-memory pressure blocks scale-up. Phase 1 scope is intentionally narrow: env-var config (OMNI_CLUSTER_NAME, AUTOSCALER_LISTEN_ADDRESS, AUTOSCALER_REFRESH_INTERVAL), annotation parser with full table-driven coverage, experimental startup banner, and a hold-open loop. No gRPC server and no Omni writes yet — those land in phases 2–3. The provisioner subcommand (no argv) is unchanged; existing Deployments bumping image tags see zero behavior drift.
• Autoscaler read-side gRPC handlers wired (phase 3c of 4) — Server now takes a *Discoverer and uses it to answer NodeGroups (returns the discovered []NodeGroup translated into proto form — id/minSize/maxSize/debug) and NodeGroupTargetSize (returns current MachineAllocation.MachineCount). NodeGroupForNode returns an explicit nil-NodeGroup "not ours" response through the experimental phase — scale-down is disabled at multiple layers, and the node→node-group mapping requires additional Omni state (MachineSetNode / ClusterMachine joins) that isn't worth the surface area while scale-down stays off. When the Server is booted without a Discoverer (early-phase testing, partial boot), handlers return Unimplemented with an operator-readable "discoverer missing" message rather than silently returning an empty list — silent-empty is indistinguishable from "cluster has no opted-in MachineSets" which is a legitimate steady state. 6 new TestServer_* cases cover the happy path (NodeGroups returns minSize/maxSize/debug), empty-cluster non-error, TargetSize found + NotFound, and the two configured/unconfigured NodeGroupForNode paths. Write handlers (NodeGroupIncreaseSize) still return Unimplemented; phase 3d enables them behind the singleton lease.
• Autoscaler MachineSet discovery (phase 3b of 4) — internal/autoscaler/discovery.go resolves one Omni cluster's autoscaler-managed node groups from a COSI state source. Discoverer.Discover enumerates MachineSets via state.WithLabelQuery(resource.LabelEqual(omni.LabelCluster, cluster)), skips control-plane MachineSets (no CP scaling support), rejects Unlimited allocations with a structured warning log, dereferences each worker's MachineClass by MachineAllocation.Name, parses the bearbinary.com/autoscale-* annotations via ParseMachineClassAutoscaleConfig, and returns a []NodeGroup the gRPC handlers will consume in phase 3c. Per-MachineSet failures (missing MachineClass, bad annotations, unsupported allocation type) log and skip the offending set — one misconfiguration never takes out scaling for the whole cluster. TestDiscover_* covers 9 scenarios against an inmem COSI state (the same pattern singleton tests use): empty cluster, other-cluster filtering, CP skip, non-opted-in skip, Unlimited reject, bad-annotation-skips-one, config propagation, missing-MachineClass skip, and out-of-bounds-still-included. The gRPC handlers still return Unimplemented — phase 3c wires discovery into NodeGroups / NodeGroupForNode / NodeGroupTargetSize; phase 3d enables the write path.
• Autoscaler gRPC server scaffold (phase 3a of 4) — internal/autoscaler/server.go wires the external-gRPC cluster-autoscaler cloud-provider contract via vendored protos under internal/autoscaler/proto/externalgrpc/ (Apache-2.0, from Kubernetes Autoscaler; see PROVENANCE.md for refresh workflow). Server boots cleanly, binds the listener, answers RPC calls, and returns codes.Unimplemented on every handler with an operator-readable message naming the next phase. Graceful shutdown drains on ctx cancel. google.golang.org/grpc promoted from indirect to direct in go.mod. TestServer_* exercises the full listen → dial → RPC → shutdown lifecycle against ephemeral ports so CI can't collide with the default :8086 or other tests. Every RPC handler is defined explicitly (rather than relying on the generated UnimplementedCloudProviderServer default) so the list of capabilities the autoscaler must support is literal and searchable — phases 3b (MachineSet discovery) and 3d (Omni writes) slot in one handler at a time.
• Autoscaler capacity gate (phase 2 of 4) — internal/autoscaler/capacity.go implements the TrueNAS-aware scale-up gate. Decision table: OutcomeAllowed (both thresholds pass or disabled), OutcomeDeniedHard (hard gate + threshold breached), OutcomeWarnedSoft (soft gate + threshold breached, still proceeds), OutcomeErrored (capacity query failed — fails closed). Pool-free-bytes check reads TrueNAS via the existing ListPools path (matches UI-reported values, accounts for ZFS parity/metadata overhead). Host-free-memory check is interface-only for this phase: TrueNASCapacityAdapter.HostFreeMemoryBytes returns ErrHostMemNotImplemented until a follow-up adds an internal/client wrapper for system.mem_info; operators who want to deploy now must set bearbinary.com/autoscale-min-host-mem-gib: "0" on annotated MachineClasses to disable the host-mem dimension until the wrapper lands. TestCheckCapacity covers all 11 decision-table branches; TestTrueNASCapacityAdapter_* pins the adapter behavior against a mock client. Still no gRPC server and no Omni writes — phase 3 wires both together behind a singleton lease.

Full Changelog: v0.15.5...v0.16.0