v0.16.2

Combined outcome of two SAST-driven hardening passes against v0.16.1 plus
a parallel-code-review enhancement plan that audited the first pass. Net
result: ~30 deepsec findings closed, six stdlib CVEs out of the binary, ten
new regression-guard tests pinning fixes that had been log-scrape-only,
and a maintainability counterweight pass that undid four micro-extractions
the lint chase created. No schema changes — existing MachineClass values
from v0.16.1 reconcile byte-identically.

Security

• GHA tag-name RCE on runner. Release workflow no longer interpolates ${{ github.ref }} into a shell command before signature verification. Attacker-controlled tag names can no longer execute on the runner.
• ISO TOFU bypass on property-read error. stepUploadISO's cache-hit path now treats a TOFU metadata read failure as cache-miss + redownload rather than silently accepting cached bytes. Regression test pins the new branch.
• Probe Bearer-leak via redirect. New shared internal/truenasrpc package strips Authorization on 3xx that crosses host. Probe and production share the same redirect policy now; previously only production stripped.
• isLocalOmniEndpoint userinfo + subdomain bypass. Host-equality tightened: rejects user:pass@host, evil.localhost, localhost.evil.com patterns that previously short-circuited the loopback check on hostile DNS.
• /healthz amplification. Reworked to a background-refresh model with a single in-flight check and rate-limited response. /healthz no longer fans out a TrueNAS RPC per request.
• ZFS passphrase trust-boundary doc. SECURITY.md now spells out that the provider does not handle dataset passphrases; operators must use the TrueNAS keystore.
• docs.yaml workflow least-privilege. Workflow-level permissions: block split into per-job blocks so the build job only gets contents: read and only deploy gets pages: write + id-token: write. A future workflow change touching the build steps can no longer accidentally write Pages content.
• Probe TLS verification on by default. scripts/verify-api-key-roles no longer hardcodes InsecureSkipVerify: true. Verification is enabled unless the operator opts out with TRUENAS_INSECURE_SKIP_VERIFY using the same strconv.ParseBool parser the production provider has always used ("1", "true", "T", etc.). Both the WebSocket dialer and the /_upload HTTP client pin MinVersion: TLS 1.2. Probe apiKey is wrapped in SecretString and zeroed after auth so a post-auth core dump cannot exfiltrate the key.
• Provider chart drops automounted SA token. Helm chart sets automountServiceAccountToken: false on the pod spec. The provider talks Omni gRPC + TrueNAS WebSocket and never reads the Kubernetes API — the default-mounted token had no legitimate consumer and only widened blast radius if the container were compromised.

Build

• Go toolchain 1.26.2 → 1.26.3. Closes six stdlib CVEs reachable from the binary: GO-2026-4918 (HTTP/2 SETTINGS_MAX_FRAME_SIZE infinite loop on outbound http.Client.Do — reachable from ISO upload path), GO-2026-4980 (html/template escaper bypass), GO-2026-4977 (mail.ParseAddress quadratic concat), plus three others. golang.org/x/{crypto,net,sys,text} indirects refreshed; net jumps v0.52.0 → v0.53.0.

Reliability

• Deprovision survives ctx cancellation without losing traces. internal/provisioner/deprovision.go zvol-cleanup and force-stop paths use context.WithoutCancel(ctx) instead of context.Background(). Cleanup still completes on shutdown but trace IDs and observability metadata propagate from the parent context. Same pattern applied to the autoscaler singleton-lease release.
• Autoscaler lease.Release respects a 5 s timeout. Matches the provisioner-side release path; pod shutdown can no longer hang on a slow Omni RPC. Trace IDs still propagate via context.WithoutCancel(baseCtx).
• fetchOwnedVM (nil, nil) sentinel replaced with errVMNotFound. Returning (*VM, error) where (nil, nil) means "treat as success" was a foot-gun; callers now distinguish via errors.Is(err, errVMNotFound).
• Test context leaks closed. deprovision_test.go and internal/autoscaler/server_test.go defer cancel() (or t.Cleanup(cancel)) immediately after context.WithCancel, so a test that returns early no longer strands the cancel func until GC.

Observability

• ISOHashMismatches gains a detection_path label so "factory rotated under the same URL" stays separate from "bytes tampered at rest" on the dashboard.
• New TOFU-marker counters + alert rules. truenas_iso_poison_marker_write_failed_total, truenas_iso_poison_marker_retries_total, truenas_iso_tofu_metadata_write_failed_total, plus three matching Prometheus alert rules. MANUAL CLEANUP REQUIRED no longer relies on log scraping.
• SingletonLeaseHeld gauge wired on the autoscaler. Previously only the provisioner side emitted it; autoscaler lease state is now visible on Grafana dashboards via truenas_singleton_lease_held{scope="autoscaler-<cluster>"}.
• provider_id attribute on graceful-shutdown counters. GracefulShutdownSuccess / GracefulShutdownTimeout carry provider_id so fleet dashboards can segment per provider.
• cleanupOrphanVMs::maybeDeleteOrphanVM records span.RecordError on StopVM failure — was previously only emitted on DeleteVM failure, so half-completed orphan teardown was invisible in trace UI.
• setIfPoisonable timer hygiene. Retries via time.NewTimer + Reset (no leaked timers on cold ctx-cancel) and returns the persisted-bool so the outer error mentions when the marker write itself failed.

Hardening

• internal/truenasrpc shared package closes the probe-vs-production drift flagged during review — the probe was missing the strict DNS allow-list and now inherits the production transport contract.
• verifyCachedISO extracted from stepUploadISO. Cache-hit path drops from 80 lines of nesting to a 12-line dispatch, sheds a redundant filesystem.stat RPC per hit, and adds a TOCTOU re-stat immediately before CDROM attach so replication firing mid-provision cannot slip tampered bytes through. Legacy cache entries re-record metadata on observation so the next hit upgrades to the full TOFU triple without forcing a redownload.
• /healthz checker rework. Background refresher with functional options (closes a test-only field-poke seam), every check wrapped in an OTel span, truenas_healthcheck_errors_total increments on every failed check including WS-level and context-deadline failures (previously only inside specific checker subpaths, leaving those failure modes invisible to the existing alert).
• autoscaler lease + capacity gate API shape. acquireAutoscalerLease returns (release func(), error) with a sentinel errAutoscalerLeaseShutdownDuringAcquire instead of a tri-state return that the caller had to re-decode. buildAutoscalerCapacityGate returns a CapacityGateBundle with an unconditional Close() (no-op when gate is disabled) so the caller can defer bundle.Close() without a nil-check dance.
• isoCacheRef struct + constructor. newISOCacheRef(dataset, path, imageID) builds the canonical struct with all three TOFU property names derived from imageID. verifyCachedISO, recordTOFUProperty, setIfPoisonable, and handleISOHashMismatch each take 4 args instead of 7–9. A future rename of a property prefix touches one line instead of twelve.
• validateDatasetPrefixSegments shared. Both Validate (root) and validateOneAdditionalDisk (per-disk) route through a single segment-splitter, parametrized on whether empty segments are rejected. The pre-existing rule-of-three drift (root rejected empty segments, per-disk silently skipped) is now explicit.
• pool.dataset.query literal de-duplication. Extracted methodPoolDatasetQuery + errPoolDatasetQueryFmt constants; replaced 7 + 5 inline usages.
• Helm chart ephemeral-storage. Provider values.yaml declares requests.ephemeral-storage: 64Mi + limits.ephemeral-storage: 256Mi. Kubelet can size the pod's writable layer + log buffers explicitly instead of falling back to BestEffort eviction.
• Dockerfile base image. Dropped the redundant :nonroot tag from the FROM line — the digest already pins the image bytes immutably. Same digest, same uid override (65534).

Regression-guard tests

• TestNodeGroupIncreaseSize_Gate{DeniedHard,Errored,SoftWarn,Allowed}_* — pin each capacity-gate Outcome arm to its gRPC status code + metric. Future edits that swap ResourceExhausted ↔ Unavailable, drop a recordScaleUpResult increment, or break soft-warn fall-through fail CI immediately.
• TestCleanupVM_DeleteRunsEvenAfterParentCancel — pins the context.WithoutCancel contract by asserting DeleteVM is called on a non-cancelled context after the parent ctx is cancelled mid-deprovision. A revert to cleanupCtx := ctx would silently break this and pass the existing "exits quickly" test.
• TestUploadFile_DataReaderErrorPropagates — pins that a failing source reader surfaces through writeUploadMultipart's pw.CloseWithError path within a bounded deadline, rather than hanging on the pipe.
• TestMaybeDeleteOrphanVM_LegacyVMNoRequestID_Skipped + TestMaybeDeleteOrphanVM_RequestIDNotInLiveSet_Deleted — regression-pin the v0.15.3 mass-delete fix (VMs with no parseable request-id MUST NOT be deleted) and the counter-positive (real orphans MUST be deleted) so the v0.15.0 incident pattern can't recur.
• Capacity-gate + lease helper smoke tests — TestBuildAutoscalerCapacityGate_HostUnset_ReturnsNilQuery, TestAcquireAutoscalerLease_Disabled_ReturnsNoopRelease, TestAcquireProviderLease_Disabled_ReturnsNilRelease, TestErrAutoscalerLeaseShutdownDuringAcquire_IsDistinguishable — pin the disabled-branch and sentinel-identity contracts the runAutoscaler caller relies on.
• internal/client.NewMockClientCtx — new ctx-aware mock transport that exposes ctx.Err() to test handlers. Required for the WithoutCancel contract pin; reusable for future tests that need to inspect cancellation behavior at the call site.

Maintainability

• Large-function refactors. stepCreateVM (provisioner core, previously cog-complexity 129) split into attachAdditionalDisks, attachPrimaryNIC, attachAdditionalNICs, applyNICConfigPatches, and applyAdvertisedSubnetsConfigPatch. stepUploadISO (cog 56) factored into downloadOrReuseISO, ensureISODatasets, downloadAndUploadISO, and handleISOHashMismatch. ensureZvol (cog 30) extracts the encrypted-existing recovery path into ensureEncryptedZvol. Data.Validate (cog 81) split into per-domain helpers. run() in main.go (cog 45) extracts buildLogger and acquireProviderLease. Provisioner cleanupVM (cog 23) extracts fetchOwnedVM and gracefullyStopVM. No behavior change in any.
• Maintainability counterweight. Four micro-extractions that the lint chase introduced were reverted after review: recordGracefulOutcome (4-line two-arm metric — back inline with the added provider_id label), attachOneAdditionalNIC (split one for-loop iteration in half — back inline), runShutdownFuncs (one-call helper wrapped in a one-call closure — back inline), and dialCleartextFallback (conflated TLS-error formatting with cleartext-redial — split honestly into the redial + warnCleartextFallback).
• singleton.Clock interface name preserved with a //nolint:revive and rationale comment — matches the uber-go/clock / benbjohnson/clock Go-ecosystem convention; renaming to "Nower" would cascade ~40 callsites for a stylistic lint with no value to readers.
• data.go map size hints. seenVolumeNames and seen pre-sized to known upper bounds.
• autoscaler/metrics.go::contains simplified from a hand-rolled case-insensitive byte loop to strings.Contains(ToLower(...), ToLower(...)). Allocates two strings per call; the comment now explicitly flags this as cold-path-only.
• Shell scripts. test-longhorn.sh's log() / fail() switched from ((PASS++)) (which returns exit status 1 the first time the counter is zero, masking helper failure as success) to PASS=$((PASS+1)); return 0. install-longhorn.sh / annotate-machineclass-autoscale.sh logger helpers assign $1 to a local msg.

Operator note

The probe TRUENAS_INSECURE_SKIP_VERIFY env var now uses the same strconv.ParseBool parser the production provider has always used. Operators whose .env had =true previously saw the probe verify TLS while production silently skipped (false sense of safety); both now honor the setting consistently. The production provider's behavior has not changed — only the probe was brought into alignment. No action required unless your .env deliberately relied on the old parser difference.

Full Changelog: v0.16.2-rc.3...v0.16.2