v0.10.13

Upgrade notes

• Cache-backed pods roll once.
  • The model-downloader init script changed (#2276) for every cache-backed serving pod, prefetch Job and uncached multi-file pod.
  • Their pod templates change, so each rolls once on upgrade.
  • On installs with --default-fsgroup=0 (the OpenShift preset), the model-cache-prep command also changed (#2273).
  • Installs with fsGroup enabled (the default) see no prep change.
• Cache root owner on the OpenShift preset (#2273).
  • With --default-fsgroup=0, model-cache-prep now chowns the cache root to <uid>:0 mode 770, instead of 100:100 mode 770.
  • <uid> is the downloader's real uid: the InferenceService's runAsUser, else the new --init-container-user flag (chart: controllerManager.initContainer.user, default 101).
  • A root prepared by an older operator is re-owned on the next prep run.
  • Files under the root keep their owners.
  • Anything that relied on uid 100 owning the root loses access. Set --init-container-user=100 to keep that owner.
  • If you mirror a different --init-container-image, set --init-container-user to the uid it runs as.
• FUSE caches (#2276).
  • On a FUSE cache (for example BlobFuse), a serving pod no longer stages single-file models itself. It uses a warm cache, and otherwise reports ModelCacheLockUnsupported. Set spec.modelCache.prefetch: true so a prefetch stager stages them.
  • Prefetch stagers for the same cache directory and claim now run one at a time. A waiting Model shows PrefetchWaiting=True.
• multiNode + autoscaling (#2272).
  • spec.autoscaling on a spec.multiNode InferenceService no longer creates a HorizontalPodAutoscaler.
  • An HPA the service created earlier is deleted.
  • AutoscalingReady=False/UnsupportedForMultiNode reports it.
• Helm upgrades. helm upgrade --reuse-values from 0.10.12 is safe (#2286). The new controllerManager.initContainer.user value is optional: left unset, the operator uses 101, the uid of the default curlimages/curl init image.
• Installing CRDs with kubectl (#2275). The InferenceService CRD is now too large for a plain kubectl apply. Use kubectl apply --server-side, as make install and the chart README now do. Helm installs are unaffected.
• ModelRouter header timeouts answer 504 (#2284).
  • A request whose backend misses the response-header deadline (rule.timeout, backend.timeout or spec.proxy.responseHeaderTimeout) now gets 504 instead of 502, on rules that are not fail-closed. Fail-closed rules keep their 503.
  • The router's audit line and llmkube_router_requests_total{outcome} gain client_cancelled, upstream_error and timeout. Streams a client abandoned or a backend broke off were previously counted as ok_stream; alerts or dashboards that read outcome should expect the new values.
• ModelRouter Service targetPort (#2280). A router with spec.endpoint.port other than 8080 now forwards to the proxy's port 8080; before, its Service pointed at a port nothing listened on. Routers on the default port see no change.

0.10.13 (2026-10-28)

Features

• agent: add the llmkube-metal-agent stop-engines command (2cd2ec9)
• agent: keep Metal engines serving across metal-agent restarts and adopt them on startup (refs #2079) (2cd2ec9)
• release: sign and notarize the macOS release binaries (#2233) (f5ff962)
• router: podLabels and podAnnotations for router-proxy pods (#2285) (a736e18)
• router: publish per-backend modalities and context length on /v1/models (#2249) (2887f54)

Bug Fixes

• apply CRDs server-side in make install and make deploy (#2275) (edaa4f7)
• chart: leave --init-container-user off when its value is unset (#2286) (ac541db)
• cli: pin only single-writer claims and make the cache pod image configurable (#2178) (2fe250b)
• controller: annotate router-proxy pods for scraping under --emit-scrape-annotations (#2268) (07fb7f4)
• controller: chown the cache root to the init container's uid when fsGroup is off (#2273) (c9d86ad)
• controller: create no HPA for multiNode services (#2272) (c8d6f76)
• controller: delete FUSE stager Jobs in the foreground without the feature-gated field (#2269) (e439f60)
• controller: make FUSE model-cache staging safe across stagers and serving pods (#2276) (6c7c1fe)
• controller: point the router-proxy Service at the port the proxy listens on (#2280) (1c69118)
• controller: run the stall-probe tests on a fake clock (#2267) (661856a)
• router: tell client cancellations, upstream errors and timeouts apart (#2284) (3a2ea74)
• use vLLM's real TurboQuant KV cache types in the sample and docs (#2270) (3b4ffb6)

Documentation

• upgrade: metal-agent engines now keep running when the agent stops and the next agent adopts them. To roll back to an older agent, run launchctl bootout gui/$(id -u)/com.llmkube.metal-agent and then llmkube-metal-agent stop-engines with the current binary before installing the older one, or every model is loaded twice. make uninstall-metal-agent stops the engines for you. Adopted engines keep the flags they started with; run stop-engines after the bootout to apply new defaults. (2cd2ec9)