Repository navigation
This is v0.4.2, a patch release of v0.4.1. It makes the manager scale to large clustered
Providers, closes the gaps in how a clustered Provider's capacity is reserved, and updates
golang.org/x/net for two security advisories. It changes no API field's meaning and no gRPC
message. Although it is a patch release, it adds one optional VirtualMachine status field
and two manager flags, and rolling back to v0.4.1 is unsupported while a clustered VM holds a
resize reservation: read Upgrade first.
- Performance: running VMs are polled every 2 minutes (5 when powered off) instead of every
10 seconds, unchanged status is not rewritten, a provider connection is validated once instead
of on every reconcile, Provider VM counts and Host committed capacity are read from indexes
instead of full scans, a new VM is described at once after itsCreate, and a placement
decision allocates a fraction of what it did. - Admission safety (clustered Providers): a placement is released only on evidence, never on
a timer alone; an admitted resize-up is reserved durably before it is sent; an admission
inherited from an earlier manager is checked again before it is executed; a VM whose size
cannot be established is never counted at a guessed size. - Operations: the manager's Kubernetes client rate limits are configurable
(--client-qps,--client-burst; chartmanager.clientQPS,manager.clientBurst). The
defaults are unchanged (20 and 40). - Security:
golang.org/x/netv0.62.0 in every VirtRigaud Go module and binary (CVE-2026-78661,
CVE-2026-97045). Roll the providers as well as the manager to pick it up. - Compatibility: install the CRDs before the manager (the chart's hook does); do not roll
back while a clustered VM holds a resize reservation, and roll the manager and the chart
back together. See Upgrade. - Fault isolation and recovery: one VM's failures do not open its Provider's circuit breaker
and block the Provider's other VMs, a Provider that is really down still opens it, and the
vSphere provider renews a lost vCenter session on its own. This was found while qualifying
rc1 and fixed in rc2 (#472). Roll
the manager and the providers together. See Fault isolation and recovery.
Fault isolation and recovery
Connection caching (F3, see Performance) removed the Validate call that, in
v0.4.1, preceded every provider call and reset the per-Provider circuit breaker. Qualifying
v0.4.2-rc1 on a lab showed that without that reset a few failing VMs could open the breaker for
every VM of their Provider (#472).
v0.4.2-rc2 fixed it, together with a vSphere session-recovery gap the fix would otherwise have
exposed, and was qualified on the same lab (see Validation). No release before
v0.4.2-rc1 had the defect, and v0.4.2 ships the fix.
-
A Provider's circuit breaker no longer opens for failures of individual VMs
(#472). One breaker guards every VM
of a Provider. In v0.4.2-rc1 a cached provider client was reused without aValidate(F3),
so nothing reset the breaker's failure count between VM calls any more: a few VMs whoseDescribe,PowerorDeletekept failing in the hypervisor (a
locked VM, a per-VM permission fault, a Proxmox API error for one VM) opened it after 10
failures. While it was open, every VM of that Provider was refused, healthy ones included:
they wentReady=False("cannot reach this VirtualMachine's Provider") and their power,
reconfigure, delete, create and clone calls waited. v0.4.1 did not behave this way.The manager now counts a failure toward the breaker by whether the provider's gRPC server
answered it. An answer withInternalorUnknown(how the libvirt, vSphere and Proxmox
providers report one VM's backend error) proves the provider and its connection work, and is
not counted. Failures the provider did not answer (connection refused or dropped, the
provider pod gone, no answer before the call's deadline, a proxy's non-gRPC error page) and
answeredUnavailableorDeadlineExceededstill count, so a Provider that is down or
overloaded still opens the breaker, fails fast while it is open, and recovers. The reasons
that were already excluded (HOST_UNAVAILABLE,VM_OPERATION_FAILED,VM_DISK_CHECK_FAILED,
OUTCOME_UNKNOWN_RETRY, the image-prepare reasons) are unchanged. Only the breaker's
accounting changed: the errors VMs report, and how an unknown outcome or a request that was
never sent is handled, are the same. In a simulated test with the default breaker settings
(10 failures, 60 s reset), 1 failing VM next to 6 healthy ones kept the breaker open or
half-open 69% of the time with the rc1 code and 70 of 100 healthy lifecycle calls were refused;
3 failing next to 6 healthy, 90% of the time; 40 failing next to 6, and 20 next to 60, all of the
time. With the fix the breaker never opened in any of these mixes and no healthy call was refused; a provider connection is still
validated once and reused.A Provider that is really down is still handled as before: when its gRPC server does not
answer, the breaker opens after 10 failures, refuses calls at once without connecting while it
is open, lets 3 trial calls through after 60 s, and closes when they succeed; the healthy VMs
are then managed again on one new connection. Some failures are still classified by their code
rather than by the VM they concern: see
What the circuit breaker still counts.The fix is in the manager, so it also applies to providers on older images. Roll the vSphere
providers with the manager (below). -
Proxmox: "exists elsewhere in the cluster" is VM-scoped. A
DescribeorDeleteof a VM
that is not on the node asked but exists on another node of the cluster answered a plain
Unavailable, which the breaker counts, so one misplaced VM retried by its finalizer could
open it. The answer now also carriesVM_OPERATION_FAILED; its code and message are
unchanged. -
vSphere: the provider logs in again after a vCenter outage on its own. The provider keeps
its vCenter session alive with a keepalive that probes it every 5 minutes and logs in again
when the probe fails. If that login also failed (vCenter was unreachable at the tick, as in a
vCenter restart or maintenance longer than 5 minutes), the keepalive stopped for good, and only
the provider'sValidatelogged in again. Since F3 the manager callsValidateonly when it
connects to a provider or the Provider's connection or circuit breaker reports trouble, and
with the fix above a provider answering "not authenticated" for every VM no longer opens the
breaker, so a vSphere provider whose session was lost this way would have answered every call
with that error until its pod restarted. The keepalive now logs the failed login and tries
again at the next tick, so the provider recovers within 5 minutes (the keepalive interval) of
vCenter accepting logins again; until then, calls fail with the authentication error and each VM
is retried on its own schedule. Roll the vSphere providers together with the manager: a vSphere
provider older than v0.4.2-rc2 behind a v0.4.2 manager does not recover from this on its own
(restart its pod).
Performance
Two kinds of measurement back this section, and they are not production measurements:
- End-to-end (Spike 1B): a disposable kind cluster with a fake clustered provider
(1,000 Hosts and 10,000 VMs at most; method, environment and raw results in
docs/performance/spike-1b-control-plane-baseline.md).
It measures the control plane only: the fake provider answers at once, so these figures say
nothing about how fast a real hypervisor creates, changes or starts VMs. - Microbenchmarks (Spike 1A): Go benchmarks of single functions on one workstation
(docs/performance/spike-1a-scheduler-baseline.md).
Your numbers depend on your API server, hypervisors and manager resources.
- Steady-state polling (F1/F2, end-to-end). Every running VM was reconciled every 10 seconds instead of
every 2 minutes, because the requeue interval compared the provider's power state with
vSphere's raw strings; each reconcile also rewrote the VM's status. Both are fixed: a status
write that would store nothing is skipped. Spike 1B smoke test at 1,000 VMs with the chart
defaults: steady VM status writes 20/s → 0, manager API requests 27.2/s → 7.1/s, and a 100-VM
burst reachedReadyat p50 154 s → 21 s. - Provider connections (F3). A cached provider client is reused without a
ValidateRPC while
its Provider's configuration (UID, generation, type, endpoint, TLS Secret version), its
connection and its circuit breaker are unchanged. This removes one provider RPC per VM
reconcile (on libvirt, a call to the host). A changed configuration, a failing connection or an
open breaker validates again. - Full scans (F4/F5, microbenchmarks).
Provider.status.connectedVMsis counted through a field index
instead of a deep copy of everyVirtualMachinein the cluster, and the Provider controller no
longer reconciles on every Host heartbeat (a 60-second resync replaces that trigger for
clustered Providers). A Host's committed-capacity gauges and its in-use check read the VMs
placed on that host from a per-host index. In benchmarks at 100,000 VMs on one Provider, the
count fell from 245 ms to 77 ms per call and the per-heartbeat read from 116 ms to 46 µs. - Post-create readiness (F7, end-to-end). A new VM is described as soon as its
Createis recorded,
instead of after a fixed 5 seconds. With 1,000 Hosts and 10,000 VMs, a 10-VM burst reaches
Readyat p50 1.3 s instead of 6.3 s. In a 100-VM burst the median stays at 20.5 s, because
VMs queue behind the Provider's serialized placement decisions; the p99 falls from 24.1–24.7 s
to 22.9 s. - Placement decisions (C′, microbenchmarks). A decision no longer copies every
VirtualMachine; it reads the
informer's index directly and sums capacity per host. In benchmarks, a decision at 100 hosts
and 10,000 VMs takes 7.9 ms instead of 18.4 ms and allocates 435 KB instead of 16.9 MB, and the
Provider's lock is held for a p50 of 7.7 ms instead of 20.4 ms in a contention benchmark. The
decisions are identical (equivalence tests). A Provider's decisions are still serialized by
its lock.
Admission safety
These apply to topology: cluster Providers. Single-host Providers are unchanged. Details:
docs/clustered-provider-inventory.md and ADR-0007's
amendments.
- A placement is released on evidence, never on a timer alone (G1/G2/G4). An in-memory
placement assumption expired after two minutes even if itspendingHostwrite had landed and
the cache did not show it yet. ApendingHostwrite that timed out after landing could also be
followed by a second placement of the same VM. Either let a competing VM overbook the host. A
pick is now released only when the cache shows its record, the API server rejected its write,
or an authoritative read resolves an ambiguous write. While a write is unresolved the VM is
not placed again. - A resize-up is reserved durably before it is sent (G3). The admitted footprint is written to
the newstatus.placement.resizeReservation(a resourceVersion-checked status write) before
Reconfigureis sent. If the API server does not keep the field (an older CRD), the resize is
held withReconfiguring=False/PlacementErrorinstead of being sent unreserved. The same write
setsReconfiguring=True/ResizeReserved, so a success left by an earlier resize can never
release the new reservation. The reservation counts until the provider has confirmed a
Reconfiguresent under it:Reconfiguring=Falsewith reasonReconcileSuccess, and a
Describethat reports the VM's vCPUs and memory maximum. Any other outcome keeps it: failed,
unanswered or never recorded, pending a restart, asynchronous, refused before it was sent, or
a reason the manager does not recognise. - Inherited admissions are checked before they are executed (C1). Leader election does not
fence writes, so apendingHostor reservation write sent by a previous leader can land after
the new leader gave the capacity to another VM. A durable admission this manager instance did
not record itself (a pending create, a pending clone target, or a resize reservation) is
checked once against the host's committed capacity beforeCreate,CloneorReconfigure
is sent. One that no longer fits is held withInheritedAdmissionOverCapacity: its record is
kept and keeps counting, and nothing is cleared automatically. Bound VMs (describe, power,
delete) are never held. - A VM whose size cannot be established is never counted at a guessed size. A placed VM whose
VMClass is deleted, or no longer granted to its namespace, used to count at the minimum
footprint (1 vCPU, 128 MiB). It now counts from its recorded size (status.currentResources,
orstatus.placement.pendingResourceswhile its create is pending). While no memory ceiling
is recorded for it, it counts at the memory hot-add ceiling of that size. When nothing records
its size (a VM bound by a build that did not record it), capacity admissions on its Provider
are refused and retried until the VMClass or the grant is restored. The manager logs the VM's name.
Operations
-
Configurable client rate limits. The manager's client-side Kubernetes API rate limits are
set by--client-qps(requests per second; default 20) and--client-burst(default 40);
in the chart,manager.clientQPSandmanager.clientBurst. Zero, negative and non-numeric
values fail the chart render, and the manager refuses to start with them. Each of the
manager's REST clients (one per resource type) has its own bucket with these limits. They are
client-side limits, not API-server limits. The effective values are logged once at startup.
They take effect at a manager restart.Measured once, in the Spike 1B environment (1,000 Hosts, 10,000 VMs, manager limits 500m CPU
and 512 MiB, a 100-VM burst): at QPS 100 / burst 40 instead of 20 / 40,Readyp50 fell from
20.52 s to 7.86 s and throughput rose from 4.3 to 12.2 VMs/s (2.8×, not 5×). Manager CPU rose
from 0.19 to 0.34 cores, with 36% of periods throttled. Placement-lock waits appeared (p50
177 ms), as did 6 retried conflicts; there were no API server 429s, flow-control rejections or
5xx. This is not a recommendation for production clusters; see
Kubernetes API client rate limits
for what raising them costs.
Compatibility
- CRD:
VirtualMachinegainsstatus.placement.resizeReservation(additive, optional). No
other schema change. Existing objects need no migration. - Upgrade order: CRDs, then the manager, then the providers, as before. With an older CRD the
new manager fails readiness, naming the missing field. - Chart: the manager Deployment now passes
--client-qps=20 --client-burst=40by default (one
manager rollout, same limits). Upgrade withhelm upgrade --reset-then-reuse-values, not
--reuse-values:--reuse-valueskeeps the v0.4.1 chart's image tags, so the manager is not
upgraded while the new CRDs are. With--reuse-valuesand the image tags set, the manager
starts without the flags and applies its defaults. Both options keep image tags you set
yourself: checkhelm get valuesand override any pinned v0.4.1 tag. - Manager image: the manager is now built from the
cmd/managerpackage, not a single file.
This only matters if you build the image yourself frombuild/Dockerfile.manager. - gRPC contract and providers: unchanged; providers need no configuration change. Their
images are rebuilt with thegolang.org/x/netupdate (see Security) and the vSphere
and Proxmox providers carry fixes (see Fault isolation and recovery),
so roll them right after the manager; roll the vSphere providers in the same change. - ImagePrepare from an older manager is still served (ADR-0009 D7). An identity-less request
from a manager older than v0.4.0 is served in deprecated legacy mode, logged and counted in
virtrigaud_provider_image_prepare_legacy_requests_total, as in v0.4.0 and v0.4.1. The
refusal that the v0.4.1 documentation announced for "the release after v0.4.1" is deferred to
a later release, not chosen yet. TheWARNline now endsrefused in a later release
(it saidrefused from the next release); the counter is unchanged.
Security
golang.org/x/netv0.62.0 in the root,sdk,protoandtest/proxmox-grpc-testGo
modules (it was v0.60.0; the test module v0.48.0). It fixes CVE-2026-78661
(GO-2026-6860) and CVE-2026-97045 (GO-2026-6866). govulncheck reported both against the
dependency only: no vulnerable function was reachable from VirtRigaud code. The update also
movesgolang.org/x/cryptoto v0.58.0,x/sysv0.49.0,x/termv0.47.0,x/textv0.43.0,
x/syncv0.24.0 andx/toolsv0.51.0, which the update requires; nothing else changed.- Not every advisory is resolved. Two known findings remain, neither of them reachable from
VirtRigaud code according to govulncheck:- GO-2026-6094 (GHSA-gcjh-h69q-9w9g) in
github.com/google/cel-gov0.22.0: the vulnerable
package is imported (throughk8s.io/apiserver), but the affected functions are not
called. The fixed version (v0.30.0) is newer than the one the Kubernetes and
controller-runtime libraries in use require; it is not part of this release. - GO-2026-5932, the deprecated
golang.org/x/crypto/openpgppackage: no VirtRigaud build
imports it, and there is no fixed version.
- GO-2026-6094 (GHSA-gcjh-h69q-9w9g) in
- Container images. The manager image has no HIGH or CRITICAL finding. The provider images
are based on Debian 12.15 and carry HIGH and CRITICAL operating-system package advisories (83
distinct CVEs, 4 of them CRITICAL: in GLib, libxml2, zlib's MiniZip and the Proxmox image's
OpenSSH client) for which Debian 12 has no fixed package: they are marked affected, fix
deferred or will not fix. The release scan ignores unfixed findings, so these do not fail it;
that is not a statement that they are harmless. Most are in packages the providers do not use
at run time (for example QEMU device emulation in the vSphere image, which uses only
qemu-img). The chart's CRD-hook image (kubectl) now ships the upstreamkubectl1.35.10
binary (was 1.35.9), built with Go 1.26.10, which fixes four HIGH Go standard-library
advisories (CVE-2026-78667, CVE-2026-78669, CVE-2026-97031, CVE-2026-111943). Upstream still
builds it withgolang.org/x/netv0.56.0 andx/textv0.39.0: Trivy reports two HIGH
advisories (CVE-2026-78669 inx/net, CVE-2026-56851 inx/text), and also reports CVE-2026-78661
and CVE-2026-97045, not yet rated, against thatx/net. No upstreamkubectlrelease is built with the fixed
versions yet. The image runs only duringhelm installandhelm upgrade, and talks only to
the cluster's API server.
Known limitations
- No provider-side fencing. A paused previous leader that resumes can still act on an
answer it receives before it notices it lost its lease. The inherited-admission check stops the
new leader from executing an admission that no longer fits, but it does not stop the old one.
Restarting a provider pod does not cancel an operation it accepted. Releasing a held admission
is an administrator decision that requires positive proof
(recovery procedure). - Leader-transition window. A
pendingHostwrite the old leader sent but never saw answered
can be stored after the new leader's cache synced. Until the new leader's cache shows it, the
host can be overbooked. That admission is not executed if it no longer fits. - Future asynchronous clustered providers must give
eachReconfigurea recoverable operation identity and accept a repeated request
idempotently before they are admitted for clustered resizes. The libvirt provider's
Reconfigureis synchronous and declarative. - A withdrawn resize can keep its reservation indefinitely. Setting the spec back to the
recorded size sends nothing, so a reservation whose lastReconfiguringreason is
ValidationError,InsufficientHostCapacity,InheritedAdmissionOverCapacityor
ResizeReserved(an outcome a stopped manager never recorded) is not released. No timer
releases it, because expiring it could free room aReconfiguresent earlier may still use. It
stays, and keeps counting, until the next resize the provider confirms or the VM is deleted:
this over-reserves capacity, it never overbooks a host. Do not remove it by hand without
proof that no earlierReconfigurecan still execute
(details).
A safe recovery mechanism is tracked in
#468. - A withdrawn refused resize keeps its
Reconfiguringcondition
(#473). After a resize refused with
InsufficientHostCapacityis withdrawn, the condition stays until the next resize. Nothing is
reserved or retried, and the phase,Readyand the accounting are not affected. This is also
the case in v0.4.1. - Rolling back to v0.4.1 is unsupported while a clustered VM holds a resize reservation, and
the manager and the chart must be rolled back together. See
Rolling back to v0.4.1. - Clustered libvirt Providers (
topology: cluster) remain experimental, as in v0.4.0 and
v0.4.1 (docs/clustered-provider-inventory.md). The
admission and accounting changes in this release apply to them only. - Placement is still serialized per Provider. With the client rate limit raised, the
per-Provider placement step, slowed by CPU throttling, was the next limit measured. - The client rate limits apply to every resource type alike and change only at a restart.
- What the circuit breaker still counts. The classification fixed in rc2
(#472) goes by the error's code.
A failure the provider answered withUnavailableorDeadlineExceededstill counts, also
when it concerns one VM: for example the Proxmox provider's answer that it cannot read one VM's
configuration to check its owner, the vSphere provider's answers about a migration disk or a
snapshot it cannot confirm, a Proxmox provider older than v0.4.2-rc2 answering "exists elsewhere in the
cluster", and a call for one VM that takes longer than the manager's deadline. Enough of these
in a row, with no successful call in between, still open the breaker for the whole Provider.
The other way round, a Provider whose gRPC server answers but whose hypervisor is unreachable
for every VM no longer opens the breaker (as in v0.4.1): each VM reports the provider's error
and is retried on its own schedule, and the provider answers those calls at once. The same holds
for other failures that affect every VM of a Provider but that the provider still answers as
InternalorUnknown(lost hypervisor credentials, a shared backend subsystem failing, the
provider process running out of resources): they are retried per VM rather than failed fast.
The libvirt and Proxmox providers recover from these on their own (libvirt dials its SSH
connection again when a command finds it dead; Proxmox authenticates every request with its API
token), and the vSphere
provider renews its session itself (see Fault isolation and recovery).
Validation
Both release candidates were qualified on the same non-production lab: Kubernetes 1.34, a
clustered libvirt Provider with a two-host pool, a single-host libvirt Provider, a Proxmox VE
Provider and vSphere Providers. v0.4.2 ships the code of v0.4.2-rc2. The lab was not a production
environment and did not test scale; the figures in Performance come from the
measurements described there.
v0.4.2-rc1: functional qualification, upgrading from v0.4.1.
- Upgrade. With the published chart and
helm upgrade --reset-then-reuse-values(the pinned
manager tag overridden). The chart's CRD hook applied the CRDs before the manager rolled. The
providers were rolled throughspec.runtime.image. Existing VMs were unchanged (identity,
placement, recorded resources, addresses, conditions) and their guests stayed reachable. - Polling and status writes. Running VMs were polled every 2 minutes and powered-off VMs
every 5. Status writes for the same VMs fell from 97 in 120 seconds (v0.4.1) to 32 in 300
seconds. - Placement. The target host was recorded before
Create. A VM that exceeded the free
capacity was heldUnschedulablewith noCreatesent, and was admitted on the documented
backoff once capacity was freed. - Resize up. The reservation was stored with
ResizeReservedbeforeReconfigure, was kept
throughRestartRequired, and was released only after the confirmed result that follows a power
cycle; accounting used the confirmed size. A resize that exceeded the free capacity was refused
(InsufficientHostCapacity) with no reservation and noReconfigure. A shrink of a running
clustered VM waited for power-off. - Clones. Clones were routed to the source's host, held while that host was full, counted
while pending, and executed exactly once (libvirt requires the source to be powered off). - Migration. One migration from a single-host libvirt Provider into the clustered pool, over
NFS staging. - Manager restarts. Two controlled manager restarts, one with an outstanding resize
reservation and one with a pending clone: both were kept, counted and completed once. - Rollback. A
helm rollbackto v0.4.1 with no outstanding reservation, then an upgrade
again. - Client rate limits. No client-side throttling at the default 20 and 40, and no reconcile
errors.
v0.4.2-rc2: fault isolation, upgrading from rc1.
- Upgrade. The manager and six Providers were upgraded from rc1 with the published chart and
images; the running image digests matched the release. Existing VMs, Hosts and object counts
were unchanged, every Provider was ready, no lifecycle call was sent for an existing VM, and no
reservation appeared. - Fault isolation. On a single-host libvirt Provider, a disposable VM's own disk was made
unreadable, so every power-on of that VM failed in libvirt. Over 16.5 minutes the provider
answered 159 of those calls with gRPCUnknown(rc1 opened the breaker after 10). The
Provider's breaker stayed closed. Healthy VMs on the same Provider stayedReady, and their
Describe, power off and on, and a new VM's provisioning succeeded. The manager did not
validate or redial the provider connection again, and healthy VMs wrote status at the polling
rate. The failing VM reported its own error and recovered on its own once its disk was restored. - The original symptom. A Provider whose Proxmox backend was unreachable had its breaker open
under rc1, and its VM'sReadymessage alternated between two texts. Under rc2 the breaker was
closed and that VM wrote no status update in 16 minutes. - vSphere. Both vSphere Providers validated (logged in), described and listed VMs normally.
- Manager restart. A new leader was elected in 21 seconds; all VMs and Hosts were unchanged,
and the new manager sent only reads and oneValidateper Provider.
Not exercised in the lab: other migration topologies, recovery of a pending Create across a
manager restart, an inherited admission that no longer fits, a genuine Provider outage (no
disposable Provider was available), and vSphere session recovery after a long vCenter outage (the
vCenter is shared). Automated tests cover the last four: the breaker opens for an unreachable
Provider, fails fast without connecting, and recovers through its half-open trial on one new
connection; the vSphere keepalive keeps retrying after a failed login.
Upgrade
Read Upgrading from v0.4.1 to v0.4.2
before upgrading. It covers CRD ordering, the new manager's re-check of inherited admissions,
the client rate limits, and rolling back. A helm rollback can succeed and still be unsafe
while any clustered VM holds status.placement.resizeReservation: the v0.4.1 manager drops the
field and counts the VM at its recorded size. Roll the manager and the chart back together; a
v0.4.1 manager does not start with the new flags. Roll the providers too, by setting
spec.runtime.image on each Provider (Helm does not change it): they carry the
golang.org/x/net update. Then check the images that actually run, as the upgrade guide shows.
From v0.4.2-rc1 or v0.4.2-rc2, only the images change: roll the manager and every provider
to v0.4.2. The CRDs are the same as both candidates'. See
Upgrading from a v0.4.2 release candidate.
Changelog
The complete, dated changelog is in CHANGELOG.md.