Pinned kindest/node v1.34.0 carries the kubelet DRA idle deadlock (kubernetes#133926, fixed in v1.34.2)

[nicknikolakakis]

What happened?

The pinned workload-cluster node image kindest/node:v1.34.0 (e2e/run.sh#L20) ships a kubelet with a DRA deadlock: kubernetes/kubernetes#133926, fixed in v1.34.2 via kubernetes/kubernetes#133934.

Once the kubelet-to-DRA-driver gRPC connection sits idle for ~30 minutes, the next pod that references a ResourceClaim hangs in ContainerCreating forever:

• pod has a single Scheduled event, nothing else, no kubelet errors
• the ResourceClaim is allocated,reserved
• the DRA driver only ever logs numClaims=0 health probes, NodePrepareResources never arrives

A goroutine dump from the node (kubectl get --raw /api/v1/nodes/<node>/proxy/debug/pprof/goroutine?debug=1) shows the deadlock: the pod worker blocked in dra.PrepareResources -> NodePrepareResources -> grpc idle.ExitIdleMode, while http2Client.Close -> monitoredPlugin.HandleConn (dra_plugin_manager.go:118) -> Connect -> ExitIdleMode holds the other side.

The nasty part: CI never hits this (clusters never idle 30 minutes), while any long-lived dev cluster always does. Overnight cluster, first ModelDeployment of the morning, engine pods stuck with zero diagnostics.

How can we reproduce it?

1. Create a kind cluster with the pinned image from e2e/run.sh
2. Install a DRA driver (I used dra-example-driver; the deadlock is in the kubelet, not the driver)
3. Schedule one pod with a ResourceClaim so the kubelet opens the driver connection, then leave the cluster idle for 30+ minutes
4. Create a ModelDeployment (or any pod referencing a ResourceClaim): it sticks in ContainerCreating with only a Scheduled event

Workaround: docker exec <node> systemctl restart kubelet, pods run within ~75s.

Suggested fix: bump the pin to a v1.34.2+ image. The open question is the containerd side: the current pin was introduced by #315 because newer kind images' containerd "has a regression that fails Modelplane", but the PR does not record what actually breaks. kindest/node:v1.34.8 ships containerd 2.3.1 and I have not tested it against Modelplane. If someone remembers the concrete failure from #315, that would tell us whether a straight bump is safe or whether the containerd issue needs solving first.

Full writeup with the goroutine evidence: https://srekubecraft.io/posts/modelplane-fleet-inference/ (demo: https://github.com/nicknikolakakis/srekubecraft-demo/tree/main/modelplane-fleet)

What environment did it happen in?

Modelplane version: v0.3.1
Crossplane: v2.4.0
Kubernetes: kind v0.32.0, kindest/node:v1.34.0@sha256:7416a61b42b1662ca6ca89f02028ac133a309a2a30ba309614e8ec94d976dc5a (containerd 2.1.3)
DRA driver: kubernetes-sigs/dra-example-driver (synthetic gpu.example.com devices)
Providers: provider-helm, provider-kubernetes (upbound org, as shipped with v0.3.1)
Platform: Docker Desktop on macOS (arm64)