Skip to content

fix(kubernetes): co-locate RWO runtime workloads - #103

Merged
MarcStdt merged 1 commit into
masterfrom
codex/pin-rwo-runtime-workloads
Aug 9, 2026
Merged

fix(kubernetes): co-locate RWO runtime workloads#103
MarcStdt merged 1 commit into
masterfrom
codex/pin-rwo-runtime-workloads

Conversation

@MarcStdt

@MarcStdt MarcStdt commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Problem

One-shot runtime Jobs and the Developer Mode StatefulSet mount the same RWO PVC as the active deployment but had no node constraint. Kubernetes could schedule them on another node, producing a Multi-Attach error and permanently blocking the command queue.

Change

  • discover the node of the running pod that mounts the runtime PVC
  • pin procedure Jobs, helper Jobs, persistent procedures, and the Developer Mode workload to that node
  • leave workloads unconstrained when there is no active PVC consumer

Validation

  • go test ./internal/runtime/kubernetes
  • go test ./...
  • go test -race ./internal/runtime/kubernetes
  • production reproduction: Developer Mode init Job was scheduled away from an active RWO PVC and hit FailedAttachVolume

@druid-infra

Copy link
Copy Markdown
Contributor
Error: This repo is not allowlisted for Atlantis.

@MarcStdt
MarcStdt merged commit 51a518a into master Aug 9, 2026
10 checks passed
@MarcStdt
MarcStdt deleted the codex/pin-rwo-runtime-workloads branch August 9, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants