Operator: allow pod scheduling (nodeSelector/tolerations/affinity) on the proxy Deployment via MCPServer resourceOverrides

[talsh-oasis]

Use case

We run MCP servers via the operator on EKS + Karpenter. To cut provisioning cold-start latency we keep a dedicated pre-warmed node pool, tainted workload-class=mcp-warm:NoSchedule, and steer the MCP server pod onto it using spec.podTemplateSpec (a matching toleration + nodeAffinity). That works well for the server.

But each MCPServer also spawns the proxy (proxyrunner) Deployment, and there's no way to give the proxy pod the same scheduling. resourceOverrides.proxyDeployment currently supports only annotations/labels/env/imagePullSecrets — no nodeSelector/tolerations/affinity. So the proxy is repelled by the warm-pool taint, always lands on a cold general node, and forces a fresh node boot (~40s) that dominates provisioning latency and trips our readiness watch. Net: the server is warm but the proxy is cold, so we hit the cold-start anyway.

What we need

A way to set nodeSelector, tolerations, and affinity on the proxy Deployment via the MCPServer CR — matching the capability spec.podTemplateSpec already gives the server pod. Most naturally, extend resourceOverrides.proxyDeployment with those fields (or a podTemplateSpec).

Why it looks tractable

The runtime already supports it — pkg/container/kubernetes/client.go applies a full pod-template patch to the proxy Deployment (applyPodTemplatePatch(..., options.K8sPodTemplatePatch)). The gap is only the operator surface: ProxyDeploymentOverrides in cmd/thv-operator/api/v1beta1/mcpserver_types.go doesn't expose scheduling, and the operator doesn't feed it into that patch. So the change would be: add the fields → wire them into the proxy pod-template patch → make manifests → extend mcpserver_resource_overrides_test.go.

Happy to contribute the PR if this direction works for you.

[ChrisJBurns]

This sounds like a good change, @talsh-oasis. Thanks for offering to pick it up, and for tracing the plumbing before filing. Your read matches the code: applyPodTemplatePatch already runs against the proxy Deployment, and the gap is only that ProxyDeploymentOverrides does not expose scheduling.

The approach you described works for us, with two scoping notes for a first PR:

• Prefer explicit nodeSelector, tolerations, and affinity fields on resourceOverrides.proxyDeployment over a full podTemplateSpec. A second broad override surface on the proxy would overlap with the metadata overrides already there, and the three explicit fields cover the pre-warmed-pool case.
• Keep it to MCPServer in this PR. Leave the equivalent surface on MCPRemoteProxy and VirtualMCPServer for a follow-up so the review stays small.

On regeneration, one note since the new fields bring in corev1 types: run task operator-generate for the deepcopy methods as well as task operator-manifests for the CRDs. Beyond that the steps you listed are right: add the fields, wire them into the proxy pod-template patch, and extend mcpserver_resource_overrides_test.go. I have labelled this good-first-issue.

Feel free to open a PR when you are ready, and tag us if you want early feedback on direction before you go deep.

[talsh-oasis]

Thanks @ChrisJBurns — that scoping works for us, and both notes make the change smaller, so happy to take them.

Confirming what I'll build:

• Three explicit fields on resourceOverrides.proxyDeployment — nodeSelector, tolerations, affinity — rather than a podTemplateSpec. Agreed on the reasoning: a second broad override surface would overlap the metadata overrides already there, and these three cover the pre-warmed-pool case completely. It also keeps the proxy surface explicit about what's supported instead of implying the whole pod spec is patchable.
• MCPServer only. MCPRemoteProxy and VirtualMCPServer left for a follow-up.
• task operator-generate as well as task operator-manifests — thanks for calling that out, the deepcopy methods for the new corev1 types would have been an easy miss.

Plan for the PR:

1. Add the three fields to ProxyDeploymentOverrides in cmd/thv-operator/api/v1beta1/mcpserver_types.go.
2. Wire them into the proxy pod-template patch so they reach the existing applyPodTemplatePatch(..., options.K8sPodTemplatePatch) path in pkg/container/kubernetes/client.go — no runtime change needed, just populating what's already applied.
3. task operator-generate + task operator-manifests.
4. Extend mcpserver_resource_overrides_test.go, covering each field individually and all three together, plus a case asserting the proxy patch is unset when no overrides are given (so the default path is provably unchanged).

One question while I'm in there: should the new fields be additive to anything the operator already sets on the proxy pod spec, or is it fine for them to be authoritative? I don't believe the operator sets proxy scheduling today, so authoritative looks correct and simpler — but if there's an internal default I've missed, I'd rather merge than clobber it.

Will tag you on the PR.

[talsh-oasis]

Started on this and hit a wrinkle worth your call before I open the PR, @ChrisJBurns.

"MCPServer only" isn't cleanly achievable, because ProxyDeploymentOverrides is reached through the shared ResourceOverrides struct, and MCPRemoteProxy embeds the same one:

// mcpserver_types.go
type ResourceOverrides struct {
    ProxyDeployment *ProxyDeploymentOverrides `json:"proxyDeployment,omitempty"`
    ProxyService    *ResourceMetadataOverrides `json:"proxyService,omitempty"`
}

// mcpremoteproxy_types.go:153
ResourceOverrides *ResourceOverrides `json:"resourceOverrides,omitempty"`

So adding the three fields publishes them on both CRDs. After task operator-manifests, toolhive.stacklok.dev_mcpremoteproxies.yaml gains nodeSelector, tolerations and affinity under resourceOverrides.proxyDeployment even though I only wired the MCPServer controller. Net effect there: the API accepts the fields and the controller silently ignores them, which seems worse than not offering them.

(That's also most of the diff size — corev1.Affinity expands to ~1900 lines of CRD schema per file. Regenerating on a clean tree produces no churn, so it's all attributable to this change.)

Three ways forward, and I'd rather you pick:

1. Split the type — give MCPServer its own overrides type (embedding ResourceMetadataOverrides as today) so scheduling only appears on MCPServer. Cleanest separation, but it duplicates a struct and is a slightly bigger API change than you had in mind.
2. Wire MCPRemoteProxy too, in this PR. The schema is going to expose the fields regardless, so honouring them keeps the API honest. Larger review than you asked for, but no dead surface. Its proxy Deployment is built in mcpremoteproxy_controller.go, so it's the same shape of change twice.
3. Ship as-is for MCPServer and treat the MCPRemoteProxy surface as documented-but-inert until the follow-up. Smallest diff, but there'd be a window where the CRD advertises something that does nothing.

I lean 2 — the shared struct means the surface exists either way, and an inert field is the kind of thing that generates bug reports. But it's your API, and if you'd prefer 1 to keep the resources' capabilities distinct I'm happy to do that instead.

For what it's worth, one question from my earlier comment is now settled by the code: the operator sets no proxy pod scheduling today (every NodeSelector/Tolerations/Affinity hit under cmd/thv-operator is SessionAffinity, a Service field). So the new fields can be authoritative — nothing to merge with.

[oryanomer1]

We need this too, so a data point on the open question: option 2.

We run MCPRemoteProxy alongside MCPServer, so the inert-surface path in option 3 would land on us directly — a CRD field that validates and then silently does nothing is exactly the kind of thing that burns an afternoon before you think to check the controller. The larger review in option 2 seems cheaper than that.