PodTemplateSpec Environment Variables Ignored

[therealnb]

Type: Regression
Introduced: November 10, 2025

Description

Environment variables defined in spec.podTemplateSpec.spec.containers[0].env are not transferred to the runconfig.json file or the resulting pods.

Root Cause

File: cmd/thv-operator/controllers/mcpserver_runconfig.go:86

func (r *MCPServerReconciler) createRunConfigFromMCPServer(m *mcpv1alpha1.MCPServer) (*runner.RunConfig, error) {
    // ...
    envVars := convertEnvVarsFromMCPServer(m.Spec.Env)  // Only reads top-level Env!
    // PodTemplateSpec.spec.containers[0].env is never read
    // ...
}

The function only reads m.Spec.Env (top-level field) and completely ignores environment variables nested in spec.podTemplateSpec.

Steps to Reproduce

1. Create an MCPServer with environment variables in podTemplateSpec:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: test-server
spec:
  image: my-image:latest
  podTemplateSpec:
    spec:
      containers:
        - name: mcp
          env:
            - name: MY_CUSTOM_VAR
              value: "test-value"

2. Check the runconfig ConfigMap:

kubectl get configmap test-server-runconfig -o jsonpath='{.data.runconfig\.json}' | jq '.env_vars'

3. Check the actual pod environment:

kubectl get pod test-server-0 -o jsonpath='{.spec.containers[0].env}' | jq

Expected Behavior

Environment variables from spec.podTemplateSpec.spec.containers[0].env should be:

1. Included in the runconfig.json file
2. Present in the deployed container

Actual Behavior

Environment variables from podTemplateSpec are silently ignored. Only spec.env (top-level) variables are transferred.

Impact

• Breaks existing deployments: Configurations that worked before Nov 10, 2025 no longer work
• Silent failure: No warning or error is generated; variables are simply missing
• Workaround exists: Move env vars to top-level spec.env (but this breaks existing configurations)

Timeline

• Before Nov 10, 2025: ✅ Working - Operator converted podTemplateSpec to CLI flags
• After Nov 10, 2025: ❌ Broken - Operator only reads top-level fields for runconfig.json

[ChrisJBurns]

@therealnb Similar to #2606, I've deployed the latest version of the Operator into Kind and have used the following YAML and I can see the env vars on the MCP server statefulset

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: fetch
  namespace: toolhive-system
spec:
  image: ghcr.io/stackloklabs/gofetch/server
  transport: streamable-http
  proxyPort: 8080
  mcpPort: 8080
  podTemplateSpec:
    spec:
      containers:
        - name: mcp
          env:
            - name: MY_CUSTOM_VAR
              value: "test-value"
  resources:
    limits:
      cpu: "100m"
      memory: "128Mi"
    requests:
      cpu: "50m"
      memory: "64Mi"

[therealnb]

@ChrisJBurns It could be an old version of the operator (I'm also using rancher desktop, but I don't think that should matter). I will try them again. Thanks for looking at this.

[therealnb]

I can confirm that this works with the latest main, but not the latest release code. Some AI verbiage...

Yes, it was definitely a bug! And like the volume mounting issue, it was also fixed in the unreleased main branch, not in v0.6.5.

The Evidence

With operator v0.6.5 (released):

• The runconfig.json only contained operator-injected env vars:

"env_vars": {
  "FASTMCP_PORT": "9900",
  "MCP_PORT": "9900", 
  "MCP_TRANSPORT": "streamable-http"
}

• ✗ Missing: RUNTIME_MODE, K8S_ALL_NAMESPACES, DB_URL, ASYNC_DB_URL, etc.

With operator from latest main (commit 083a69f):

• All custom environment variables were present in the pod:

ASYNC_DB_URL=sqlite+aiosqlite:///data/mcp_optimizer.db
DB_URL=sqlite:///data/mcp_optimizer.db
K8S_ALL_NAMESPACES=true
RUNTIME_MODE=k8s
# Plus operator-injected ones:
FASTMCP_PORT=9900
MCP_PORT=9900

• ✅ All custom env vars from podTemplateSpec applied correctly

When Was It Fixed?

Both issues (#2605 and #2606) were fixed in the same unreleased code between v0.6.5 (Nov 13) and the current main (Nov 18). The maintainer's GitHub comment confirmed they tested both fixes together with "an operator image built on the latest main commit."

Thanks for looking at this, @ChrisJBurns !