[Kubernetes Operator] Unable to pass env vars to container

[rubenharutyunov]

I have the following MCPServer resource:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: grafana
  namespace: grafana-mcp
spec:
  image: docker.io/mcp/grafana
  transport: stdio
  port: 8000
  env:
    - value: grafana.example.com
      name: GRAFANA_URL
  permissionProfile:
    type: builtin
    name: network
  resources:
    limits:
      cpu: "20m"
      memory: "64Mi"
    requests:
      cpu: "10m"
      memory: "32Mi"

Based on examples I see in this repo, the operator should pass the env vars defined in the manifest into the mcp server pod it creates, however I see them set only in the runner pod. Here's what I see in k describe pod grafana-0:

Containers:
  mcp:
    Container ID:  docker://554e935eb935ec6527cbebeff8ca6f7a5c6f00ecdb1213cf0499eafc3f267575
    Image:         docker.io/mcp/grafana
    Port:          <none>
    Host Port:     <none>
    Args:
      --transport
      stdio
    State:          Running
      Started:      Fri, 30 May 2025 15:48:39 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      MCP_TRANSPORT:  stdio
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gb5hv (ro)

I'm running the latest dev versions: 0.0.1-dev.400_bcba31b

[ChrisJBurns]

I'll look into this @rubenharutyunov

Just as a quick test, I'm wondering if the podTemplateSpec works?

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: fetch
  namespace: toolhive-system
spec:
  image: docker.io/mcp/fetch
  transport: stdio
  port: 8080
  permissionProfile:
    type: builtin
    name: network
  podTemplateSpec:
    spec:
      containers:
        - name: mcp
          env:
          - value: grafana.example.com
             name: GRAFANA_URL

I will look into the top-level env field issue that you've raised in the meantime.

[rubenharutyunov]

@ChrisJBurns yes, podTemplateSpec works

[rubenharutyunov]

Probably the reason podTemplateSpec works is that it's being passed directly to the runner and applied to the MCP server container:

    Args:
      run
      --foreground=true
      --port=8000
      --name=grafana
      --transport=stdio
      --host=0.0.0.0
      --target-port=8000
      --k8s-pod-patch={"metadata":{"creationTimestamp":null},"spec":{"containers":[{"name":"mcp","env":[{"name":"GRAFANA_URL","value":"grafana.example.com"}],"resources":{}}]}}
      --permission-profile=network
      docker.io/mcp/grafana

I think the proper way to fix this is to introduce a new field to set env vars for the MCP server itself. @ChrisJBurns I don't mind having a look at this one as well.

[JAORMX]

@rubenharutyunov @ChrisJBurns in this PR I'm introducing more targeted places to override the different parts of the deployment #514 . Perhaps thats where this could go.

[rubenharutyunov]

@JAORMX if I understand correctly, the PR introduces a new field to manage overrides for the runner. Should we then use that to add env vars to runner container and use .spec.env to pass them to the MCP server itself?

[ChrisJBurns]

@JAORMX @rubenharutyunov

Thinking into this a bit more, and using the logic that we've set in the Operator Design doc regarding CRD field vs podTemplateSpec, I'm thinking that we remove the top-level spec.env field and instead use the resourceOverrides fields to set the pod specs for the different components. This should allow us a great deal of customisability and ability to override the underlying resources without the need to ever add "just one more" top level field to support custom use cases.

For example:

resourceOverrides:
    proxyDeployment:
      metadata:
        annotations:
          monitoring.example.com/scrape: "true"
          example.com/team: "platform"
        labels:
          environment: "production"
      podTemplateSpec:
        containers:
        - name: mcp
          env:
          - value: my-var
             name: PROXY_VARIABLE
    mcpServerStatefulSet:
      podTemplateSpec:
        containers:
        - name: mcp
          env:
          - value: grafana.example.com
             name: GRAFANA_URL

[rubenharutyunov]

@ChrisJBurns thanks, introducing resource overrides for both proxy and mcp server would definitely help. However I'm not sure about removing .spec.env entirely since in most cases to run an MCP server users only need to provide env and args and having to always rely on patching the actual statefulset via .resourceOverrides. mcpServerStatefulSet. podTemplateSpec seems overcomplicated for most of the use-cases.

[ChrisJBurns]

@rubenharutyunov That's fair enough, we shall leave it in then! Perhaps rename it to mcpEnv and runnerEnv ? This making it more clear and allowing users to provide them for both pods

[rubenharutyunov]

Perhaps rename it to mcpEnv and runnerEnv ?

I would say .env for MCP server env vars should be enough. I think it should be fine to have it as an "alias" to .resourceOverrides.mcpServerStatefulSet.podTemplateSpec.env and use .resourceOverrides.proxyDeployment for proxy-related config.