Volumes Not Mounted in Pods

[therealnb]

Type: Missing feature / regression
Introduced: November 10, 2025

Description

Volumes defined in spec.volumes are accepted by the MCPServer CRD but are never:

1. Added to runconfig.json
2. Created in the pod spec
3. Mounted in the container

This makes it impossible to deploy applications that need writable filesystem access (databases, caches, logs) because the operator enforces readOnlyRootFilesystem: true.

Root Cause

File: cmd/thv-operator/controllers/mcpserver_runconfig.go:87

envVars := convertEnvVarsFromMCPServer(m.Spec.Env)
volumes := convertVolumesFromMCPServer(m.Spec.Volumes)  // ← Converted but never used!

The volumes are read and converted to the builder format, but they're never:

• Added to the runconfig.json (not in the RunConfigBuilder options)
• Applied to the pod spec when creating workloads

Steps to Reproduce

1. Create an MCPServer with volumes:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: test-server
spec:
  image: my-image:latest
  volumes:
    - name: data
      hostPath: ""  # emptyDir
      mountPath: /data
      readOnly: false
  env:
    - name: DATA_PATH
      value: /data

2. Deploy and check the runconfig:

kubectl get configmap test-server-runconfig -o jsonpath='{.data.runconfig\.json}' | jq '.volumes'
# Output: null

3. Check the pod spec:

kubectl get pod test-server-0 -o jsonpath='{.spec.volumes[*].name}'
# Output: Only "kube-api-access-*" (no user-defined volumes)

kubectl get pod test-server-0 -o jsonpath='{.spec.containers[0].volumeMounts}' | jq
# Output: Only serviceaccount mount (no user-defined mounts)

4. Application fails trying to write:

kubectl logs test-server-0
# Error: unable to open database file
# Or: Read-only file system

Expected Behavior

Volumes defined in spec.volumes should be:

1. Included in runconfig.json OR applied directly to pod spec
2. Created as Kubernetes volumes in the pod
3. Mounted in the container at the specified mount paths
4. Writable (when readOnly: false)

Actual Behavior

• Volumes are accepted in MCPServer CRD validation ✓
• Volumes appear in spec.volumes ✓
• Volumes are NOT in runconfig.json ✗
• Volumes are NOT in pod spec ✗
• Applications crash trying to write to read-only filesystem ✗

Impact

• Blocks critical deployments: Applications needing disk writes cannot be deployed
• No workaround available: Cannot be fixed from MCPServer configuration
• Affects MCP Optimizer: Cannot deploy the official MCP Optimizer in Kubernetes
• Security conflict: readOnlyRootFilesystem: true (good security) + no volume mounts (bug) = impossible to write files

Evidence from MCP Optimizer Deployment

Security Context Applied by Operator:

{
  "allowPrivilegeEscalation": false,
  "privileged": false,
  "readOnlyRootFilesystem": true,  // ← Makes entire filesystem read-only
  "runAsNonRoot": true,
  "runAsUser": 1000
}

MCPServer Configuration:

spec:
  volumes:
    - name: data
      hostPath: ""
      mountPath: /data
      readOnly: false
  env:
    - name: DB_URL
      value: "sqlite:///data/mcp_optimizer.db"

Result:

• Volume not mounted
• /data directory doesn't exist or is read-only
• SQLite error: unable to open database file
• Pod in CrashLoopBackOff

---

What Used to Work (Before Nov 10, 2025)

The original configuration in helm/mcp-optimizer/values.yaml (commit 48921bc):

spec:
  podTemplateSpec:
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: tmp
          emptyDir: {}
      containers:
        - name: mcp
          volumeMounts:
            - name: data
              mountPath: /data
            - name: tmp
              mountPath: /tmp
          env:
            - name: DB_URL
              value: "sqlite:///data/mcp_optimizer.db"

This worked because the operator:

1. Read podTemplateSpec
2. Converted it to CLI flags for thv-proxyrunner
3. Proxyrunner created pods with proper volumes and env vars

Environment

• ToolHive Operator Version: 0.2.22
• ToolHive Operator CRDs: 0.0.34
• Kubernetes: Rancher Desktop (Docker-based)
• Test Application: MCP Optimizer v0.1.1
• Test Date: November 17, 2025

[ChrisJBurns]

@therealnb I've tested this locally by using an operator image built on the latest main commit and using kind and the task kind-with-toolhive-operator-local command using the below MCP Server yaml.

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: fetch
  namespace: toolhive-system
spec:
  image: ghcr.io/stackloklabs/gofetch/server
  transport: streamable-http
  proxyPort: 8080
  mcpPort: 8080
  podTemplateSpec:
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: tmp
          emptyDir: {}
      containers:
        - name: mcp
          volumeMounts:
            - name: data
              mountPath: /data
            - name: tmp
              mountPath: /tmp
  resources:
    limits:
      cpu: "100m"
      memory: "128Mi"
    requests:
      cpu: "50m"
      memory: "64Mi"

I can confirm that the args are on the proxyrunner

apiVersion: v1
kind: Pod
metadata:
  name: fetch-cdbd5cb6-zwrj7
  namespace: toolhive-system
spec:
  containers:
  - args:
    - run
    - --k8s-pod-patch={"metadata":{},"spec":{"volumes":[{"name":"data","emptyDir":{}},{"name":"tmp","emptyDir":{}}],"containers":[{"name":"mcp","resources":{},"volumeMounts":[{"name":"data","mountPath":"/data"},{"name":"tmp","mountPath":"/tmp"}]}],"serviceAccountName":"fetch-sa"}}
    - ghcr.io/stackloklabs/gofetch/server

I can also confirm that on the MCPServer StatefulSet pod

apiVersion: v1
kind: Pod
metadata:
  name: fetch-0
  namespace: toolhive-system
spec:
  containers:
    image: ghcr.io/stackloklabs/gofetch/server
    volumeMounts:
    - mountPath: /data
      name: data
    - mountPath: /tmp
      name: tmp
  volumes:
  - emptyDir: {}
    name: data
  - emptyDir: {}
    name: tmp

Not entirely sure why your volumes aren't working, are you perhaps using an older version of the Operator?

[therealnb]

@ChrisJBurns It could be an old version of the operator (I'm also using rancher desktop, but I don't think that should matter). I will try them again. Thanks for looking at this.

[therealnb]

I can confirm that this works with the latest main, but not the latest release code. Some AI verbiage...

Correct! The latest released operator v0.6.5 (tagged November 13, 2025) did NOT work.

Here's what happened:

1. v0.3.8 (your initial version): ✗ Volumes not mounted, duplicate env vars
2. v0.6.5 (latest release): ✗ Same bugs - volumes not mounted, duplicate env vars
3. Latest main (commit 083a69f8, Nov 18): ✅ WORKING - volumes mounted correctly

The Evidence

When we tested with v0.6.5, we saw:

• ✗ Volumes NOT in pod spec: only runconfig and kube-api-access
• ✗ Environment variables NOT transferred from podTemplateSpec to the actual pods
• ✗ Duplicate MCP_PORT error

After building from main (20+ commits ahead of v0.6.5):

• ✅ Volumes mounted: data and tmp with emptyDir
• ✅ Environment variables properly applied
• ✅ No duplicate errors

Thanks for looking at this, @ChrisJBurns !