Error when trying to run from updated proxtPort and mcpPort

[sachinmalanki]

Bug description

Was working fine till I update the helm chart to 0.3.2 and crd to 0.0.47.
Getting this error in the operator -

{"level":"error","ts":1762259790.5987027,"caller":"controller/controller.go:474","msg":"Reconciler error","controller":"mcpserver","controllerGroup":"toolhive.stacklok.dev","controllerKind":"MCPServer","MCPServer":{"name":"yardstick","namespace":"ai"},"namespace":"ai","name":"yardstick","reconcileID":"db2874d1-e74c-4c96-bec6-8e1921998395","error":"invalid RunConfig: target port is required for transport type streamable-http","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\tsigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296"}   

Steps to reproduce

Config -

- apiVersion: toolhive.stacklok.dev/v1alpha1
    kind: MCPServer
    metadata:
      name: yardstick
      namespace: ai
    spec:
      image: ghcr.io/stackloklabs/yardstick/yardstick-server:0.0.2
      transport: streamable-http
      env:
      - name: TRANSPORT
        value: streamable-http
      proxyPort: 8080
      mcpPort: 8080
      resources:
        limits:
          cpu: "100m"
          memory: "128Mi"
        requests:
          cpu: "50m"
          memory: "64Mi"

Additional context

Any additional information or logs you think might help.

[ChrisJBurns]

@sachinmalanki Thanks for the issue. I did the underlying ticket to remove the targetPort so I'll get straight into this!

[ChrisJBurns]

@sachinmalanki I think you may have some alignment issues with the CRDs and the Helm Chart for the Operator. The 0.3.2 version of the Operator Chart installs version 0.5.0 version of the operator image. Whereas the port renaming got released under 0.5.1. If you specify the 0.5.1 version of the Operator image in the Helm Chart this should include the code that works with the new mcpPort and proxyPort values.

[sachinmalanki]

Let me give it a try.

[sachinmalanki]

I'm not sure why after upgrading the helm chart for toolhive-operator-crds from 0.0.13 to 0.0.47 and toolhive-operator from 0.2.1 to 0.3.2, Grafana MCP server has stop working.
Current mcp server config -

  - apiVersion: toolhive.stacklok.dev/v1alpha1
    kind: MCPServer
    metadata:
      name: grafana
      namespace: ai
    spec:
      args:
      - "--endpoint-path=/grafana"
      - "--transport=streamable-http"
      image: xxx.dkr.ecr.eu-west-1.amazonaws.com/mcp/grafana-mcp-server:v1.4.11.2025
      transport: streamable-http
      proxyPort: 9090
      mcpPort: 8000      
      permissionProfile:
        type: builtin
        name: network
      secrets:
        - name: mcp-token
          key: grafana_token
          targetEnvName: GRAFANA_SERVICE_ACCOUNT_TOKEN
      env:  
        - name: GRAFANA_URL
          value: "https://xxx.grafana.net/"        
      resources:
        limits:
          cpu: "200m"
          memory: "256Mi"
        requests:
          cpu: "100m"
          memory: "128Mi" 

MCP Server Logs -

│ {"level":"info","ts":1762523641.4747105,"caller":"logger/logger.go:39","msg":"Successfully loaded configuration from /etc/runconfig/runconfig.json"}                                                                                    │
│ {"level":"info","ts":1762523641.474757,"caller":"logger/logger.go:39","msg":"Auto-discovered and loaded configuration from runconfig.json file"}                                                                                        │
│ {"level":"info","ts":1762523641.474777,"caller":"logger/logger.go:34","msg":"OIDC validation disabled, using local user authentication"}                                                                                                │
│ {"level":"warn","ts":1762523641.4748325,"caller":"logger/logger.go:54","msg":"Failed to get current user, using 'local' as default: user: Current requires cgo or $USER set in environment"}                                            │
│ {"level":"info","ts":1762523641.4748487,"caller":"logger/logger.go:39","msg":"Setting up streamable-http transport..."}                                                                                                                 │
│ {"level":"info","ts":1762523641.474861,"caller":"logger/logger.go:39","msg":"Deploying workload grafana from image xxx.dkr.ecr.eu-west-1.amazonaws.com/mcp/grafana-mcp-server:v1.4.11.2025..."}                                │
│ {"level":"info","ts":1762523641.4837701,"caller":"logger/logger.go:34","msg":"Configuring pod security context for Kubernetes"}                                                                                                         │
│ {"level":"info","ts":1762523641.4838052,"caller":"logger/logger.go:34","msg":"Configuring container security context for Kubernetes"}                                                                                                   │
│ {"level":"info","ts":1762523641.503294,"caller":"logger/logger.go:39","msg":"Applied statefulset grafana"}                                                                                                                              │
│ {"level":"info","ts":1762523641.5178895,"caller":"logger/logger.go:39","msg":"Created headless service grafana for SSE transport"}                                                                                                      │
│ {"level":"info","ts":1762523641.5195436,"caller":"logger/logger.go:39","msg":"Container created: grafana"}                                                                                                                              │
│ {"level":"info","ts":1762523641.5195663,"caller":"logger/logger.go:39","msg":"Starting streamable-http transport for grafana..."}                                                                                                       │
│ {"level":"info","ts":1762523641.5195808,"caller":"logger/logger.go:39","msg":"Setting up transparent proxy to forward from host port 9090 to http://mcp-grafana-headless:8000"}                                                         │
│ {"level":"info","ts":1762523641.5196106,"caller":"logger/logger.go:39","msg":"Applied middleware: mcp-parser"}                                                                                                                          │
│ {"level":"info","ts":1762523641.5196152,"caller":"logger/logger.go:39","msg":"Applied middleware: auth"}                                                                                                                                │
│ {"level":"info","ts":1762523641.5199156,"caller":"logger/logger.go:34","msg":"No auth info handler provided; skipping /.well-known/ endpoint"}                                                                                          │
│ {"level":"info","ts":1762523641.5199296,"caller":"logger/logger.go:39","msg":"HTTP transport started for grafana on port 9090"}                                                                                                         │
│ {"level":"info","ts":1762523641.5266457,"caller":"logger/logger.go:39","msg":"MCP server grafana started successfully"}                                                                                                                 │
│ {"level":"info","ts":1762523641.5269752,"caller":"logger/logger.go:39","msg":"Server grafana has no group, updating 0 globally registered client(s) for backward compatibility"}                                                        │
│ {"level":"info","ts":1762523641.5269868,"caller":"logger/logger.go:39","msg":"No target clients found for server grafana"}                                                                                                              │
│ {"level":"info","ts":1762523641.526993,"caller":"logger/logger.go:34","msg":"Press Ctrl+C to stop or wait for container to exit"}                                                                                                       │
│ {"level":"info","ts":1762523651.5205374,"caller":"logger/logger.go:39","msg":"MCP server not initialized yet, skipping health check for grafana"}                                                                                       │
│ {"level":"info","ts":1762523661.520908,"caller":"logger/logger.go:39","msg":"MCP server not initialized yet, skipping health check for grafana"}                                                                                        │
│ {"level":"info","ts":1762523671.520226,"caller":"logger/logger.go:39","msg":"MCP server not initialized yet, skipping health check for grafana"}                                                                                        │
│ {"level":"info","ts":1762523681.5208638,"caller":"logger/logger.go:39","msg":"MCP server not initialized yet, skipping health check for grafana"} 

Proxy Server Logs -

│ time=2025-11-07T13:53:56.618Z level=INFO msg="Starting Grafana MCP server using SSE transport" version=(devel) address=0.0.0.0:8000 basePath=/                                                                                          │

Also using an ingress -

  - apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: mcp-server-ingress
      namespace: ai
      labels:
        app: mcp-server
      annotations:
        # AWS Load Balancer Controller annotations
        alb.ingress.kubernetes.io/scheme: internal
        kubernetes.io/ingress.class: alb
        alb.ingress.kubernetes.io/load-balancer-name: xxx
        alb.ingress.kubernetes.io/backend-protocol: HTTP
        alb.ingress.kubernetes.io/subnets: xxx
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/certificate-arn: "xxx"
        alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    spec:
      ingressClassName: alb
      rules:
      - host: mcp.tec.pmn.we1.nexus.xxx.co.uk
        http:
          paths:
          - path: /grafana
            pathType: Prefix
            backend:
              service:
                name: mcp-grafana-proxy
                port:
                  number: 9090                                                              
      # Optional: Default backend for unmatched requests
      defaultBackend:
        service:
          name: mcp-grafana-proxy
          port:
            number: 9090  

Also not sure why proxy server is starting in SSE mode after specifying Streamable Http mode.

Any help would be appreciated 🙏

[ChrisJBurns]

@sachinmalanki That's not great! Apologies for this, let me run this locally and see if I can get it to work. Regarding the SSE logging entry, thank you for reminding me about this, I've raised #2507 to fix this, the long story short is that the service that is being created works for Streamable Http too, so its still doing the right thing, the log is just misleading.

I'll run this locally to see if I can figure out why it doesn't work e2e - bare with 💪

[ChrisJBurns]

Could you send over the headless service and the proxy svc manifests? I don't need to see all of it, more specifically, I just want to see if the ports are being set correctly on both of them

[sachinmalanki]

grafana-headless service

│ apiVersion: v1                                                                                                                                                                                                                          │
│ kind: Service                                                                                                                                                                                                                           │
│ metadata:                                                                                                                                                                                                                               │
│   creationTimestamp: "2025-11-04T12:17:17Z"                                                                                                                                                                                             │
│   labels:                                                                                                                                                                                                                               │
│     app: grafana                                                                                                                                                                                                                        │
│     toolhive: "true"                                                                                                                                                                                                                    │
│     toolhive-basename: grafana                                                                                                                                                                                                          │
│     toolhive-name: grafana                                                                                                                                                                                                              │
│     toolhive-port: "9090"                                                                                                                                                                                                               │
│     toolhive-tool-type: mcp                                                                                                                                                                                                             │
│     toolhive-transport: streamable-http                                                                                                                                                                                                 │
│   name: mcp-grafana-headless                                                                                                                                                                                                            │
│   namespace: ai                                                                                                                                                                                                                         │
│   resourceVersion: "421409971"                                                                                                                                                                                                          │
│   uid: ed9bf2d2-0e25-4545-8c3e-070e103d8841                                                                                                                                                                                             │
│ spec:                                                                                                                                                                                                                                   │
│   clusterIP: None                                                                                                                                                                                                                       │
│   clusterIPs:                                                                                                                                                                                                                           │
│   - None                                                                                                                                                                                                                                │
│   internalTrafficPolicy: Cluster                                                                                                                                                                                                        │
│   ipFamilies:                                                                                                                                                                                                                           │
│   - IPv4                                                                                                                                                                                                                                │
│   ipFamilyPolicy: SingleStack                                                                                                                                                                                                           │
│   ports:                                                                                                                                                                                                                                │
│   - name: port-8000                                                                                                                                                                                                                     │
│     port: 8000                                                                                                                                                                                                                          │
│     protocol: TCP                                                                                                                                                                                                                       │
│     targetPort: 8000                                                                                                                                                                                                                    │
│   selector:                                                                                                                                                                                                                             │
│     app: grafana                                                                                                                                                                                                                        │
│   sessionAffinity: None                                                                                                                                                                                                                 │
│   type: ClusterIP                                                                                                                                                                                                                       │
│ status:                                                                                                                                                                                                                                 │
│   loadBalancer: {}                                                                                                                                                                                                                                        

grafana-proxy service

│ apiVersion: v1                                                                                                                                                                                                                          │
│ kind: Service                                                                                                                                                                                                                           │
│ metadata:                                                                                                                                                                                                                               │
│   creationTimestamp: "2025-11-04T12:17:16Z"                                                                                                                                                                                             │
│   labels:                                                                                                                                                                                                                               │
│     app: mcpserver                                                                                                                                                                                                                      │
│     app.kubernetes.io/instance: grafana                                                                                                                                                                                                 │
│     app.kubernetes.io/name: mcpserver                                                                                                                                                                                                   │
│     toolhive: "true"                                                                                                                                                                                                                    │
│     toolhive-name: grafana                                                                                                                                                                                                              │
│   name: mcp-grafana-proxy                                                                                                                                                                                                               │
│   namespace: ai                                                                                                                                                                                                                         │
│   ownerReferences:                                                                                                                                                                                                                      │
│   - apiVersion: toolhive.stacklok.dev/v1alpha1                                                                                                                                                                                          │
│     blockOwnerDeletion: true                                                                                                                                                                                                            │
│     controller: true                                                                                                                                                                                                                    │
│     kind: MCPServer                                                                                                                                                                                                                     │
│     name: grafana                                                                                                                                                                                                                       │
│     uid: 10f97fc2-b4d1-44ec-820f-4e92e5cd9458                                                                                                                                                                                           │
│   resourceVersion: "419225529"                                                                                                                                                                                                          │
│   uid: 2be7abb4-4c97-48ae-8f8d-982abaaf13ad                                                                                                                                                                                             │
│ spec:                                                                                                                                                                                                                                   │
│   clusterIP: 172.20.218.224                                                                                                                                                                                                             │
│   clusterIPs:                                                                                                                                                                                                                           │
│   - 172.20.218.224                                                                                                                                                                                                                      │
│   internalTrafficPolicy: Cluster                                                                                                                                                                                                        │
│   ipFamilies:                                                                                                                                                                                                                           │
│   - IPv4                                                                                                                                                                                                                                │
│   ipFamilyPolicy: SingleStack                                                                                                                                                                                                           │
│   ports:                                                                                                                                                                                                                                │
│   - name: http                                                                                                                                                                                                                          │
│     port: 9090                                                                                                                                                                                                                          │
│     protocol: TCP                                                                                                                                                                                                                       │
│     targetPort: 9090                                                                                                                                                                                                                    │
│   selector:                                                                                                                                                                                                                             │
│     app: mcpserver                                                                                                                                                                                                                      │
│     app.kubernetes.io/instance: grafana                                                                                                                                                                                                 │
│     app.kubernetes.io/name: mcpserver                                                                                                                                                                                                   │
│     toolhive: "true"                                                                                                                                                                                                                    │
│     toolhive-name: grafana                                                                                                                                                                                                              │
│   sessionAffinity: None                                                                                                                                                                                                                 │
│   type: ClusterIP                                                                                                                                                                                                                       │
│ status:                                                                                                                                                                                                                                 │
│   loadBalancer: {}    

This is the current proxy and headless service. Please let me know if you need any other info.

Thanks for you help @ChrisJBurns 🙏

[ChrisJBurns]

@sachinmalanki I was able to reproduce the issue locally using Kind and a local Grafana instance installed in my Kind cluster with the docker.io/mcp/grafana image.

The problem comes from a small leftover change from some runconfig updates we made a few weeks ago. The args you specified were being overwritten by an empty array, which meant the MCP server for Grafana was starting in SSE mode instead of StreamableHTTP. You can verify this yourself if you compare the cmd_args field in the Grafana RunConfig ConfigMap (which has the correct args you specify) with the missing args field on the Grafana MCP pod. I’ll push a fix for this so the MCP Grafana server starts correctly with the right transport type.

Just for now and to verify that this fixes it for you, if you modify the StatefulSet of the Grafana MCP server, and add the following bits, it should start up nicely and toolhive proxy shouldn't have any issues connecting.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: grafana
  namespace: ai
...
...
spec:
    spec:
      containers:
+    - args:
+      - --endpoint-path=/grafana
+      - --transport=streamable-http
   ...
...

I'll link the PR to fix this to this issue so you don't have to rely on a manual statefulset change

[juancarlosm]

I can confirm that spec.args is not working, I needed to use podTemplateSpec to make it work:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: deepwiki
spec:
  image: supercorp/supergateway:latest
  transport: stdio
  podTemplateSpec:
    spec:
      containers:
      - name: mcp
        args:
          - --sse
          - https://mcp.deepwiki.com/sse
  telemetry:
    prometheus:
      enabled: true

[ChrisJBurns]

@juancarlosm Thanks for confirming, it should be fixed when we release after #2515 is merged.

[sachinmalanki]

I'm now able to connect with the podTemplateSpec. Thanks for the temp fix