MCPRemoteProxy operator doesn't reconcile spec changes to deployment configuration

[JAORMX]

Bug Description

The MCPRemoteProxy operator doesn't properly reconcile changes to the MCPRemoteProxy spec. When the spec is updated (e.g., adding jwksUrl or changing OIDC configuration), the operator doesn't update the deployment's ConfigMap or trigger pod recreation with the new configuration.

Steps to Reproduce

1. Deploy an MCPRemoteProxy:

kubectl apply -f mcp-spec-remote-proxy.yaml

2. Verify it's running:

kubectl get mcpremoteproxy -n toolhive-system mcp-spec-remote
# PHASE: Ready

3. Update the spec (e.g., add jwksUrl):

spec:
  oidcConfig:
    inline:
      issuer: http://keycloak.example.com/realms/toolhive
      clientId: mcp-remote-proxy
      audience: mcp-remote-proxy
      insecureAllowHTTP: true
      jwksAllowPrivateIP: true
      jwksUrl: http://keycloak.example.com/realms/toolhive/protocol/openid-connect/certs  # NEW

4. Apply the update:

kubectl apply -f mcp-spec-remote-proxy.yaml
# mcpremoteproxy.toolhive.stacklok.dev/mcp-spec-remote configured

5. Check the deployed configuration:

kubectl get mcpremoteproxy -n toolhive-system mcp-spec-remote -o yaml | grep jwksUrl
# Shows: jwksUrl: http://keycloak.example.com/...

6. Delete the pod to force recreation:

kubectl delete pod -n toolhive-system <pod-name>
# Wait for new pod to start

7. Check if the new configuration is used:

# The pod still uses the OLD configuration without jwksUrl!

Current Behavior

• MCPRemoteProxy resource is updated successfully
• kubectl get mcpremoteproxy shows the new spec
• But the deployment and ConfigMaps are NOT updated
• Pods continue running with old configuration
• Even deleting pods doesn't help - new pods use old config
• Only way to apply changes is to delete the entire MCPRemoteProxy and recreate it

Expected Behavior

When the MCPRemoteProxy spec is updated:

1. Operator should detect the change (via ObservedGeneration)
2. Update the runconfig ConfigMap with new configuration
3. Update the deployment if needed (e.g., environment variables, volumes)
4. Trigger pod rollout with new configuration
5. Update status.observedGeneration to match

Similar to how Kubernetes Deployments work:

kubectl apply -f deployment.yaml  # Changes spec
# Deployment controller automatically rolls out new pods

Workaround

Currently, the only way to apply configuration changes is to delete and recreate:

kubectl delete mcpremoteproxy -n toolhive-system mcp-spec-remote
kubectl apply -f mcp-spec-remote-proxy.yaml
# Apply health probe patches again

This is very disruptive and defeats the purpose of declarative configuration.

Impact

• Configuration changes require full resource deletion/recreation
• No way to hot-reload OIDC configuration
• Difficult to iterate on policies and settings
• Makes MCPRemoteProxy operationally painful to manage
• Breaks GitOps workflows expecting declarative updates

Root Cause Analysis

The operator likely:

1. Creates initial deployment and ConfigMaps
2. Sets up watches for MCPRemoteProxy resources
3. But doesn't properly handle Update events
4. Or doesn't compute a hash of the spec to detect changes
5. Or doesn't trigger ConfigMap/Deployment updates on spec changes

Suggested Fix

Implement proper reconciliation:

func (r *MCPRemoteProxyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // ... fetch MCPRemoteProxy ...
    
    // Compute hash of current spec
    specHash := computeHash(proxy.Spec)
    
    // Check if spec changed
    if proxy.Status.ObservedGeneration != proxy.Generation || 
       proxy.Status.ConfigHash != specHash {
        
        // Update ConfigMap with new configuration
        if err := r.updateRunConfig(ctx, proxy); err != nil {
            return ctrl.Result{}, err
        }
        
        // Update deployment if needed
        if err := r.updateDeployment(ctx, proxy); err != nil {
            return ctrl.Result{}, err
        }
        
        // Update status
        proxy.Status.ObservedGeneration = proxy.Generation
        proxy.Status.ConfigHash = specHash
        if err := r.Status().Update(ctx, proxy); err != nil {
            return ctrl.Result{}, err
        }
    }
    
    return ctrl.Result{}, nil
}

Environment

• ToolHive version: v0.4.2
• Kubernetes: v1.31+
• Operator: toolhive-operator

Related

• Similar pattern needed for MCPServer, MCPToolConfig, MCPExternalAuthConfig
• This is a fundamental operator reconciliation issue