Fix: Upgrade mcp-oauth to v0.2.56 for cross-client audience scope merging

[teemow]

Summary

• Upgrades mcp-oauth from v0.2.55 to v0.2.56
• Fixes SSO token forwarding for Kubernetes OIDC authentication
• Adds automatic retry loop for failed MCPServer services with exponential backoff
• Improves transient error detection to include HTTP 5xx server errors

Problem

When users authenticated to muster, client-requested OAuth scopes (openid profile email offline_access) were overriding the provider's configured cross-client audience scopes. This resulted in tokens that lacked the dex-k8s-authenticator audience required for Kubernetes API access.

Symptom: muster agent could connect to mcp-kubernetes via SSO, but Kubernetes API calls failed with "the server has asked for the client to provide credentials".

Additionally, when MCP servers experienced transient failures (e.g., 503 Service Unavailable), muster would not automatically retry connecting to them.

Root Cause

1. OAuth scope merging: The CopyScopes function in mcp-oauth used either client-requested scopes OR provider defaults, not both. Cross-client audience scopes configured via requiredAudiences in MCPServer CRs were being ignored.

2. Missing retry logic: The orchestrator lacked automatic retry for failed MCP servers, requiring manual intervention for transient issues.

Fix

1. mcp-oauth v0.2.56 includes a fix that merges mandatory scopes (cross-client audiences like audience:server:client_id:dex-k8s-authenticator) from provider defaults into client-requested scopes.

2. Automatic retry loop: Added a background goroutine in the orchestrator that checks failed/unreachable MCP servers every 30 seconds and attempts reconnection when their backoff period expires.

3. HTTP 5xx detection: Extended transient error detection to recognize HTTP 5xx server errors (500-504, 507-509) as retry-eligible.

Changes

Orchestrator (internal/orchestrator/orchestrator.go)

• Added RetryInterval constant (30 seconds) at file-level with other constants
• Added sync.WaitGroup for tracking in-flight retry goroutines (clean shutdown)
• Added retryFailedMCPServers() - background loop respecting context cancellation
• Added attemptReconnectFailedServers() - checks backoff and spawns restart goroutines
• Added shouldAttemptRetry() - extracted eligibility check for readability

MCP Server Service (internal/services/mcpserver/service.go)

• Extended isTransientConnectivityError() with consolidated HTTP 5xx patterns
• Added coverage for status codes 500, 501, 502, 503, 504, 507, 508, 509

Tests (internal/services/mcpserver/service_test.go)

• Added comprehensive HTTP 5xx test cases including:
  • All covered status codes (500-504, 507-509)
  • Mixed case error messages
  • Wrapped errors
  • 4xx errors (verified as NOT transient)

Test Plan

• Unit tests pass (make test)
• Deploy to gazelle and verify SSO token forwarding works with Kubernetes OIDC
• Verify muster auth status shows gazelle-mcp-kubernetes as Connected [SSO: Forwarded]
• Verify kubernetes_cluster_health tool returns healthy status instead of auth error
• Verify failed MCP servers are automatically retried after backoff expires

Related

• mcp-oauth issue: Cross-client audience scopes not merged with client-requested scopes mcp-oauth#203