Feat: Implement ADR-008 unified authentication (All Phases Complete)

[teemow]

Summary

This PR implements all phases of ADR-008: Unified Authentication Architecture.

Closes #196

Implementation Status

Phase	Status	Description
Phase 1	Done	Shared OAuth core in pkg/oauth/
Phase 2	Done	auth://status MCP resource
Phase 3	Done	Issuer-keyed agent token store
Phase 4	Done	AuthWatcher + submit_auth_token tool
Phase 5	Done	Structured 401 detection
Phase 6	Done	Fragile pattern cleanup (final commit)
Security Review	Done	All recommendations implemented
Stdlib Refactoring	Done	Use golang.org/x/oauth2 for PKCE & token exchange
Code Review	Done	Applied review recommendations

Recent: Final Cleanup (Phase 6)

Removed the last remaining fragile string-matching patterns:

1. internal/orchestrator/api_adapter.go - Removed strings.Contains(err.Error(), "authentication required") fallback. Now uses only structured AuthRequiredError via errors.As().

2. internal/aggregator/server.go - Replaced strings.Contains(connectErr.Error(), "401") || strings.Contains(connectErr.Error(), "Unauthorized") with pkgoauth.Is401Error(connectErr).

3. Updated tests - Tests now verify structured error detection instead of string matching.

This completes the ADR-008 goal of eliminating fragile inference-based authentication detection.

Recent: Code Review Fixes

Applied the following code review recommendations:

1. Fixed potential panic in WithHTTPTimeout - Added nil check for defensive coding
2. Replaced custom contains helper with strings.Contains - Uses stdlib instead of reimplementing
3. Fixed variable shadowing in GenerateState - Renamed stateBytes variable to b to avoid shadowing the constant
4. Consolidated AuthStatusResponse types - Moved AuthStatusResponse, ServerAuthStatus, and AuthRequiredInfo to pkg/oauth/types.go to avoid duplication between internal/aggregator and internal/agent packages

Recent: OAuth Stdlib Refactoring

Analyzed the OAuth code and identified opportunities to leverage the standard golang.org/x/oauth2 library:

Changes Made

1. PKCE Generation (pkg/oauth/pkce.go)

   • Replaced custom PKCE implementation with oauth2.GenerateVerifier() and oauth2.S256ChallengeFromVerifier()
   • Simplified GeneratePKCERaw() signature (no longer returns error)

2. Authorization URL Building (internal/agent/oauth/client.go)

   • Replaced manual URL construction with oauth2.Config.AuthCodeURL()
   • Uses oauth2.S256ChallengeOption() for PKCE parameters

3. Token Exchange (internal/agent/oauth/client.go)

   • Replaced ~40 lines of manual HTTP handling with oauth2.Config.Exchange()
   • Uses oauth2.VerifierOption() for PKCE verification

Benefits

• ~350 lines removed: Net reduction by using stdlib
• RFC 7636 compliance: Stdlib ensures proper PKCE implementation
• Maintainability: Less custom code to maintain
• Consistency: Same PKCE logic across all OAuth flows

What Remains Custom (Justified)

Functionality	Reason
Metadata Discovery	MCP requires dynamic issuer discovery, not known upfront
WWW-Authenticate Parsing	Stdlib doesn't parse resource server challenges
Token Storage	Different requirements for agent (file) vs server (memory/valkey)
State Management	Carries MCP-specific metadata (session ID, server name)

Security Review Implementation

A comprehensive security review was conducted and the following recommendations were implemented:

1. Token Logging Prohibition

• Added explicit SECURITY comments throughout the codebase prohibiting token logging
• All handlers that process tokens now have documentation stating token values must NEVER be logged
• Only server names, issuers, and operation outcomes are logged

2. Configurable HTTP Timeout

• Added WithHTTPTimeout(timeout time.Duration) option to pkg/oauth/Client
• Useful for environments with slow network connections to identity providers
• Default timeout remains 30 seconds

3. Exponential Backoff for Polling Failures

• AuthWatcher now implements exponential backoff (1s min, 5min max, 2x multiplier)
• Prevents overwhelming the server during connectivity issues
• Logs warning after 3 consecutive failures
• Automatically recovers when connection is restored

4. Structured Audit Logging

Added SECURITY_AUDIT prefixed log entries for:

• Token storage: When tokens are stored/persisted
• Token deletion: When tokens are removed
• Token clearing: When all tokens are cleared
• SSO operations: Token lookup, submission attempts, success/failure
• Browser auth: When browser authentication is required

Log events include structured fields:

• event: Standardized event name (e.g., token_stored, sso_auth_success)
• server_url, issuer_url: Context for the operation
• session: Session ID for traceability
• error: Error details on failure

Test Coverage

Package	Coverage
pkg/oauth	89.3%
internal/agent/auth_watcher	Covered
internal/agent/oauth/token_store	Covered

All 135 BDD scenarios pass.

Changes

Phase 1: Shared OAuth Core (pkg/oauth/)

Created a shared OAuth package that both agent and server can import:

• types.go: Token, Metadata, AuthChallenge, PKCEChallenge, ClientMetadata, AuthStatusResponse, ServerAuthStatus, AuthRequiredInfo
• client.go: OAuth client with metadata discovery, token exchange, token refresh
• pkce.go: PKCE generation (now delegates to golang.org/x/oauth2)
• www_authenticate.go: WWW-Authenticate header parsing
• Comprehensive unit tests for all components

Benefits:

• ~80% reduction in duplicated OAuth code
• Single source of truth for OAuth types and utilities
• Agent and Server share the same implementation

Phase 1: Refactor Existing OAuth Packages

Updated internal/oauth/ and internal/agent/oauth/ to use the shared core:

• Delegate to pkg/oauth for parsing, PKCE, metadata discovery
• Maintain backwards-compatible APIs
• Updated tests to use new APIs

Phase 2: Auth Status Resource

Added auth://status MCP resource that provides structured auth state:

{
  "muster_auth": {
    "authenticated": true
  },
  "server_auths": [
    {
      "server_name": "mcp-kubernetes",
      "status": "auth_required",
      "auth_challenge": {
        "issuer": "https://dex.example.com",
        "scope": "openid profile",
        "auth_tool_name": "x_mcp-kubernetes_authenticate"
      }
    }
  ]
}

Benefits:

• Agent can now get explicit auth state instead of inferring from tool names
• Includes issuer URL for SSO decisions
• Foundation for continuous auth watching

Phase 3: Issuer-Keyed Agent Token Store

Enhanced the agent's token store with issuer-based lookup:

• Added GetByIssuer(issuerURL string) method for SSO token lookup
• Added HasValidTokenForIssuer(issuerURL string) method
• Added findTokenByIssuerFromFilesLocked() for persistent storage lookup
• Full test coverage for new functionality

Benefits:

• Enables SSO by looking up tokens by issuer URL
• Tokens for one server can be reused for another server with same issuer

Phase 4a: AuthWatcher

Created continuous auth state watcher (internal/agent/auth_watcher.go):

• Polls auth://status resource at configurable intervals (10s default)
• Detects new auth challenges and resolved challenges
• Automatically forwards tokens via SSO when matching issuer found
• Callback system for auth events (OnBrowserAuthRequired, OnAuthComplete, OnTokenSubmitted)
• Exponential backoff on repeated failures (1s-5min)
• Full test coverage

Benefits:

• Replaces one-shot SSO with continuous watching
• Automatically authenticates new servers without restart
• Explicit auth state instead of tool name inference
• Resilient to temporary connectivity issues

Phase 4b: Submit Auth Token Tool

Added submit_auth_token tool to aggregator:

• Allows agent to submit OAuth tokens for pending auth servers
• Supports both session-scoped and global token submission
• Connects to server and fetches capabilities after token submission
• Notifies session of tool changes
• Security audit logging for all operations

Benefits:

• Enables SSO token forwarding from agent to server
• Allows agent-side token management

Phase 5: Structured 401 Detection

Improved 401 error handling in internal/mcpserver/types.go:

• Refactored AuthRequiredError to include AuthChallenge from pkg/oauth
• Added helper methods: HasValidChallenge(), GetIssuer(), GetScope(), GetResourceMetadataURL()
• Leverages pkg/oauth.ParseWWWAuthenticateFromError() for consistent parsing
• Deprecated ParseAuthInfoFromError() in favor of shared utilities

Benefits:

• Structured error handling instead of string matching
• Full auth challenge information preserved
• Consistent parsing across codebase

Phase 6: Fragile Pattern Cleanup

Removed remaining fragile patterns identified in code audit:

• api_adapter.go: Removed strings.Contains fallback for auth detection
• server.go: Use pkgoauth.Is401Error() instead of string matching
• Updated tests to verify structured error handling

Benefits:

• No more string-based auth detection anywhere in the codebase
• All auth detection uses structured error types
• Cleaner, more maintainable code

Code Review Improvements

Applied recommendations from code review:

• Fixed data race in TokenStore.GetByIssuer() by using proper lock ordering (read lock for cache check, write lock for file scanning with cache population)
• Removed dead code: Deleted unused tryConnectWithTokenForSSO() function
• Fixed extractToolError(): Implemented proper error extraction instead of returning nil
• Consolidated duplicate types: Moved AuthStatusResponse, ServerAuthStatus, and AuthRequiredInfo to pkg/oauth/ package used by both agent and aggregator
• Compile regex once: Moved WWW-Authenticate parameter regex to package level for performance
• Fixed potential panic: Added nil check in WithHTTPTimeout
• Fixed variable shadowing: Renamed variable in GenerateState to avoid shadowing constant
• Use stdlib: Replaced custom contains helper with strings.Contains

Security Considerations

The implementation follows OAuth 2.1 security best practices:

1. PKCE: Uses golang.org/x/oauth2 stdlib with S256 challenge only
2. Token Storage: Files created with 0600 permissions, directory with 0700
3. Token Expiry: 60-second buffer prevents using nearly-expired tokens
4. State Parameter: 256 bits of entropy for CSRF protection
5. No Token Logging: Token values are never logged (explicitly documented)
6. Audit Trail: All security-sensitive operations logged with structured events

Migration Notes

• No backwards compatibility required for token storage
• Users re-authenticate once after upgrading
• New tokens are stored with issuer URL, enabling SSO immediately
• Clean slate approach simplifies implementation and avoids migration edge cases

Testing

• All unit tests pass
• All 135 BDD scenarios pass
• Build succeeds
• pkg/oauth coverage: 89.3%

ADR

See ADR-008: Unified Authentication Architecture for the full design.