Improve OTel trace signal quality: stack traces, route cardinality, rate-limit status, and container resource metadata

[Copilot]

This PR addresses observability gaps in the current OpenTelemetry integration: error spans were missing stack traces, HTTP server spans used high-cardinality paths, rate-limit failures were not consistently marked as errors, and container metadata was not attached to resources.

• Error spans now include stack traces

  • Updated RecordError call sites in internal/server/unified.go to pass oteltrace.WithStackTrace(true) for tool denial, tool-call limit, DIFC deny, circuit-breaker open, and backend execution failure paths.

• Rate-limit outcomes are explicitly error-classified

  • When rate limiting is detected in backend tool execution, both spans now set:
    • tracing.RateLimitHit=true
    • span status = codes.Error with "rate limit exceeded"
  • This now applies to both gateway.backend.execute and the enclosing mcp.tool_call span.

• HTTP server spans now use semantic-convention route/path attributes

  • internal/tracing/http.go now always records semconv.URLPathKey from r.URL.Path.
  • semconv.HTTPRouteKey is set only when r.Pattern is available (normalized to strip method prefix and method-matched), and is omitted otherwise.
  • This preserves low-cardinality route templates for routed handlers while retaining concrete request paths for debugging.

• OTel resource now includes container detector

  • Added resource.WithContainer() in internal/tracing/provider.go so container attributes are emitted automatically when available.

• Focused coverage for route attribute behavior

  • Updated TestWrapHTTPHandler_UsesHTTPRouteWhenPatternAvailable in internal/tracing/provider_test.go to verify http.route template capture while keeping url.path.
  • Added TestWrapHTTPHandler_UsesURLPathWhenPatternUnavailable to verify fallback behavior when no route template is available.

route := r.Pattern
if method, path, ok := strings.Cut(route, " "); ok && strings.EqualFold(method, r.Method) {
    route = path
} else if ok {
    route = ""
}

attrs := []attribute.KeyValue{
    semconv.HTTPRequestMethodKey.String(r.Method),
    semconv.URLPathKey.String(r.URL.Path),
}
if route != "" {
    attrs = append(attrs, semconv.HTTPRouteKey.String(route))
}

[Copilot]

Pull request overview

This PR improves OpenTelemetry trace signal quality across the gateway by adding stack traces to recorded errors, reducing HTTP span cardinality via route templates, explicitly classifying rate-limit outcomes as errors, and enriching OTEL resources with container metadata.

Changes:

• Add oteltrace.WithStackTrace(true) to multiple RecordError call sites in internal/server/unified.go.
• Update HTTP server span attributes to prefer http.route (derived from r.Pattern) instead of url.path, and add a focused unit test for route-template capture.
• Add container resource detection via resource.WithContainer() during tracer provider initialization.

Show a summary per file

File	Description
internal/tracing/provider.go	Adds container resource detector during OTEL resource initialization.
internal/tracing/http.go	Switches HTTP span attribute from url.path to http.route, using r.Pattern when available.
internal/tracing/provider_test.go	Adds coverage to ensure route templates are captured as http.route.
internal/server/unified.go	Records stack traces on error spans and marks rate-limit execution spans as errors.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

internal/tracing/provider.go:149

• resource.New can return a partially populated Resource along with a non-nil error (e.g., one detector fails). With resource.WithContainer() added, detector failures are more likely, and the current code discards all resource attributes by replacing with resource.Empty(). Consider keeping the returned res and only logging the warning (or only falling back to resource.Empty() when res is nil/empty).

	res, err := resource.New(ctx,
		resource.WithTelemetrySDK(),
		resource.WithSchemaURL(semconv.SchemaURL),
		resource.WithContainer(),
		resource.WithAttributes(
			semconv.ServiceName(serviceName),
			semconv.ServiceVersion(version.Get()),
		),
		resource.WithProcessPID(),
		resource.WithHost(),
	)
	if err != nil {
		// Non-fatal: proceed with empty resource
		logTracing.Printf("Warning: failed to create OTEL resource: %v", err)
		res = resource.Empty()
	}

• Files reviewed: 4/4 changed files
• Comments generated: 3

[lpcox]

@copilot address review feedback

[Copilot]

@copilot address review feedback

Addressed the review feedback in commit 058ccce.

• http.route is now only set when a method-matched route template is available, while url.path is always recorded.
• Updated tracing tests to assert route-template behavior and fallback behavior when no template exists.
• Rate-limit responses now also set error status/attributes on the top-level mcp.tool_call span (in addition to backend execution span).