Instrumentation

Instrumentation

The controller framework provides comprehensive instrumentation capabilities for observability, including logging, tracing, and metrics collection. This system allows you to monitor your controllers' behavior, debug issues, and gain insights into performance characteristics.

Instrumentation can be used without using the full controller framework, making it a flexible choice for various applications. It does not rely on the framework's context nor generic types.

Overview

The instrumentation system is built around the Instrumenter interface, which provides:

• Structured logging with contextual information
• Distributed tracing for request flow visualization
• Context management for request-scoped data
• Integration with popular observability tools

Core Components

Instrumenter Interface

The Instrumenter interface defines the contract for instrumentation providers:

type Instrumenter interface {
    InstrumentRequestHandler(handler handler.TypedEventHandler[client.Object, reconcile.Request]) handler.TypedEventHandler[client.Object, reconcile.Request]
    GetContextForRequest(req reconcile.Request) (*context.Context, bool)
    GetContextForEvent(event any) *context.Context
    NewQueue(mgr ctrl.Manager) func(controllerName string, rateLimiter workqueue.TypedRateLimiter[reconcile.Request]) workqueue.TypedRateLimitingInterface[reconcile.Request]
    Cleanup(ctx *context.Context, req reconcile.Request)
    NewLogger(ctx context.Context) logr.Logger
    Tracer
}

type Tracer interface {
    StartSpan(globalCtx *context.Context, localCtx context.Context, spanName string, opts ...trace.SpanStartOption) (context.Context, trace.Span)
}

InstrumentedReconciler

The InstrumentedReconciler wraps your reconciler with instrumentation capabilities:

type InstrumentedReconciler struct {
    internalReconciler reconcile.TypedReconciler[reconcile.Request]
    Instrumenter
}

Setup and Configuration

Basic Setup

1. Create an Instrumenter: Build an instrumenter with your desired configuration
2. Wrap your reconciler: Use InstrumentedControllerManagedBy to create an instrumented controller
3. Configure observability backends: Set up your logging and tracing providers

Builder Pattern

The framework provides a builder pattern for easy configuration:

instrumenter := instrument.NewInstrumenter(mgr).
    WithLoggerFunc(loggerFunc).
    WithTracer(tracer).
    Build()

Sentry Integration

Sentry provides both error tracking and performance monitoring through logging and tracing.

Prerequisites

Add Sentry dependencies to your go.mod:

go get github.com/getsentry/sentry-go
go get github.com/getsentry/sentry-go/otel

Configuration

package main

import (
    "github.com/getsentry/sentry-go"
    sentryotel "github.com/getsentry/sentry-go/otel"
    "go.opentelemetry.io/otel"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    
    "github.com/u-ctf/controller-fwk/instrument"
)

func main() {
    // Initialize Sentry
    sentry.Init(sentry.ClientOptions{
        Dsn:              "https://your-sentry-dsn@sentry.io/project-id",
        EnableTracing:    true,
        TracesSampleRate: 1.0, // Adjust sampling rate for production
        Environment:      "production", // or "development", "staging"
        Release:          "v1.0.0",
    })

    // Set up OpenTelemetry with Sentry
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithSpanProcessor(sentryotel.NewSentrySpanProcessor()),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(sentryotel.NewSentryPropagator())

    // Create instrumenter with Sentry logger and tracer
    instrumenter := instrument.NewInstrumenter(mgr).
        WithLoggerFunc(instrument.NewSentryLoggerFunc(ctrl.Log.WithName("controller"))).
        WithTracer(instrument.NewSentryTracer(tp.Tracer("controller"))).
        Build()

    // Use in your reconciler setup
    if err := (&YourReconciler{
        Client:        mgr.GetClient(),
        Instrumenter:  instrumenter,
        EventRecorder: mgr.GetEventRecorderFor("your-controller"),
        RuntimeScheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller")
        os.Exit(1)
    }
}

Important Notes for Sentry

⚠️ Logger and Tracer Must Be Used Together: Currently, the Sentry logger and tracer must be used as a pair. Even if you disable tracing in Sentry's client options (EnableTracing: false), you still need to configure both the logger and tracer components for the instrumentation to work correctly.

// ✅ Correct - Use both logger and tracer together
instrumenter := instrument.NewInstrumenter(mgr).
    WithLoggerFunc(instrument.NewSentryLoggerFunc(baseLogger)).
    WithTracer(instrument.NewSentryTracer(tracer)).
    Build()

// ❌ Incorrect - Don't use only one component
instrumenter := instrument.NewInstrumenter(mgr).
    WithLoggerFunc(instrument.NewSentryLoggerFunc(baseLogger)).
    // Missing tracer - will cause issues
    Build()

Sentry Features

With Sentry integration, you get:

• Error Tracking: Automatic capture of reconciliation errors with stack traces
• Performance Monitoring: Trace reconciliation performance across requests
• Release Tracking: Link errors and performance to specific releases
• Context: Rich context including Kubernetes resource information

Jaeger Integration

Jaeger provides distributed tracing capabilities for visualizing request flows.

Prerequisites

Add OpenTelemetry Jaeger dependencies:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/exporters/jaeger
go get go.opentelemetry.io/otel/sdk/trace

Configuration

package main

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
    
    "github.com/u-ctf/controller-fwk/instrument"
)

func setupJaeger() (*sdktrace.TracerProvider, error) {
    // Create Jaeger exporter
    jaegerExporter, err := jaeger.New(
        jaeger.WithCollectorEndpoint(
            jaeger.WithEndpoint("http://localhost:14268/api/traces"),
        ),
    )
    if err != nil {
        return nil, err
    }

    // Create resource with service information
    res, err := resource.New(context.Background(),
        resource.WithAttributes(
            semconv.ServiceNameKey.String("my-controller"),
            semconv.ServiceVersionKey.String("v1.0.0"),
            semconv.DeploymentEnvironmentKey.String("production"),
        ),
    )
    if err != nil {
        return nil, err
    }

    // Create tracer provider
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(jaegerExporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.AlwaysSample()), // Adjust for production
    )

    return tp, nil
}

func main() {
    // Set up Jaeger tracing
    tp, err := setupJaeger()
    if err != nil {
        setupLog.Error(err, "failed to setup Jaeger")
        os.Exit(1)
    }
    defer func() { _ = tp.Shutdown(context.Background()) }()

    otel.SetTracerProvider(tp)

    // Create instrumenter with Jaeger tracer
    instrumenter := instrument.NewInstrumenter(mgr).
        WithLoggerFunc(func(ctx context.Context) logr.Logger {
            return ctrl.Log.WithName("controller")
        }).
        WithTracer(instrument.NewOtelTracer(tp.Tracer("controller"))).
        Build()

    // Use in your reconciler
    if err := (&YourReconciler{
        Client:        mgr.GetClient(),
        Instrumenter:  instrumenter,
        EventRecorder: mgr.GetEventRecorderFor("your-controller"),
        RuntimeScheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller")
        os.Exit(1)
    }
}

Jaeger Features

With Jaeger integration, you get:

• Distributed Tracing: Visualize request flows across services
• Performance Analysis: Identify bottlenecks and latency issues
• Service Dependencies: Understand service interaction patterns
• Custom Spans: Add custom instrumentation points

Controller Integration

Using Instrumented Controllers

Replace the standard controller builder with the instrumented version:

func (r *YourReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return instrument.InstrumentedControllerManagedBy(r.Instrumenter, mgr).
        For(&yourv1.YourResource{}).
        Named("your-controller").
        Build(r)
}

Reconciler Implementation

Your reconciler can access instrumentation through the embedded Instrumenter:

type YourReconciler struct {
    client.Client
    instrument.Instrumenter
    record.EventRecorder
    RuntimeScheme *runtime.Scheme
    // ... other fields
}

func (r *YourReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Instrumentation is automatically applied by InstrumentedReconciler
    // The logger is automatically enhanced with tracing context
    
    // Your reconciliation logic here
    // Logger is available via logf.FromContext(ctx)
    
    return ctrl.Result{}, nil
}

Advanced Configuration

Custom Logger Functions

Create custom logger functions for specific needs:

func customLoggerFunc(ctx context.Context) logr.Logger {
    baseLogger := ctrl.Log.WithName("my-controller")
    
    // Add custom context information
    if traceID := getTraceIDFromContext(ctx); traceID != "" {
        baseLogger = baseLogger.WithValues("traceId", traceID)
    }
    
    return baseLogger
}

instrumenter := instrument.NewInstrumenter(mgr).
    WithLoggerFunc(customLoggerFunc).
    Build()

Multiple Instrumenters

You can create different instrumenters for different controllers:

// High-volume controller with sampling
highVolumeInstrumenter := instrument.NewInstrumenter(mgr).
    WithLoggerFunc(instrument.NewSentryLoggerFunc(ctrl.Log)).
    WithTracer(instrument.NewSentryTracer(sampledTracer)).
    Build()

// Critical controller with full tracing
criticalInstrumenter := instrument.NewInstrumenter(mgr).
    WithLoggerFunc(instrument.NewSentryLoggerFunc(ctrl.Log)).
    WithTracer(instrument.NewSentryTracer(fullTracer)).
    Build()

Best Practices

Performance Considerations

1. Sampling: Use appropriate sampling rates for high-volume environments
2. Context Cleanup: The framework automatically handles context cleanup
3. Resource Management: Instrumenters manage their own resources

Error Handling

1. Structured Errors: Use structured logging for better error analysis
2. Context Preservation: Maintain context across async operations
3. Graceful Degradation: Handle instrumentation failures gracefully

Security

1. Sensitive Data: Avoid logging sensitive information
2. Network Security: Secure connections to observability backends
3. Authentication: Use proper authentication for observability services

Troubleshooting

Common Issues

1. Missing Traces: Ensure tracer provider is properly configured
2. Context Loss: Verify context is properly propagated
3. High Memory Usage: Check sampling rates and batch sizes

Debugging

Enable debug logging to troubleshoot instrumentation issues:

ctrl.SetLogger(zap.New(zap.UseDevMode(true)))

Migration Guide

From Manual Instrumentation

If you're currently using manual instrumentation:

1. Remove manual logger and tracer setup from reconcilers
2. Configure instrumenter at the manager level
3. Use InstrumentedControllerManagedBy instead of standard builder
4. Update reconciler struct to embed instrument.Instrumenter
5. Pass the instrumenter as the first argument to InstrumentedControllerManagedBy

Version Compatibility

The instrumentation system is compatible with:

• controller-runtime v0.15+
• OpenTelemetry Go v1.19+
• Go 1.19+