Add OpenTelemetry tracing and metrics

[hansent]

Summary

Adds OpenTelemetry distributed tracing and metrics to the inference server. Integrates with the tracing in async-serverless#192 for end-to-end traces across the system.

Disabled by default (OTEL_TRACING_ENABLED=False). Zero overhead when off. Safe when no collector is running.

Tracing

• HTTP requests — auto-instrumented via opentelemetry-instrumentation-fastapi (extracts traceparent, creates server spans)
• Outgoing requests — auto-instrumented via opentelemetry-instrumentation-requests (injects traceparent, creates http.client spans)
• Model load/infer — model.load and model.infer spans with model.id attributes
• Roboflow API calls — roboflow_api.call and roboflow_api.http_get spans
• Error recording — all route handler errors recorded on active spans
• Workflow thread propagation — OTel context forwarded through ThreadPoolExecutor
• Log correlation — trace_id/span_id injected into structlog entries
• On-demand tracing — X-Force-Trace: true header forces sampling on standalone requests; honoured parent decisions take priority

Metrics (alongside existing Prometheus)

Never sampled — 100% of requests.

Metric	Type	Attributes
inference.models.loaded	UpDownCounter	—
inference.model.loads / .unloads	Counter	model.id
inference.model.load.duration	Histogram	model.id
inference.model.infer.count	Counter	model.id
inference.model.infer.duration	Histogram	model.id
inference.roboflow_api.duration	Histogram	roboflow_api.function
inference.errors	Counter	error.type

All metrics include service.instance.id for per-pod breakdown.

Configuration

Env Var	Default	Description
OTEL_TRACING_ENABLED	False	Master switch
OTEL_SERVICE_NAME	inference-server	Service name
OTEL_EXPORTER_PROTOCOL	grpc	grpc or http
OTEL_EXPORTER_ENDPOINT	localhost:4317	OTLP collector
OTEL_SAMPLING_RATE	1.0	Trace sampling (parent-based)
OTEL_TRACE_EXPORT_INTERVAL_MS	5000	Span push interval
OTEL_METRIC_EXPORT_INTERVAL_MS	10000	Metric push interval

Local dev tool

./development/otel/start-otel-dev.sh          # Grafana + Tempo + Prometheus
./development/otel/start-otel-dev.sh --clean   # fresh start
./development/otel/start-otel-dev.sh stop

OTEL_TRACING_ENABLED=True OTEL_EXPORTER_PROTOCOL=http OTEL_EXPORTER_ENDPOINT=localhost:4318 python debugrun.py

Pre-configured dashboard at http://localhost:3000 (admin/admin).

Files changed (12 files)

File	Change
requirements/requirements.http.txt	OTel packages
inference/core/env.py	OTEL_* env vars
inference/core/telemetry.py	New — tracing, metrics, force-trace, export error handling
inference/core/interfaces/http/http_api.py	Setup, shutdown, CORS, access log
inference/core/managers/base.py	Spans + metrics on model load/infer/unload
inference/core/interfaces/http/error_handlers.py	Error recording + metric
inference/core/roboflow_api.py	API call spans + timing
inference/core/logger.py	trace_context_log_processor
inference/core/workflows/.../executor/core.py	OTel context through ThreadPool
inference_sdk/http/client.py	Cleanup (manual injection removed)
development/otel/start-otel-dev.sh	Local dev tool
development/otel/dashboard.json	Grafana dashboard

🤖 Generated with Claude Code

[ecarrara]

Since RequestsInstrumentor().instrument() is called in telemetry.py, every requests call automatically gets an HTTP client span with http.url, method, status code, etc.

This span looks redundant.

[ecarrara]

You might be able to extract the current trace_id using the otel library, something like:

# in telemetry.py

def current_trace_id():
    from opentelemetry import trace
    import opentelemetry.context as context

    current_span = trace.get_current_span()
    if current_span.span_context.is_valid:
        return format(current_span.span_context.trace_id, "032x")

[ecarrara]

Code Review: Add OpenTelemetry tracing and metrics

Strengths

• Clean API surface — all public helpers are noop-safe, so callers never need to guard against OTel being absent
• Force-trace feature (X-Force-Trace header) is a thoughtful addition for on-demand debugging in production
• Export error filter suppresses noisy connection-refused tracebacks — good operational ergonomics
• Dev tooling — the shell script + Grafana dashboard make it easy to try locally
• Zero overhead when disabled — feature-flagged at startup, not per-request

---

Issues & Suggestions

1. otel_context.attach() without detach() leaks context in thread pool (Bug)

inference/core/workflows/execution_engine/v1/executor/core.py — otel_context.attach() returns a token that must be passed to otel_context.detach(token) when the scope ends. In a ThreadPoolExecutor, threads are reused, so attached contexts accumulate:

# Current:
otel_context.attach(otel_ctx)

# Should be:
token = otel_context.attach(otel_ctx)
try:
    # ... step execution ...
finally:
    otel_context.detach(token)

This is a correctness bug — leaked contexts can cause spans in subsequent tasks on the same thread to be parented incorrectly.

2. _force_trace_flag ContextVar is never reset

_ForceTraceASGIMiddleware sets _force_trace_flag.set(True) but never resets it to False. Since ASGI runs requests in separate tasks, this is probably fine in practice (each task gets the default False). However, if any middleware or framework code reuses the same task context, the flag could leak. A try/finally reset would be safer:

async def __call__(self, scope, receive, send):
    if scope["type"] == "http":
        for header_name, header_value in scope.get("headers", []):
            if header_name == FORCE_TRACE_HEADER:
                if header_value.lower() == b"true":
                    _force_trace_flag.set(True)
                break
    try:
        await self.app(scope, receive, send)
    finally:
        _force_trace_flag.set(False)

3. Duplicate record_api_call on every exception path (DRY)

In roboflow_api.py, record_api_call(function.__name__, time.perf_counter() - t_start) is repeated in every except block plus the happy path. A try/finally would be cleaner and also ensures the metric is recorded if an unexpected exception type is raised:

try:
    try:
        result = function(*args, **kwargs)
        return result
    except RetryRequestError as error:
        raise error.inner_error
except ...:
    ...
finally:
    record_api_call(function.__name__, time.perf_counter() - t_start)

4. Access log trace_id parses raw header instead of OTel context

http_api.py — The comment says this is due to "ContextVar isolation in BaseHTTPMiddleware", which is a known Starlette issue. However, this means the access log only gets the incoming traceparent trace ID, which won't exist for server-initiated traces (e.g., X-Force-Trace without a parent). Consider documenting this limitation or using trace.get_current_span() if you can move the access log away from BaseHTTPMiddleware.

5. insecure=True hardcoded for gRPC exporter

telemetry.py — The gRPC exporter always uses insecure=True. Fine for local dev but could be a concern in production if someone points OTEL_EXPORTER_ENDPOINT at a remote collector. Consider making this configurable or defaulting based on the endpoint.

6. OTEL_SERVICE_NAME collides with standard OTel env var

The OTel SDK itself reads OTEL_SERVICE_NAME as a standard environment variable. Since you're also reading it in env.py and passing it explicitly to Resource.create(), there's a risk of confusion — the SDK's auto-configuration and your explicit config may interact unexpectedly. Consider using a prefixed name (e.g., INFERENCE_OTEL_SERVICE_NAME) or documenting that you intentionally reuse the standard var.

7. No tests

At minimum, unit tests for:

• start_span / record_error / set_span_attribute noop behavior when OTel is absent
• _ForceTraceRootSampler sampling logic
• _ExportErrorFilter suppress/reset behavior
• trace_context_log_processor structlog integration

---

Summary

Priority	Issue
Must fix	otel_context.attach() without detach() — context leak in thread pool
Should fix	record_api_call duplication → use try/finally
Should fix	Add unit tests for the telemetry module
Consider	Reset _force_trace_flag in finally block
Consider	Make gRPC insecure configurable
Consider	Rename or document OTEL_SERVICE_NAME collision with standard OTel env var