perf: Resolve the tracer once per polling handler

[keelerm84]

Summary

Stacked on #792 -- base is mk/SDK-2842/response-write-span, so the diff here is just the one commit. GitHub will retarget this to v9 when #792 merges.

tracing.Tracer() is otel.Tracer("ld-relay"), and it resolves through the global provider on every call. With tracing enabled that reaches sdktrace.TracerProvider.Tracer, which takes an exclusive p.mu.Lock() and does a map lookup keyed on an instrumentation.Scope struct; with tracing disabled it reaches the global delegating provider, which locks as well. So the cost is paid either way. The polling handlers called it three or four times per request.

Each handler now resolves the tracer once into a local and starts every span from it, including the one in traceWriteResponse, which takes the tracer as its first argument. bulkEventHandler and the two auth-middleware sites start a single span each, so they are left alone.

What it is actually worth

The review that raised this (#784 review, finding #5) put it at ~300-550 ns/request. That does not survive measurement: it is derived from the contended cost of a tracer lookup in a tight loop, which is not how a handler behaves.

One lookup in isolation, 8 cores:

	serial	parallel (tight loop)
tracing disabled	41.9 ns	100.4 ns
tracing enabled	49.8 ns	119.2 ns
hoisted local	~0	~0

End to end on the evaluation endpoint with 50 flags, median of 5 runs:

Benchmark	before	after
BenchmarkEvaluateAllFlags (serial, tracing off)	9800 ns/op	9789 ns/op
BenchmarkEvaluateAllFlagsTracedConcurrent (parallel, tracing on)	3717 ns/op	3625 ns/op

About 90 ns/request, roughly 2%, and not resolvable at all in the serial benchmark. A real handler spends microseconds between lookups, so the provider mutex is barely contended and each lookup costs closer to its uncontended ~45 ns.

This is therefore a cleanup that removes real work, not a throughput improvement, and it should not be described as one in a release note. Doing one lookup instead of four is strictly less work and reads better, which is the case for landing it.

The trap this avoids

The tracer is deliberately not memoized in a package-level variable. Relay resolves per request, and the span-recorder test helper swaps the global provider, so a memoized tracer would latch the first provider it ever saw and silently stop recording. TestHandlerTracerFollowsTheCurrentProvider installs a second recorder after the relay is already serving and asserts handler spans follow the new provider; mutating tracing.Tracer() to memoize fails it.

The new concurrent benchmark carries a comment saying plainly that it cannot resolve a 2% change and exists to exercise the traced concurrent path rather than to gate a number.

The existing span assertions in relay_endpoints_spans_test.go are untouched and still pass, which is the evidence that span names, parentage and attributes did not move.

---

Note

Low Risk
Observability-only refactor with no request semantics changes; existing span tests still cover behavior.

Overview
Polling and evaluation handlers used to call tracing.Tracer() on every span start (often three or four times per request). Each call goes through the global OTel provider and can take a lock.

Handlers now assign tr := tracing.Tracer() once per request and start all spans (store, evaluate, serialize, and response write) from that local. traceWriteResponse takes the tracer as its first argument instead of resolving it internally.

Single-span paths like bulkEventHandler are unchanged. Tracers are not cached in a package variable so a swapped global provider still applies on the next request.

TestHandlerTracerFollowsTheCurrentProvider asserts spans follow a provider installed mid-flight. BenchmarkEvaluateAllFlagsTracedConcurrent exercises the concurrent traced evaluation path; the change is a small cleanup, not a major throughput win.

^{Reviewed by Cursor Bugbot for commit 3bda603. Bugbot is set up for automated code reviews on this repo. Configure here.}