Summary
The ateapi postgres store (cmd/ateapi/internal/store/atepg/) creates its pgxpool without any tracer hook, so no span is ever emitted for database work. Every RPC trace ends at the gRPC server span produced by the otelgrpc stats handler; the time spent in PostgreSQL is indistinguishable from handler/auth overhead.
Observed
On a live GKE install with tracing enabled end to end:
kubectl ate get actor <name> -a <atespace> --trace produces a trace with exactly one span: ateapi.Control/GetActor (~97 ms in our run). For a single-row read that is slow enough to want a breakdown, and the trace offers none.- This is not specific to reads: rich mutation traces (25-span suspend, 33-span resume) contain
step.*, atelet, ateom, and GCS client spans — but zero postgres spans, despite every one of those flows doing multiple DB round-trips (actor load, state transitions, outbox).
Root cause
cmd/ateapi/internal/store/atepg/atepg.go has no OTel/tracing references at all; the pool is built from pgxpool.ParseConfig with ConnConfig.Tracer unset.- The only vendored OTel instrumentation libraries are
otelgrpc and net/http — there is no pgx instrumentation in the dependency tree.
Suggested fix
Attach a tracer via pgx v5's native hook: set config.ConnConfig.Tracer on the pool config, either with otelpgx or a small in-tree pgx.QueryTracer that opens spans from the request context. Spans should parent under the active RPC span so GetActor shows db.query children with statement names.
Related test-documentation nit
A bug-bash test we ran expects kubectl ate get actor --trace to also capture a "worker handshake". That expectation is wrong by design: GetActor is a pure store read (cmd/ateapi/internal/controlapi/actor.go → s.store.GetActor), and worker spans (step.AssignWorker, atelet.AteomHerder/*) only appear on mutation traces. Worth correcting wherever that test recipe lives, but the actionable gap in this issue is the missing DB spans.
Summary
The ateapi postgres store (
cmd/ateapi/internal/store/atepg/) creates itspgxpoolwithout any tracer hook, so no span is ever emitted for database work. Every RPC trace ends at the gRPC server span produced by theotelgrpcstats handler; the time spent in PostgreSQL is indistinguishable from handler/auth overhead.Observed
On a live GKE install with tracing enabled end to end:
kubectl ate get actor <name> -a <atespace> --traceproduces a trace with exactly one span:ateapi.Control/GetActor(~97 ms in our run). For a single-row read that is slow enough to want a breakdown, and the trace offers none.step.*, atelet, ateom, and GCS client spans — but zero postgres spans, despite every one of those flows doing multiple DB round-trips (actor load, state transitions, outbox).Root cause
cmd/ateapi/internal/store/atepg/atepg.gohas no OTel/tracing references at all; the pool is built frompgxpool.ParseConfigwithConnConfig.Tracerunset.otelgrpcandnet/http— there is no pgx instrumentation in the dependency tree.Suggested fix
Attach a tracer via pgx v5's native hook: set
config.ConnConfig.Traceron the pool config, either withotelpgxor a small in-treepgx.QueryTracerthat opens spans from the request context. Spans should parent under the active RPC span soGetActorshowsdb.querychildren with statement names.Related test-documentation nit
A bug-bash test we ran expects
kubectl ate get actor --traceto also capture a "worker handshake". That expectation is wrong by design:GetActoris a pure store read (cmd/ateapi/internal/controlapi/actor.go→s.store.GetActor), and worker spans (step.AssignWorker,atelet.AteomHerder/*) only appear on mutation traces. Worth correcting wherever that test recipe lives, but the actionable gap in this issue is the missing DB spans.