Tail-drop empty/up-to-date shape-GET spans at the origin to cut OTel event volume

[alco]

Summary

Drop (or down-sample) the OpenTelemetry spans for empty / up-to-date shape-GET responses at the origin, the way the Cloudflare stream-worker already does for its own spans in electric-sql/stratovolt#1282. This is the single biggest remaining lever on the electric-region event volume.

Part of the Honeycomb event-budget effort tracked in electric-sql/stratovolt#1610 (goal: downgrade to the 450M Pro tier).

Why — the data

Measured in production electric-region (2026-06-30, post span-flatten, raw/billed counts):

• Plug_shape_get (the request root) is ~81% of all electric-region trace spans (~361M/mo).

• 95.1% of those roots are empty responses — shape_req.is_empty_response = true /
  shape_req.is_up_to_date = true:

  is_empty_response	is_up_to_date	share of Plug_shape_get
  true	true	95.1%
  false	true	4.7%
  false	false	0.16%

So ~343M events/month are root spans (plus their descendants) for long-poll requests that returned no new data. These carry essentially no diagnostic value — the equivalent worker-side spans are ~93% of cloudflare-shape-api, which #1282 drops losslessly.

Key point: the signal is already computed

packages/sync-service/lib/electric/plug/serve_shape_plug.ex already stamps the root span:

"shape_req.is_empty_response" => attrs[:ot_is_empty_response] || false,   # ~line 557
"shape_req.is_up_to_date"     => maybe_up_to_date                          # ~line 561

We don't need to detect anything new — we just need to act on is_empty_response at export time.

The wrinkle: this needs a tail decision

Electric.Telemetry.Sampler.include_span?/1 samples at span creation (head-based, by name only):

def include_span?(name), do: !excluded?(name) && included?(name)

But emptiness is only known at span end. So a head-based sampler can't make this call. We need a tail mechanism — e.g. a span processor / exporter filter in the OTel pipeline that drops a root span and its descendants when shape_req.is_empty_response == true. The worker precedent (#1282) does exactly this via its per-invocation span buffer + flushTrace.

Proposed approach (to refine)

• Add a tail filter in Electric's OTel export pipeline that, for Plug_shape_get roots with is_empty_response = true, drops the trace (root + children like stream_chunk).
• Config-gate it as a sampling ratio rather than all-or-nothing — e.g. ELECTRIC_EMPTY_SPAN_SAMPLE_RATIO to keep ~1:N empties so we retain latency/health visibility on the up-to-date path while stamping SampleRate for correct counts (mirror the existing keep-on-error semantics from Read and apply the SampleRate attribute to shape spans from the tracestate request header #4562).
• Always keep non-empty responses and all errors (≥500 already keep-on-error).
• Verify children of dropped empty roots are not orphaned/exported.

Expected win

~343M events/month (~95% of Plug_shape_get, ~80% of electric-region trace, ~34% of the current ~1.0B/mo run-rate) — losslessly, and complementary to per-source sampling (stratovolt#1609): dropping empties is strictly preferable to sampling them and reduces the need for aggressive per-source rates.

Pointers

• packages/sync-service/lib/electric/telemetry/sampler.ex — current head sampler
• packages/sync-service/lib/electric/telemetry/open_telemetry.ex — with_span / with_child_span
• packages/sync-service/lib/electric/plug/serve_shape_plug.ex — already stamps the attributes
• packages/sync-service/config/runtime.exs — OTel exporter/processor + ELECTRIC_EXCLUDE_SPANS wiring
• Prior art: stratovolt#1282 (worker-side empty-response drop), electric#4562 (keep-on-error + SampleRate)