Stop persisting SDK traces from background tracker polling (or split PERSIST_SDK_TRACING)

[minchaminder]

Problem

On a production self-hosted deployment we traced recurring Postgres CPU alerts to the tracing-record table growing without bound. Root cause: background tracker polling persists full SDK traces on every carrier API call, separate from user-initiated API request tracing.

Two write paths exist today:

Path	Entry point	Meta shape	Dedup
API middleware	save_tracing_records(context, tracer)	request_log_id, carrier_*, optional object_id	Yes (by request_log_id)
Background tracker job	bulk_save_tracing_records(gateway.tracer, context) via _save_tracing() / save_tracing_records(request_batches)	connection only	No

In our prod DB (~5.3M tracing rows, ~11 GB table):

• ~93% of rows are from the tracker job (meta ? 'connection' and no request_log_id)
• ~7% from real API debugging (meta ? 'request_log_id')
• meta.object_id is never populated on tracker traces, so debugging falls back to record::text ILIKE → full table scans on multi-million-row tables

Steady-state insert rate from tracker polling alone: ~50k rows/day (1,300 undelivered trackers × ~2h poll interval × request/response/error keys). Catch-up after worker restarts can spike to ~100k rows/hour.

Upstream tracking.py still calls _save_tracing() after each batch (see process_carrier_trackers → _process_batch).

Why the current global flag is not enough

PERSIST_SDK_TRACING gates both paths. Turning it off fixes DB pressure but also disables useful API-side traces for shipment/rate/label debugging.

Docs even suggest PERSIST_SDK_TRACING=false in production for performance — but that throws away the traces operators actually need.

Proposal

Pick one (or combine):

1. PERSIST_TRACKER_TRACING (default false) — new setting; background Huey tracker job skips bulk_save_tracing_records while API middleware keeps tracing.
2. Stop persisting tracker traces entirely — tracking status updates are the product; raw carrier HTTP dumps on every poll are rarely useful and are not linked to tracker_id today anyway.
3. If persistence is kept, minimum fixes:
   • Set meta.object_id / tracker_id on every batch row
   • Dedup per tracker per poll window
   • Skip or cap error traces for known-broken carrier polls

Workaround today

• PERSIST_SDK_TRACING=false on worker container only (API container keeps tracing)
• Or remove/guard _save_tracing() / save_tracing_records(request_batches) in a fork

Questions for maintainers

• Was background tracker trace persistence intentional for debugging, or inherited when bulk_save_tracing_records was added?
• Would a PERSIST_TRACKER_TRACING flag (default off) be acceptable upstream?
• Any objection to not persisting failed carrier poll traces when delivered=false and poll count exceeds N?

Happy to open a PR if there is agreement on direction. We have a detailed RCA from prod data if useful.

Code refs (upstream):

• modules/events/karrio/server/events/task_definitions/base/tracking.py — _save_tracing()
• modules/core/karrio/server/tracing/utils.py — bulk_save_tracing_records, PERSIST_SDK_TRACING guard