Skip to content

feat(request-trace-uploader): add configurable per-record-type sampling #1444

Description

@kristinapathak

Why

High-volume deployments cannot export every record. Sampling must be coherent: a request's payload record and its metadata record have to be kept or dropped together, or the export is useless for both audit and replay.

Scope

  • Apply sampling after record suppression. Suppressed and non-suppressed records are sampled identically.
  • Declare the sampling key per record type rather than as a cross-type fallback chain. request_end uses request.request_id, request_payload uses payload.request_id, and the three tool events use agent_context.session_id. A fallback chain across these would mix identifier namespaces and break coherence, because a session contains many requests.
  • Make the decision a deterministic function of the key alone, so records sharing a key are kept or dropped identically across record types, segments, and uploader restarts. Per-record randomness is not acceptable: a request's payload and its request_end are written to the same file family but may land in different segments uploaded at different times, and determinism is what makes them agree without shared state.
  • Document that tool events carry no request identifier, so tool sampling is coherent at session granularity and does not join to request-level sampling.
  • Define policy for records with no resolvable key: keep, drop, or reject at config validation. Default keep.
  • Emit records_sampled_out_total by record type and reason, where reason is rate or no_key.

Tests

  • Deterministic keep or drop for the same key across record types, segments, and process restarts.
  • A request's request_payload and request_end receive the same decision.
  • Tool events key on session and are unaffected by request-level decisions.
  • Missing-key policy under each setting.
  • Rate boundaries at 0.0 and 1.0.

Note

Record-level sampling requires rewriting each segment, which forfeits byte-exact jsonl.gz passthrough on the object-store backend. If passthrough matters more than granularity, segment-level sampling is the alternative, but it cannot give coherent per-request retention.

Parent: #1004

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions