Why
High-volume deployments cannot export every record. Sampling must be coherent: a request's payload record and its metadata record have to be kept or dropped together, or the export is useless for both audit and replay.
Scope
- Apply sampling after record suppression. Suppressed and non-suppressed records are sampled identically.
- Declare the sampling key per record type rather than as a cross-type fallback chain.
request_end uses request.request_id, request_payload uses payload.request_id, and the three tool events use agent_context.session_id. A fallback chain across these would mix identifier namespaces and break coherence, because a session contains many requests.
- Make the decision a deterministic function of the key alone, so records sharing a key are kept or dropped identically across record types, segments, and uploader restarts. Per-record randomness is not acceptable: a request's payload and its
request_end are written to the same file family but may land in different segments uploaded at different times, and determinism is what makes them agree without shared state.
- Document that tool events carry no request identifier, so tool sampling is coherent at session granularity and does not join to request-level sampling.
- Define policy for records with no resolvable key: keep, drop, or reject at config validation. Default keep.
- Emit
records_sampled_out_total by record type and reason, where reason is rate or no_key.
Tests
- Deterministic keep or drop for the same key across record types, segments, and process restarts.
- A request's
request_payload and request_end receive the same decision.
- Tool events key on session and are unaffected by request-level decisions.
- Missing-key policy under each setting.
- Rate boundaries at 0.0 and 1.0.
Note
Record-level sampling requires rewriting each segment, which forfeits byte-exact jsonl.gz passthrough on the object-store backend. If passthrough matters more than granularity, segment-level sampling is the alternative, but it cannot give coherent per-request retention.
Parent: #1004
Why
High-volume deployments cannot export every record. Sampling must be coherent: a request's payload record and its metadata record have to be kept or dropped together, or the export is useless for both audit and replay.
Scope
request_endusesrequest.request_id,request_payloadusespayload.request_id, and the three tool events useagent_context.session_id. A fallback chain across these would mix identifier namespaces and break coherence, because a session contains many requests.request_endare written to the same file family but may land in different segments uploaded at different times, and determinism is what makes them agree without shared state.records_sampled_out_totalby record type and reason, where reason is rate or no_key.Tests
request_payloadandrequest_endreceive the same decision.Note
Record-level sampling requires rewriting each segment, which forfeits byte-exact
jsonl.gzpassthrough on the object-store backend. If passthrough matters more than granularity, segment-level sampling is the alternative, but it cannot give coherent per-request retention.Parent: #1004