chore(logging): reduce noise, route only audit to OTel

[alexluong]

Tighten Outpost logging. Three changes, each independently revertable.

Reduce noise. Per-event/per-request handler lines in deliverymq, publishmq, logmq, and log retention demoted info→debug. API request logs only emit at 400+ (4xx info, 5xx error, 2xx/3xx debug). Webhook 5xx/timeouts no longer log as Error — they're expected operational outcomes already captured by the audit and ClickHouse log.

Adopt wide events for audit. Replace the multi-line audit pattern (processing event, delivery task enqueued, retry scheduled, scheduled retry canceled, delivery attempt completed) with one rich event per unit of work: event.received and delivery.attempted. Each carries the full outcome (status, timings, retry decision, IDs) so consumers don't have to join across lines or worry about ordering. References:

• https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/
• https://stripe.com/blog/canonical-log-lines\
• Question: should we add an audit log for every API request?

Split sinks. Only Audit() lines flow to the OTel logs SDK; regular Info/Debug/Warn/Error stay on stdout. Operator-facing diagnostics and debug noise no longer leak to the customer-visible OTel sink. Trace correlation in stdout logs is preserved manually via trace_id/span_id fields.

Example logs — event delivery scenarios

Happy path — event published, matched 1 destination, delivered successfully

INFO  "event.received"      { event_id, tenant_id, topic, matched_destination_ids: [d1], enqueued_destination_ids: [d1], matched_destination_count: 1, enqueued_destination_count: 1, duplicate: false, event_received_at, duration_ms: 7 }
DEBUG "processing delivery task" { event_id, tenant_id, destination_id: d1, attempt: 1 }
INFO  "delivery.attempted"  { attempt_id, event_id, tenant_id, topic, destination_id: d1, destination_type: "webhook", attempt_status: "success", attempt_code: "200", attempt_number: 1, attempt_max: 11, manual: false, eligible_for_retry: true, attempt_started_at, attempt_duration_ms: 142, retry_id, retry_scheduled: false, retry_canceled: false }
DEBUG "logmq handler"       { message_id }
DEBUG "added to batch"      { message_id, event_id, attempt_id, tenant_id }
DEBUG "processing batch"    { message_count: 1 }
INFO  "batch persisted"     { count: 1, insert_duration_ms: 8 }

Webhook returns 5xx, retry eligible

INFO  "event.received"      { ... }
DEBUG "processing delivery task" { ..., attempt: 1 }
INFO  "delivery.attempted"  { ..., attempt_status: "failed", attempt_code: "500", attempt_duration_ms: 1247, retry_scheduled: true, retry_backoff_ms: 30000, retry_canceled: false }
# logmq sequence as above

Webhook 5xx, retry budget exhausted

DEBUG "processing delivery task" { ..., attempt: 11 }
INFO  "delivery.attempted"  { ..., attempt_status: "failed", attempt_number: 11, attempt_max: 11, retry_scheduled: false, retry_canceled: false }

Manual retry, success

DEBUG "processing delivery task" { ..., manual: true }
INFO  "delivery.attempted"  { ..., attempt_status: "success", manual: true, retry_canceled: true }

Destination deleted / not found / disabled

DEBUG "processing delivery task" { ... }
DEBUG "destination deleted"      { event_id, tenant_id, destination_id }
# message ack'd, no delivery.attempted, no logmq entry

Idempotency hit (duplicate delivery task)

DEBUG "processing delivery task" { ... }
DEBUG "delivery task skipped (idempotent)" { ..., idempotency_key }
# no delivery.attempted, no logmq entry

No matching destinations

INFO  "event.received"      { ..., matched_destination_count: 0, enqueued_destination_count: 0 }
# nothing enqueued, no downstream activity

System failures (operator-actionable)

# publishmq, MatchEvent fails (DB/Redis down)
ERROR "failed to match event destinations" { error, event_id, tenant_id }
INFO  "event.received"      { ..., match_failed: true }

# publishmq, delivery MQ broken
ERROR "failed to enqueue delivery task" { error, event_id, tenant_id, destination_id }
INFO  "event.received"      { ..., enqueue_failed: true, enqueued_destination_count: 0 }

# deliverymq, scheduler unreachable
ERROR "failed to schedule retry"       { ... }
INFO  "delivery.attempted"  { ..., retry_schedule_failed: true }
ERROR "consumer handler error"         { name, error }

# deliverymq, logmq broken
ERROR "failed to publish attempt log"  { ... }
ERROR "consumer handler error"         { ... }

# logmq, ClickHouse insert fails
ERROR "failed to insert log entries"   { error, entry_count, insert_duration_ms }