Skip to content

Add audit trail integration and migrate NATS publisher to protobuf #19

@haasonsaas

Description

@haasonsaas

Context

Two gaps in siphon's production readiness:

1. No audit trail

Siphon has zero calls to the shared audit service. Every ingested webhook, every DLQ event, every poller fetch is invisible to the audit trail. This is notable for compliance-sensitive sources (Stripe payment events, Salesforce CRM data, HubSpot contacts).

2. NATS publisher still uses JSON (issue #12)

internal/publish/nats.go uses json.Marshal(event) for the CloudEvent payload. The proto types (NormalizedEvent, TapEventData) are defined in proto/tap/v1/event.proto but the publisher hasn't been migrated to protobuf encoding. Issue #12 is substantively open despite the proto definitions landing.

Requirements

Audit integration

  • Emit audit event per ingested webhook: provider, tenant, entity type, dedup ID, outcome (accepted/rejected/DLQ)
  • Emit audit event for DLQ replay operations
  • Emit audit event for poller fetch cycles (provider, records fetched, cursor position)
  • Use ConnectRPC client to audit.v1 (fire-and-forget with retry)

Proto migration

  • Migrate internal/publish/nats.go from json.Marshal(cloudEvent) to protobuf-encoded TapEventData
  • Update all downstream consumers (pipeline tap-consumer, etc.) to expect proto encoding
  • Add a format header to NATS messages for backward compatibility during migration

Operational improvements

  • Make shutdown timeout configurable (currently hardcoded 10*time.Second in run.go)
  • Enable HPA by default in Helm values for production profiles (webhook ingestion is bursty)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions