Audit service architecture and storage ownership

[sriaradhyula]

Related PR

• feat(audit): add lightweight audit service runtime #1946

Summary

This proposes making audit-service the owner of audit log ingestion, storage, read APIs, and storage backend selection.

Event producers send JSON audit batches to a lightweight FastAPI service. The service validates and normalizes records, places them on a bounded in-memory queue, and flushes asynchronously to either local disk or S3. If audit-service is unavailable or the queue is full, producers should warn/drop audit events and keep the product path non-blocking.

Proposed flow

flowchart LR
  subgraph PRODUCERS["Audit event producers"]
    WEB["Web UI / BFF"]
    PY["Python services"]
    BOT["Slack / Webex bots"]
    FGA["OpenFGA authz bridge"]
  end

  UI["RBAC Audit UI / operators"]
  HEALTH["health checks"]
  SVC["audit-service FastAPI"]
  QUEUE["bounded asyncio.Queue"]
  DROP["producers warn/drop<br/>non-breaking audit path"]
  WORKER["background flush worker"]
  LOCAL["local NDJSON(.gz)<br/>/var/lib/caipe-audit-service"]
  CLEANUP["local retention cleanup<br/>default 1 day"]
  S3["S3 Parquet objects<br/>minute/date partitions"]

  WEB -->|"POST /v1/audit/events<br/>JSON event batches"| SVC
  PY -->|"POST /v1/audit/events"| SVC
  BOT -->|"POST /v1/audit/events"| SVC
  FGA -->|"POST /v1/audit/events"| SVC
  UI -->|"GET /v1/audit/events"| SVC
  HEALTH -->|"GET /readyz<br/>GET /v1/audit/status"| SVC

  SVC -->|"validate + normalize"| QUEUE
  QUEUE -.->|"queue full / service down"| DROP
  QUEUE -->|"batch by size or interval"| WORKER
  WORKER -->|"AUDIT_SERVICE_BACKEND=local"| LOCAL
  LOCAL --> CLEANUP
  WORKER -->|"AUDIT_SERVICE_BACKEND=s3"| S3
  SVC -->|"read query"| LOCAL
  SVC -->|"read query"| S3

Loading

Design notes

• audit-service is the only component that reads/writes audit storage.
• UI, OpenFGA bridge, Python services, and bots only publish audit events over HTTP.
• Audit is non-breaking: if audit-service is down or overloaded, producers should warn/drop and continue.
• Local disk is the default fallback with a short retention window.
• S3 is the durable production backend and should store Parquet objects partitioned by time.
• Health/status should show whether audit-service is reachable and which storage backend is active.

Discussion points

• Should local disk stay NDJSON(.gz) for operational simplicity while S3 uses Parquet?
• What batch size and flush interval should be the default?
• Which producers must be migrated first?
• What UI health wording should we use when audit-service is unavailable but the platform is otherwise usable?