Proposal: in-application event framework — design + Phase 1 plumbing

[adamsachs]

Proposal for design review. Primary goal is to gather feedback on the design doc and align on the framework's shape before any production code consumes it. The plumbing implementation is included so reviewers can see the design landed end-to-end, but it is gated behind a kill-switch (CONFIG.events.enabled) and ships with no production publishers or subscribers wired in.

Ticket: related to https://ethyca.atlassian.net/browse/ENG-3325 and https://ethyca.atlassian.net/browse/ENG-3510

What to review (in priority order)

1. docs/fides/docs/event-framework/01-design.md — the framework's design, prescriptive when-to-use boundary, transport choice rationale (Celery + existing Redis broker; alternatives explicitly considered and rejected), failure modes, and open questions. This is the most important file in the PR.
2. docs/fides/docs/event-framework/02-implementation-plan.md — phased rollout plan. Phase 1 is this PR; Phases 2 and 3 are the first two consumers (monitor stewardship inheritance + monitor stats refresh).
3. docs/fides/docs/event-framework/03-phase-1-plumbing.md — concrete checklist for the framework PR: package layout, per-component acceptance bars, the canonical end-to-end ping validation, coverage check pattern.
4. docs/fides/docs/event-framework/04-stewardship-implementation-plan.md — self-contained plan for the first feature consumer (monitor stewardship inheritance), designed so another agent can pick it up on a fresh branch once dependencies land.
5. The plumbing code — src/fides/common/events/ and the test suite. Implementation reflects the design but the design is the artifact under review.

Description Of Changes

A new in-application event framework for asynchronous reactive work. Publishers emit typed events when application state changes; the framework dispatches registered handlers via Celery to recompute / propagate / refresh derived state. Goals:

• Decouple publishers from consumers via a registry, so adding a new subscriber doesn't require modifying every action site.
• Standardize observability, retry, and reconciliation patterns across features.
• Establish prescriptive boundaries (when to use, when not) so the framework doesn't get misused.

Key design decisions (covered in detail in §1–5 of the design doc):

• Transport: Celery, with the existing Redis broker. No new infrastructure.
• Programming model: typed events as frozen dataclasses + @subscribes_to decorator-registered handlers + autodiscovery. Two-line API for subscribers.
• Operational home (default): fides Celery queue, picked up by the existing "Other" worker. Per-handler queue override available.
• Delivery semantics: at-least-once, not guaranteed — explicit non-goal. Subscribers must be idempotent against current DB state. Where correctness matters, an independent reconciler is required.
• Where the bones live: fides OSS. fidesplus subscribes from across the boundary.

Non-goals (the prescriptive part — see §2.1):

• Does not guarantee delivery.
• Does not provide ordering guarantees.
• Does not detect missed publishes.
• Is not a replacement for synchronous calls.
• Is not pub/sub on the wire.

Code Changes

• New package src/fides/common/events/ (10 files, ~700 lines):
  • base.py — frozen-dataclass + JSON-primitive validation; rejects UUID/datetime/Enum at decoration time.
  • registry.py — in-process registry of subscribers + @publishes markers.
  • decorators.py — @subscribes_to(EventType, queue="fides") and @publishes(*EventType).
  • publisher.py — publish_after_commit(session, event) with after-commit / after-rollback session hooks.
  • dispatcher.py — fan-out via task.apply_async; works in eager mode + production; per-subscriber failure isolation; kill-switch via CONFIG.events.enabled.
  • celery_integration.py — task wrapper factory; event_id passed as explicit task kwarg (not headers, so it works in eager mode + production identically).
  • context.py — ContextVar exposing get_current_event_id() to subscribers.
  • autodiscover.py — register_subscriber_module + import_subscriber_modules for cross-repo registration.
  • testing.py — published_events, assert_event_published, clear_pending_events.
  • __init__.py — public API.
• New config section: src/fides/config/event_settings.py (EventSettings.enabled: bool = True), wired into FidesConfig.
• Edits to src/fides/api/tasks/__init__.py (+14 lines): import celery_integration; after celery_app is created, call import_subscriber_modules() so subscribers register at boot.
• Tests under tests/common/events/ (11 files, ~800 lines):
  • _ping.py — canonical reference fixture (event + subscriber + repository) demonstrating the recommended publish-from-repository pattern.
  • test_ping.py — 8 end-to-end tests: commit, rollback, ordering, kill-switch, marker recording, event_id flow-through, no-subscriber path.
  • Per-module unit tests for validation, registry, decorators, publisher, dispatcher.
  • test_coverage_checks.py — hard-fail check that every production subscriber's event type has an in-tree @publishes marker, with a documented exclude-list pattern.
• Four design docs under docs/fides/docs/event-framework/.

42 tests passing. No production publishers or subscribers consume the framework yet — that's intentional.

Steps to Confirm

The framework's correctness is demonstrated entirely via the test suite — no production code path consumes it yet, so there's nothing to manually click through.

1. Run the test suite:

   pytest tests/common/events/ -v

   Expected: 42 passing.
2. Verify the kill switch works: CONFIG.events.enabled = False should make publish_after_commit queue events on the session but skip dispatch on commit. Covered by test_ping.py::test_kill_switch_disables_dispatch.
3. Verify rollback safety: events queued on a rolled-back session should never fire. Covered by test_ping.py::test_publish_then_rollback_does_not_dispatch.
4. Read the design doc and tell us where it's wrong, vague, or missing constraints we'd regret leaving out.

Pre-Merge Checklist

• Design doc reviewed and approved by the team
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration label to the entry if your change includes a DB migration — N/A, no migrations
  • Add a high-risk label to the entry if your change includes a high-risk change — N/A, gated behind kill-switch, no production callers
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • No UX review needed (no UI changes)
• Followup issues:
  • Followup issues created (Phase 2 stewardship + Phase 3 stats; ticket creation deferred until design lands)
  • No followup issues
• Database migrations:
  • No migrations
• Documentation:
  • Design doc + implementation plan + Phase 1 plumbing checklist + stewardship implementation plan all included in PR
  • Documentation issue created in fidesdocs
  • No new client scopes
  • No documentation updates required

Open Questions for Reviewers

These are surfaced explicitly in 01-design.md §10 — please weigh in:

1. @publishes marker enforcement strictness: required (CI-enforced) or recommended? Currently recommended; v1 leans soft.
2. Reconciler cadence for stewardship inheritance: starting at 15 minutes — too tight, too loose, or right?
3. Stats event granularity: single MonitorOperationCompleted(monitor_config_key, operation_type) or one event class per operation? Leaning the simpler model.
4. Event package layout: fides.api.events.<domain> for fides-emitted; fidesplus.api.events.<domain> for fidesplus-emitted. Confirm before Phase 2.
5. event_id provenance: generated at publish time, shared across subscribers for the same publish. Confirmed in design — flag if a different model is preferred.

Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ignored		May 4, 2026 2:37pm
fides-privacy-center	Ignored		May 4, 2026 2:37pm