[Design Discussion] Generic eventing framework

[ThumulaPerera]

Related Feature Issue

#5588

Problem Summary

Today every event has the same shape (a few common fields plus a key-value map), there's no taxonomy, and delivery is best-effort only.

Upcoming features such as audit and analytics change what events must do:

• One fact, several purposes. A user being created is an audit record and an observability signal. Publishing it twice would duplicate code and risk the two drifting apart.
• Each purpose needs different data. Audit needs actor, target, action and outcome. Observability needs component and status. These fields must always be present, and audit data, which can hold personal data, must not leak to observability sinks.
• Each purpose needs a different guarantee. A lost audit record is a compliance gap, so audit needs at-least-once. A lost trace is harmless, so observability needs only best-effort. Deployments may weigh this differently, so the guarantee must be configurable.
• Publishing must stay async. A slow or unavailable subscriber must never slow down or break a login.
• The runtime stays lightweight. No new infrastructure (such as a message broker) can be required. ThunderID can run on several nodes.

High-Level Approach

Model

flowchart LR
    E["Event<br/>USER_CREATED"]
    E --> A["Category: audit<br/>at-least-once"]
    E --> O["Category: observability<br/>best-effort"]
    A --> S1["Audit store"]
    O --> S2["OTel exporter"]
    O --> S3["Console logger"]

Loading

• A category is the purpose of an event, for example audit or observability.
• An event belongs to one or more categories, so one fact is published once.
• A subscriber belongs to exactly one category. Each category carries different data, so a subscriber processes it differently anyway. Each subscriber therefore has exactly one guarantee and one payload shape.
• Each category has one delivery guarantee, configured deployment-wide.

An event in both an at-least-once and a best-effort category needs no special handling: each subscriber follows its own category's path.

Event

flowchart LR
    subgraph Event
        direction TB
        ENV["Envelope<br/>id · name · traceId · deploymentid · timestamp"]
        AP["Audit payload<br/>actor · target · action · outcome<br/>+ details"]
        OP["Observability payload<br/>component · status<br/>+ details"]
    end
    ENV ~~~ AP ~~~ OP

Loading

evt := eventbus.NewEvent(ctx, "USER_CREATED").
    With(audit.NewPayload(admin, userRef, audit.ActionCreate, audit.OutcomeSuccess)).
    With(observability.NewPayload("user-service", observability.StatusSuccess))

bus.Publish(ctx, evt)

• Envelope: the common fields, filled in by the framework.
• Payloads: one for each category. Attaching a payload is what puts the event in that category, so an event can never claim a category without its data.
• Required fields are constructor arguments, so a missing field fails to compile.
• Details: an optional key-value map in each payload, for facts specific to one event, such as the reason on TOKEN_REVOKED. Without it, each payload would grow a field for every event's specifics.

Delivery

flowchart LR
    P["Publisher"] -->|Publish| BUS{{"Event bus"}}

    BUS -->|best-effort| Q1["In-memory queue<br/><i>envelope + observability payload</i>"] --> W1["Worker"] --> S1["OTel exporter"]
    BUS -->|best-effort| Q2["In-memory queue<br/><i>envelope + observability payload</i>"] --> W2["Worker"] --> S2["Console logger"]
    BUS -->|at-least-once| T[("Delivery row<br/><i>envelope + audit payload</i>")] --> W3["Worker"] --> S3["Audit store"]

Loading

The bus splits the event by category. Each queue item or delivery row carries the envelope plus only that subscriber's category payload. An observability subscriber never receives the audit payload, and the audit payload is the only one stored in an audit row.

• Publish only records the event and returns. Subscribers always run in the background, which keeps publishing async.
• Each subscriber has its own worker, which handles one event at a time. It's the simplest model, subscribers don't need to be goroutine-safe, and a slow subscriber only slows itself.
• Best-effort: if the queue is full or delivery fails, the event is dropped and counted.
• At-least-once: one row for each (event, subscriber), kept until delivered, so each subscriber retries on its own.

Architecture Overview

1. Recording at-least-once events: two alternatives

A. Record after commit. Simple, but events can be lost.

sequenceDiagram
    participant Svc as User service
    participant DB as Entity DB
    participant ES as Event store
    Svc->>DB: commit(create user)
    Note over Svc,ES: Loss window
    Svc->>ES: Publish(USER_CREATED)

Loading

• The setup: one central event store, and Publish is called after commit.
• Events are lost if:
  • the process crashes before the publish,
  • the event store is down,
  • recording fails. The change is already committed, so the event is dropped.
• Events with no persistence (authentication succeeded) take the same path.

B. Transactional outbox. No events are lost, but it's complex.

sequenceDiagram
    participant Svc as User service
    box Entity DB, one transaction
        participant T as users table
        participant OB as outbox table
    end
    Svc->>T: insert user
    Svc->>OB: Publish(USER_CREATED)
    Svc->>T: commit (both or neither)

Loading

What makes it complex:

• An outbox table in every database: config, entity and runtime.
• Transactions everywhere: each publish call site must run in a transaction and pass it to Publish.
• A second path: events with no persistence, and changes in non-SQL stores, have no transaction to join. They're recorded before responding, and the request fails if recording fails.
• Rolled-back changes: best-effort events inside a transaction may describe a change that was rolled back.

2. Delivering at-least-once events

flowchart TD
    A["Claim next due row"] --> B["subscriber.Handle(...)"]
    B --> C{"Result?"}
    C -->|nil| D["Delete row"]
    C -->|"Permanent(err)"| F["Mark failed"]
    C -->|error| E{"Attempts left?"}
    E -->|yes| G["Retry after backoff"]
    E -->|no| F

Loading

• Delivered rows are deleted immediately. Nothing reads them afterwards, and no cleanup job is needed.
• Retries are capped by a number of attempts. It's the simplest limit, and it stops a broken sink from filling the table.
• Permanent(err) skips retries for failures that can never succeed, such as a payload the sink rejects.
• Failed rows are kept, with an error log and a metric, until a retention period ends. That way no audit record disappears silently, and an operator can replay it.
• Claims time out, so a node that crashes while holding a row can't leave it stuck. Another node picks it up.
• On shutdown, nothing is lost: rows that weren't delivered stay pending.

Example: one USER_CREATED event goes to two audit subscribers, an audit store and a SIEM forwarder.

Delivery row	Attempt 1	Next
USER_CREATED → audit store	nil	deleted
USER_CREATED → SIEM forwarder	timeout	retried after backoff

Only the SIEM forwarder is retried, and the audit store never gets a duplicate.

3. Contracts

Publisher and bus

type Publisher interface {
    Publish(ctx context.Context, evt *Event) error
}

The publisher must	The publisher can rely on
build the event with constructors, and not change it after Publish	Publish never running subscriber code
call Publish after commit (A) or inside the transaction (B)	best-effort categories never blocking and never returning an error
handle an error: log it (A, because the change is already committed) or return it so the transaction rolls back (B)	an error only when at-least-once recording fails
not know or care who the subscribers are

Subscriber and bus

type Subscriber[P any] interface {
    Handle(ctx context.Context, env Envelope, payload P) error
}

eventbus.Subscribe(audit.Category, "audit-store", auditStore) // auditStore: Subscriber[audit.Payload]

• The handler is typed for its category's payload, so a subscriber can't read the wrong payload.
• The ID ("audit-store") must be stable across restarts and releases. At-least-once rows are stored, and the ID is how a worker finds the subscriber again after a restart or on another node.

The subscriber must	The subscriber can rely on
return nil (delivered), an error (retry), or Permanent(err) (fail without retrying)	receiving only the envelope and its own category's payload
dedupe by event ID (at-least-once)	a timeout on each delivery, with a timeout or panic counted as an error
respect ctx	one event at a time on each node
not depend on ordering, because no subscriber needs it

Security Considerations

• Least data: subscribers receive only their own category's payload, so audit PII never reaches observability sinks.
• Stored payloads: delivered rows are deleted immediately, and failed rows are deleted when the retention period ends.
• Visible failures: every failed audit delivery is logged, counted and kept for replay.
• Availability (B only): logins depend on the event store. Refusing a login is safer than allowing it without an audit record.

Impacted Areas

• internal/system/observability/: its subscribers become observability subscribers.
• providers.Event and providers.ObservabilityProvider.
• Publishers in authentication, flow execution, token issuance and revocation.
• Persistence: an event store (A) or an outbox per database (B), for PostgreSQL and SQLite.
• Server lifecycle: starting and stopping the delivery workers.

Alternatives Considered

Sub-categories (audit.authentication)

• Pros: subscribers can subscribe more precisely.
• Cons: an extra level to define, configure and route on.
• Decision: rejected. Subscribers can filter by event name.

A subscriber spanning several categories

• Pros: one subscriber can handle audit and observability together.
• Cons: it gets payloads of different shapes, and needs a rule for which guarantee applies.
• Decision: rejected. One subscriber has one category.

A cursor per subscriber over an event log, instead of one row per (event, subscriber)

• Pros: fewer writes, and ordered delivery.
• Cons: one failing event blocks the subscriber. Commit-order gaps can skip events: event 101 can commit before 100, the cursor moves past 100, and 100 is never delivered.
• Decision: rejected. No subscriber needs ordering.

An external broker (Kafka, NATS)

• Pros: delivery guarantees that are proven in production.
• Cons: extra infrastructure in every deployment, and the DB write and the broker publish can still diverge, the same loss window as A.
• Decision: rejected. It goes against the Lightweight Runtime with GitOps Support pillar.

Questions for Community Input

1. A or B? Is a small loss window acceptable for audit?
2. Replay: is a manual reset enough for first cut, or is a replay API needed?

[indeewari]

Hi @ThumulaPerera ,

1. Is this eventing architecture only for the internal stories ? Are the external subscribers with different domains welcome ?

2. Hope this is designed in a way any event can be subscribed by multiple subscribers and any deployment can define its own set of subscribers at declarative level ?

Thanks and Regards

[ThumulaPerera]

Is this eventing architecture only for the internal stories ? Are the external subscribers with different domains welcome ?

This is for internal stories. But, we can build support to external subscribers on top of this. We can write a internal consumer which listens to these events and publishes those to a webhook or an event bus.

Hope this is designed in a way any event can be subscribed by multiple subscribers and any deployment can define its own set of subscribers at declarative level ?

In this design, subscription granularity is event category (purposes) not individual events. For example, you can subscribe to all audit events but you cannot subscribe only to user.created events. But this is up for discussion. If there is a use-case, we can change the granularity.
Subscriptions (i.e. which internal subscribers subscribe to which events) will be configurable at deployment level.

[ThumulaPerera]

@senthalan, @rajithacharith and I had a review of the design and following are the outcomes.

Decisions

1. Chose Option A for recording at-least-once

   • Outboxes in every database plus transactions at every publish site are too complex. We accept a small loss window even for at-least-once.

2. Separate DB for audit

   • Its storage must be immutable and has a different backup policy and access pattern. Aligns with [Design Discussion] Audit logging #5601

3. Retries use randomized exponential backoff

   • Capped by a number of attempts, then the row is marked permanently failed. How to handle failed rows (replay, retention) is to be decided later. See 1 in open items.

4. Category model confirmed.

   • An event can be in many categories, a subscriber is in one, and each category's guarantee is a deployment-wide setting.

5. Each category has its own typed payload.

   • We rejected one big event with fields pulled out later: it loses compile-time checks.

6. Core change only, no API or UI. It needs full unit test coverage. The design is final apart from the open items.

Open items

1. Commit order across databases:

   • Change first, then event: a change can be left with no audit record.
   • Event first, then change: a ghost audit record that the immutable store can't correct.

2. persist delivery mode:

   • Publish writes straight to the final store, with no temporary record. Is narrowing the loss window worth the cost on the login path?

3. Event naming convention.