[Design Discussion] Make auth observability events principal-aware

[lashinijay]

Related Feature Issue

#4142

Problem Summary

• ThunderID's authentication observability events are principal-blind: TOKEN_ISSUANCE_STARTED, TOKEN_ISSUED, and the AUTHENTICATION flow events carry only client_id, grant_type, scope, and timing, so an agent's token is shape-identical to any other machine client's, and an agent's on-behalf-of token is indistinguishable from a normal user login without an external registry lookup on the raw client_id. No event records the authenticated subject or the actor-to-subject delegation relationship, which means the defining characteristic of OBO authentication (agent X acting for user Y) is neither observable nor auditable.
• Correlation is also broken: the login flow identifies the application by internal app_id while token issuance uses client_id with no cross-reference, and FLOW_STARTED emits a different trace_id than its own child events, so login → authentication → authorization code → token issuance cannot be stitched into a single trace.

High-Level Approach

The proposal is to enrich the existing event payloads rather than introduce new event types or a new pipeline. Every auth event already carries a free-form Data map that flows through the console, file and OpenTelemetry sinks untouched, so the work is about deciding which identity facts to put in that map, and plumbing the few that are not already available at the publish site.

Architecture Overview

This is an enrichment to already publishing diagnostic logs, plus two new data paths that carry identifiers across request boundaries. The single structural addition is an existing provider being injected into the token builder.

Concretely, the additions are:

Addition	Kind	Where
6 data keys (act_type, sub_type, sub_id, act_sub, is_delegated, correlation_id)	vocabulary constants	system/observability/event
Application.EntityCategory	runtime-only struct field (json:"-" yaml:"-")	pkg/thunderidengine/providers
TokenDTO.ActorSub, TokenDTO.SubjectID, TokenDTO.SubjectCategory, TokenResponseDTO.CorrelationID	internal DTO fields	oauth/oauth2/model
AuthorizationCode.CorrelationID, AuthorizationCode.SubjectID, AuthorizationCode.SubjectCategory	fields on an existing JSON blob	oauth/oauth2/authz
correlation_id, sub_id and sub_type assertion claims	claims on the existing internal flow assertion	oauth/oauth2/constants
AccessTokenBuildContext.SubjectEntityID	build-context field	oauth/oauth2/tokenservice
ActorProvider dependency	constructor injection into the token builder	oauth/oauth2/tokenservice

Events report who the subject is (sub_id) and what kind of principal they are (sub_type). Both are needed: the identity answers "what happened for this principal", and the category answers "was this a user login or an agent-for-agent exchange", which is the primary requirement.

The value published is the subject's entity resource ID, which is not always the same as the sub claim on the issued token. Application.SubjectAttribute is an existing per-user-type mapping that sets the token sub to any schema attribute that is unique, required, and string-typed, so on a deployment mapping email or username the token subject is a directly identifying attribute. Publishing that onto events would put personal data into the log and trace sinks under a configuration this change does not control.

The resource ID avoids that and is better on three further counts:

• It is stable across applications. The same user can carry a different sub in two applications, so the token subject is not a usable join key; the resource ID is.
• It matches what flow events already emit, so the flow half and the token half of one authentication join on the same value.
• It is what ActorProvider.GetActor is keyed on, so resolving sub_type works on every deployment rather than silently failing wherever a subject mapping is configured.

Naming follows the token vocabulary rather than inventing a parallel one for events: sub_id and sub_type for the subject, act_sub and act_type for the actor. One set of names across claims and events, and it matches the axis symmetry #5042 anticipates.

One consequence to document for consumers: on a deployment that configures Application.SubjectAttribute, an event's sub_id and the corresponding token's sub claim hold different values - the resource ID and the mapped attribute respectively. They coincide on every other deployment. Anything correlating events against token contents must join on the resource ID rather than on the claim.

The pre-existing user_id on flow events is retired in favor of sub_id, as a clean break with no deprecation period. The subject may be an agent - one authenticating through a flow, or one standing as the subject of a token exchange — so a user_-prefixed key encodes exactly the assumption this change exists to remove; it is populated from an entity reference that is already an agent's ID whenever an agent authenticates, so the name is only correct by accident today.

The user_id key is write-only: it is published in flowexec/engine.go and read nowhere else in the backend, the Console, or the tests, so nothing inside ThunderID depends on it and the only possible consumers are external. Observability is disabled by default, so the population of deployments with any consumer at all is limited to those that opted in. And the key is unreliable today — it is gated on ctx.AuthenticatedUser.IsAuthenticated, and #4142 reports the authenticated subject missing from live AUTHENTICATION flow events, which is why this proposal reads the subject from the entity reference instead. Publishing both keys during a deprecation window would mean steering consumers toward the less reliable of the two while it remains the documented one.

This gives a symmetric vocabulary across the two axes:

Axis	Identity	Kind
Acting principal	act_sub	act_type
Subject	sub_id	sub_type

Alignment with the sub_type claim (#5042)

#5042 adds a sub_type claim to client_credentials tokens with values application and agent, derived from the same OAuthClient.EntityCategory this proposal reads. Its naming was researched against the IANA JWT registry and the relevant IETF drafts and argued in #3578.

This proposal adopts that vocabulary rather than inventing a parallel one. act_type and sub_type publish user, agent, and application.

• For token exchange and jwt-bearer, where the subject arrives as a presented token rather than an authentication, a ThunderID-minted subject token now carries sub_type directly. That is a usable source for sub_type on exactly the agent-for-agent case this proposal must not misreport as a user.

Integration with the existing observability pipeline

The pipeline is untouched. Events are still constructed with event.NewEvent(...), published through ObservabilityProvider.PublishEvent, and fanned out to the console, file and OpenTelemetry subscribers, none of which need to know about the new fields - they serialize the Data map generically. The OTel subscriber already turns every data key into a span attribute, so the new fields become queryable Jaeger tags with no subscriber change.

                    ┌─ console subscriber
publish site ──► publisher ──┼─ file subscriber          (all three: unchanged)
                    └─ otel subscriber ──► span attributes

Two publish sites are enriched, each reading from a source that already exists in its layer:

Publish site	New fields	Source of truth
oauth/oauth2/token (TOKEN_ISSUANCE_STARTED, TOKEN_ISSUED, TOKEN_ISSUANCE_FAILED)	act_type, app_id, sub_id, sub_type, act_sub, is_delegated, correlation_id	OAuthClient.EntityCategory (already populated by inboundclient when the client is loaded by client_id); the access-token DTO, which now carries the subject's resource ID and category alongside the token subject it already held
flow/flowexec (FLOW_*, FLOW_NODE_EXECUTION_*)	act_type, client_id, sub_id, sub_type, correlation_id	Application.EntityCategory (set by actorprovider from the entity it already loads); AuthUser.EntityReference (set by the authn providers)

servicemanager.go is unchanged - there is no new Initialize to register. Neither publish site gains a dependency; the only injection is ActorProvider into the token builder, described next.

Resolving the subject once, where the token is built

The subject's identity and category are recorded on the token DTO in BuildAccessToken, which every grant passes through, next to where the actor is already recorded from ActorClaims. Flow events need no equivalent step: AuthUser.EntityReference already carries both the entity ID and its category, set by the authn providers.

Case	Resource ID	Category
Subject is the client (client_credentials)	OAuthClient.ID	OAuthClient.EntityCategory — short-circuit, no lookup
authorization_code	the sub_id assertion claim, via the authorization code	the sub_type assertion claim — no lookup
refresh_token	carried on the token family, so it survives rotation	carried on the token family, else ActorProvider.GetActor(sub)
Token exchange / jwt-bearer / ID-JAG	the presented token's subject, when it is a resource ID	the presented token's sub_type (#5042) when present, else ActorProvider.GetActor(sub)
Not resolvable	omitted	omitted, rather than guessed

The builder resolves the category only when the grant handler has not already supplied one. Where an upstream layer knows the answer for free the lookup is skipped, and where it does not the builder resolves it - so the login path costs nothing and no grant can end up with the field unset by accident.

That last property is the reason resolution lives here at all. Every grant must pass through BuildAccessToken to mint an access token, so a grant added later inherits the behavior without its author knowing this document exists. The alternative considered was to make each component that establishes a subject responsible for carrying the category forward, with no resolution in the builder at all. That spreads one fact across four independent plumbing points and makes omission silent, since an absent field is indistinguishable from an unresolvable one. Treating the upstream value as a fast path over a guaranteed fallback keeps both properties rather than trading one for the other.

Entity reads are served by the cache-backed entity store, so the fallback lookup is normally an in-memory hit.

The two data paths

An authentication spans several HTTP requests, so no request-scoped value can join them. The correlation id and the subject's resource ID therefore travel on carriers that already cross those boundaries, following the route the token family id (tfid) already takes:

flowchart LR
  A["flow execution<br/>(execution id, entity reference)"] -->|correlation_id + sub_id + sub_type claims| B["flow assertion<br/>(signed JWT)"]
  B -->|parsed by authz| C["AuthorizationCode<br/>(JSON in runtime_transient)"]
  C -->|read by grant handler| D["AccessTokenBuildContext<br/>→ TokenDTO"]
  D --> E["TOKEN_ISSUED<br/>correlation_id, sub_id, sub_type"]
  A --> F["FLOW_* events<br/>correlation_id, sub_id, sub_type"]

Loading

sub_id is a new claim because the assertion does not currently carry the resource ID. Its sub claim holds the mapped token subject, and the field named userID in the parsed claims is populated from that same sub - so despite the name, no resource ID survives the assertion today.

sub_type rides the same carrier. AuthUser.EntityReference already holds the subject's category at the point the assertion is minted, so stamping it costs nothing and spares the login path an entity read it would otherwise pay on every issuance. The builder still resolves the category when no claim arrives, so this is a fast path rather than the only path.

Both carriers are schemaless with respect to this change: the assertion is a JWT (a new claim needs no migration) and the authorization code is a JSON document in the runtime store keyed by NamespaceAuthzCode (a new field needs no migration). No schema change and no data migration are required.

Grants with no originating flow (client_credentials, token exchange) fall back to the request trace id for correlation, so that field is uniformly populated without a special case at the consumer.

Extension-surface consideration

Application.EntityCategory will be added to pkg/thunderidengine/providers, which is the engine's SPI rather than an internal package. It is marked json:"-" yaml:"-" so it never appears in the application API or in declarative resources, and it is re-derived on every flow-context load rather than persisted. A third-party ActorProvider implementation that does not set it simply leaves it empty, in which case act_type is omitted from flow events and everything else behaves as before — the degradation is graceful rather than an error.

Operational impact

• Configuration: none. The fields appear on existing events wherever observability is already enabled; there is no new toggle.
• Data volume: each auth event grows by up to six small key/value pairs. Three are low-cardinality enums or booleans; sub_id and act_sub are opaque identifiers, the latter only on delegated issuance. For the OTel sink these are span attributes rather than new spans, so trace counts are unchanged.
• Cardinality: sub_id is a per-principal value, so it is the one attribute here with meaningful cardinality on the OTel sink. It is bounded by the number of distinct principals authenticating, not by request volume.
• Latency: none on the paths that matter. client_credentials reads the category off the client it already loaded, and authorization_code reads it off the flow assertion, so neither adds a lookup. A cache-backed entity read remains as the fallback for grants whose subject arrives without a category - refresh where the family did not carry one, and the exchange grants - and it happens during token construction rather than event publication, so it is on the response path when it does occur.
• Trace grouping: FLOW_STARTED and FLOW_FAILED move from the request trace into their siblings' flow trace.
• Consumer-visible removal: user_id is retired from flow events in favor of sub_id. Together with the trace-grouping change, these are the only two changes visible to an existing consumer.
• Failure behavior: unchanged. Publishing stays best-effort and out of band; a sink that is unavailable does not affect token issuance.

PII considerations

Observability sinks have a different profile from the token itself: they are retained longer, shipped off-host, and readable by operators who are not token holders. Anything published onto an event should be assessed against that, not against what the client already receives.

The codebase already takes a position on this. Application logs record user identifiers through log.MaskedString, which keeps only the first and last character of the value, and always log the resource ID rather than a human-readable attribute. The event pipeline has no equivalent masking — it serializes the Data map verbatim — so what is put on an event must be safe unmasked.

Every identifier this change publishes is opaque or categorical:

Field	Value	Personal data
act_type	user | agent | application	No — low-cardinality enum
sub_type	user | agent | application	No — low-cardinality enum
is_delegated	boolean	No
sub_id	subject's entity resource ID	No — opaque UUID, not configurable
act_sub	acting principal's resource ID, from oauthApp.ID or the refresh token's act.sub	No — opaque UUID, not configurable
app_id / client_id	application or OAuth client identifier	No
correlation_id	flow execution id, or the request trace id as fallback	No — system-generated

The field that would have carried personal data is the token's sub claim, and it is not published. Where a deployment maps email or username through Application.SubjectAttribute, that value is the token subject — so publishing sub would have put a directly identifying attribute into the sinks, conditional on a configuration this change does not control. Publishing the resource ID keeps the mapping unreachable from the event path entirely, rather than safe-by-default and exposed under a configuration a deployment may reasonably choose.

The stream therefore stays free of directly identifying data regardless of how subject attributes are configured, while still supporting per-subject correlation through a stable opaque key.

Security considerations

The change publishes more context onto events that already exist and adds two identifiers that cross request boundaries.

Data published to the sinks. The PII assessment above is the primary control: everything published is an opaque resource identifier or a low-cardinality enum, and no directly identifying value becomes publishable under any configuration. The change publishes no token values, no client secrets, no assertions, and no authentication factors. Error descriptions were already published by both sites and are unchanged.

The correlation identifier is a handle for a live flow. The value carried is the flow execution id, which is also the continuation handle a client presents to resume an in-progress authentication — the flow service resolves it and rejects an unknown one with ErrorInvalidExecutionID. Publishing it holds for three reasons:

• It is not newly exposed on flow events. Those already published execution_id before this change; correlation_id duplicates a value the same event carried.
• On token events it is exposed for the first time, but only after the flow has terminated. The service removes the flow context on every terminal path, so by the time an authorization code is redeemed and TOKEN_ISSUED fires, the id no longer resolves to a resumable context. What reaches the token sinks is a spent handle.
• It is never returned to the client. TokenResponseDTO.CorrelationID is an internal field, not serialized on the token response, and the authorization code it arrives on is a server-side document in the runtime store.

The residual risk is an operator with log access replaying a still-live execution id observed on a flow event — unchanged from before this change, since execution_id was already there.

The new assertion claims are not trust inputs. correlation_id and sub_id are added to the internal flow assertion, which is signed, so both inherit the assertion's integrity and a tampered value fails verification like any other. Neither is used to authorize, to gate a decision, or to select a code path: correlation_id is only stamped onto events, and sub_id is used to report the subject and to key a category lookup whose failure mode is an omitted field. A wrong value produces a mis-stitched trace or an absent category, not a wrong authorization outcome. In particular, sub_id does not replace the token's sub claim and does not participate in issuing or validating tokens.

Signal integrity for the fields that could mislead. sub_type is resolved from the subject's own entity record rather than inferred from the request, specifically so an agent acting as a token subject is not reported as a user. Where it cannot be resolved it is omitted rather than defaulted, so a consumer never sees a confident wrong answer. The same applies to act_type. This matters because these fields will be read as evidence about who did what — a field that guesses is worse than a field that is absent.

No new external surface. Application.EntityCategory is added to the engine SPI but marked json:"-" yaml:"-", so it appears on no API response and in no declarative resource, and it is re-derived rather than persisted. No schema change, no migration, no new endpoint, no new configuration toggle.

Impacted Areas

• Diagnostic logs

Alternatives Considered

Leave the join to the consumer. Keep publishing client_id and let whoever reads the logs resolve it against the application or agent API to learn what kind of principal it was. Rejected because it pushes a join onto every consumer, requires each of them to hold API credentials, and cannot be done retroactively — once an agent is deleted, its historical events become uninterpretable. Enriching at publish time makes the event self-describing for as long as it is retained.

is_agent boolean instead of act_type. Rejected because a boolean collapses user, agent and application into two states and cannot express the subject side at all, so a second boolean would be needed almost immediately. The enum costs the same to publish and extends without a schema decision each time a principal kind is added.

Publish the token's sub claim as the subject. The literal reading of "Add subject" in #4142. Rejected because sub is deployment-configurable through Application.SubjectAttribute and may be an email address or username, which would put personal data into the sinks; because it varies per application for the same principal, making it useless as a join key; and because it is not what GetActor is keyed on, so it cannot resolve the subject's category. Publishing the entity resource ID satisfies the requirement without any of the three problems.

Publish nothing for the subject's identity, only its category. An earlier draft of this proposal, on the grounds that identity serves audit rather than diagnostics. Rejected once it was clear that the resource ID carries no personal data: the scope argument was really an argument against publishing sub specifically, and it does not extend to an opaque identifier. Omitting identity altogether would have left "what happened for this principal" unanswerable and left the category lookup broken on subject-mapped deployments, for no privacy gain.

Publish a hashed or pseudonymized subject. A middle option that would preserve per-subject correlation without a plaintext identifier. Rejected because a hash of a low-entropy value such as an email address is recoverable by dictionary, so it is pseudonymization rather than anonymization and remains personal data — and because the resource ID already provides correlation with no reversibility question at all.

Name the field user_id, matching the existing flow-event key. Rejected because the subject may be an agent, so the name would assert a principal class the change exists to stop assuming. The existing user_id is retired for the same reason rather than carried forward as an alias, since shipping both would guarantee dashboards built on the wrong one.

Mint a dedicated correlation identifier. Rather than reusing the flow execution id, generate a fresh id at the start of an authentication and carry it. Rejected because it adds a second identifier that means almost the same thing as one the system already has, and every consumer would then need to know which of the two to pivot on. The execution id already exists, is already published on flow events, and already survives the request boundaries that matter.

Reuse the token family id (tfid) as the correlation key. It travels the exact route this change needs, so it was the obvious candidate. Rejected because it does not exist early enough: flow events are published before any token is minted, so the flow half of the trace could not carry it. It is also scoped to a token family for revocation purposes, which is a different lifetime from one authentication.

Resolve the subject's category only in the builder, with no upstream fast path. The simplest version of the choke point: BuildAccessToken looks the category up for every grant except client_credentials. Rejected because it charges the login path a lookup for a fact the flow already had in hand, and the flow assertion is already being extended for sub_id and correlation_id, so carrying the category costs one more claim on a carrier that is being touched anyway.

Carry the category forward from each component, with no resolution in the builder. The mirror image: every component that establishes a subject sets the category, and the builder assumes it is present. Rejected because it spreads one fact across four independent plumbing points and makes omission silent — a grant added later would emit no category, and an absent field is indistinguishable from an unresolvable one, so nothing would surface the mistake. The adopted design takes the upstream value when it exists and resolves it otherwise, which keeps the no-lookup property of this option and the cannot-be-forgotten property of the previous one.

Keep FLOW_STARTED and FLOW_FAILED on the request trace. Adding correlation_id alone would stitch the flow together without moving any event between traces, avoiding one of the two consumer-visible behavior changes here. Rejected because those two events would remain separated from their own children in any trace-native view, which is where an operator actually looks first; a correlation field that only works in log search solves half the problem the issue describes.