Audit logging: design and open questions

[RAVEENSR]

Agent Manager had no audit trail. This proposal describes the design implemented in #1493 and the decisions behind it, so we have a single place to discuss changes to it.

Problem

There was no way to answer basic questions about who changed what:

• Who rotated that API key?
• Who granted themselves deploy-production?
• Who deleted that agent?

A search for audit across the repository returned two slog.Info lines labelled "Audit log for configuration update/deletion" and nothing else. Those carried no actor, no outcome, and no durability.

Mapping the code surfaced four related conditions:

1. RBAC_ENABLED defaults to false in code (config/config_loader.go). When off, requirePermission returns before any check. The Helm chart sets it true, so chart installs enforce authorization, but nothing recorded which mode a deployment was in.
2. All five authorization deny sites in middleware/authorization.go wrote a 403 and nothing else. Two did not even log. Denied privilege escalation was invisible.
3. JWT validation failures were logged without subject, source IP or path, so credential stuffing against the API was undetectable.
4. The internal gateway server checks an api-key header inside each handler. A rejected key produced no event, which is the only signal a gateway-credential brute force would ever give.

Design

flowchart TB
    REST["REST API<br/>231 routes"]
    MCP["MCP tools<br/>13 tools"]
    INT["Internal server<br/>gateway api-key"]
    SYS["Reconcilers<br/>and schedulers"]

    COV["Coverage tier<br/>one wrapper, every route and tool<br/>records the envelope"]
    SEM["Semantic tier<br/>~50 explicit emits<br/>records what changed"]

    DEDUP{"already recorded<br/>semantically?"}
    SKIP["suppressed<br/>one record per operation"]

    REDACT["Redaction<br/>allow-list keyed by action"]
    REC["Recorder<br/>buffered and batched"]
    SINK["Sink interface"]

    STDOUT["stdout<br/>log_type=audit"]
    FUTURE["database or SIEM<br/>addable later"]
    STORE[("cluster log pipeline<br/>operator's retained store")]

    REST --> COV
    MCP --> COV
    REST --> SEM
    MCP --> SEM
    INT --> SEM
    SYS --> SEM

    COV --> DEDUP
    DEDUP -- "yes, and it succeeded" --> SKIP
    DEDUP -- "no, or it failed" --> REDACT
    SEM --> REDACT

    REDACT --> REC
    REC --> SINK
    SINK --> STDOUT
    SINK -.-> FUTURE
    STDOUT --> STORE

    classDef entry fill:#dce9f5,stroke:#4a7ba7,color:#12212e
    classDef tier fill:#e4efdd,stroke:#6d9455,color:#18240f
    classDef pipe fill:#efe3ef,stroke:#8d5f8d,color:#241424
    classDef out fill:#e2ecf0,stroke:#5b8494,color:#0f1e25
    classDef muted fill:#f2f2f2,stroke:#a0a0a0,color:#3d3d3d,stroke-dasharray: 4 4

    class REST,MCP,INT,SYS entry
    class COV,SEM tier
    class REDACT,REC pipe
    class SINK,STDOUT,STORE out
    class FUTURE,SKIP muted

Loading

Two tiers

The coverage tier is a single wrapper installed in middleware.RouteRegistrar. At registration time the registrar already holds the two things a record needs — the route pattern and the rbac.Permission that gates it — so wrapping there covers all 231 registered routes without touching any route file, controller or service. addTool does the same for MCP.

Of those 231 routes, 130 are state-changing and all are audited except one documented exemption (generate-name, which suggests a name and persists nothing). A further 12 read routes are audited because they disclose credential material or security configuration; the rest of the reads are not, since recording every list call multiplies volume for little forensic gain.

It records the envelope: who called what, from where, and what came back. It cannot tell you what changed.

The semantic tier is ~50 explicit emit sites at the operations where the envelope is not enough. POST .../permissions/add → 200 does not say which permission was granted to which role, which is the entire question. These record the change itself.

When a semantic record exists for a successful request, the coverage tier stands down, so each operation produces one record — the richer one. On failure the coverage record is kept, because a request rejected before reaching the service emits nothing semantic and that rejection is exactly what must not go unrecorded.

Coverage cannot regress

api/audit_coverage_test.go walks the registrar's route ledger and fails the build if a mutating route is unaudited or unlabelled, if a policy entry has gone stale, or if the read policy broadens accidentally. audit.NewRouteMeta panics at startup on a route it cannot label.

addTool requires an audit.Action alongside its permissions and panics without one — the same fail-closed discipline it already applied to permissions. A new mutating tool cannot ship unattributed.

Both tiers use the same action constants, so a question like "who deployed agent X" is one query regardless of which surface the call arrived on. A test pins all 32 route-to-action pairs.

What is deliberately never recorded

Three properties are structural rather than filtered:

• Bodies are never read, request or response. Git credentials, client secrets, upstream auth values and the API keys returned once on creation are therefore out of reach of a record, not filtered out of one. This is what makes the coverage tier safe to enable on every route.
• requestPath holds the route pattern, never the raw URL, which removes path and query-string leakage by construction.
• URLs are reduced to scheme, host and path. A URL is the one declared value that can carry a credential in its own syntax — RFC 3986 userinfo (https://user:pass@idp.example/jwks), or a token in the query. A detail declared KindURL has its userinfo, query and fragment removed before the record is written, with a marker saying components were dropped. This is enforced at redaction rather than at the emit site, so the next caller to record a URL cannot forget it, and a test fails if a URL-valued detail is declared as anything else. Dropping the whole query rather than filtering known parameter names follows the same reasoning as the allow-list below.

Anything a caller attaches by hand passes an allow-list keyed by action. Undeclared keys are dropped and reported under _droppedKeys. This is deliberately not a deny-list: the existing sanitizeAttributesForLogging in controllers/identity_controller.go filters exactly password, so newPassword, apiToken and secret all pass through it. A deny-list fails on the field nobody thought of.

For the free-form attribute map on user creation, records carry the attribute key names and a count, never values, plus a flag when a key looks credential-shaped.

Failure policy

Ordinary records are buffered and dropped under pressure rather than blocking a request — a slow sink must not become an outage. Drops are counted and reported.

Security-critical operations refuse to proceed when the record cannot be written. Because their effect usually lands in an external system, they record intent first and outcome after:

sequenceDiagram
    autonumber
    participant C as Caller
    participant S as Service
    participant A as Audit trail
    participant X as External system<br/>(Thunder, OpenBao, gateway)

    Note over S,A: Tier-1 operation, e.g. rotate an API key

    S->>A: write intent record (outcome: unknown)
    alt trail unavailable
        A--xS: error
        S--xC: 503 — operation refused
        Note right of S: no credential is issued,<br/>so nothing untraceable exists
    else intent recorded
        A-->>S: ok
        S->>X: perform the change
        X-->>S: result
        S->>A: write outcome record (success or failure)<br/>linked by attemptEventId
        S-->>C: response
    end

    Note over A: a record left at "unknown" means the process<br/>died mid-operation — deliberate forensic signal

Loading

Frequent operations that issue no credential — builds, console test keys, gateway configuration — record after the fact, so a CI pipeline is never blocked by the trail.

Record shape

Modelled on the Kubernetes-audit / CloudTrail shape (actor · action · resource · outcome · details), which is what auditors already read.

{
  "log_type": "audit",
  "event": {
    "eventId": "87b3970f-c1e4-4df4-befd-9dc02834dcfc",
    "occurredAt": "2026-08-05T08:13:37Z",
    "action": "git-secret:create",
    "actionClass": "credential",
    "severity": 4,
    "actorType": "user",
    "actorId": "alice@example.com",
    "actorTokenId": "jti-7c1e",
    "surface": "api",
    "sourceIp": "203.0.113.77",
    "correlationId": "c0ffee-...",
    "requestPath": "/orgs/{orgName}/git-secrets",
    "ouId": "ou-9f2",
    "resourceType": "git-secret",
    "resourceName": "github-deploy-key",
    "outcome": "success",
    "statusCode": 201,
    "requiredPermission": "amp:git-secret:create",
    "rbacEnforced": true
  }
}

OCSF was considered and rejected as the storage schema: this domain's operations ("rotate an LLM proxy API key", "promote an agent to production") all collapse into Application Activity (6001) / activity_id: Other, so the class taxonomy buys nothing forensically while making every schema change a mapping argument. A one-way OCSF projection at the sink is the right place for that interop if we want it.

Storage

Records go to stdout as JSON tagged log_type=audit, where the cluster log pipeline already collects container output. That needs no new infrastructure, and it gives the trail a copy the service cannot rewrite — there is no write path from this process back into the log store.

The Sink interface exists so a database or direct SIEM feed can be added later by writing one sink and changing one line of wiring, without touching any of the ~50 emit sites.

Enforcement posture is in the trail

Because RBAC_ENABLED defaults to false, the trail documents its own enforcement state rather than leaving it in a config file nobody reads:

• system:rbac-disabled at startup when authorization is off
• rbacEnforced on every record, alongside the permission that would have applied
• system:startup bounds any gap in the trail to a restart, so "nothing happened" is distinguishable from "the service was not running"

Open questions

These are the parts worth arguing about.

1. Retention. This is the one that decides whether any of this is usable. Records go to stdout; the observability plane's default log retention is far shorter than a typical audit evidence period (SOC 2 runs 3–12 months; ISO 27001 A.5.33 requires retention per a defined policy). Until log_type=audit is routed to storage with a real policy, the trail expires before an audit needs it. Where should it go, and who owns configuring it?

2. Authentication events. Login, logout, failed login, MFA, password change and session revocation happen in WSO2 Thunder. There is no login endpoint in this repository; Agent Manager only ever sees an already-minted JWT. These are precisely the events SOC 2 CC6.1/CC6.6 and ISO 27001 A.8.15 expect, and a trail without them is not complete.

The token jti is recorded on every event as the join key, and the correlation recipe (sub + jti + time window) is documented. Two options for closing this:

• Thunder ships its own JSON into the same log pipeline; correlation happens at query time in the SIEM.
• Agent Manager polls a Thunder audit API and stores the events itself.

The first is simpler and does not require Thunder to expose a stable audit API. Preference?

3. Durable store and a read API. Currently out of scope: no query endpoint, no console page. Operators read their SIEM. Is that acceptable for GA, or do we need an in-product view? Adding a Postgres sink is cheap; the read API and console page are the real cost.

4. Who may read the audit log. If we add a read API: anyone who can grant permissions can grant themselves audit-read, and that grant is itself audited. No arrangement of in-product RBAC fixes this, because the admins administer the RBAC. Genuine separation of duties requires shipping to a system Agent Manager admins do not control. Do we want a dedicated Auditor role anyway, as a least-privilege option?

5. Tamper evidence. Hash-chaining records and publishing chain heads periodically was designed but not implemented. With records going to a store this service cannot write back into, the marginal value is low — write-once log storage gives the same protection with no code. Worth revisiting only if the threat model includes tampering with the log store itself.

6. RBAC_ENABLED=false. The trail now records this state, but should the service refuse to start in that mode when a strict flag is set? That is a breaking operational change, so it was left out.

7. The publisher route. api/monitor_publisher_routes.go registers a mutating route with no rbac.Permission, protected only by an audience check, while rbac.MonitorScorePublish exists and is already granted to RoleAdmin. Adding the permission could break the evaluation job, so it needs its own change and its own testing.

Current coverage

Complete within this service: every mutating REST route, every MCP tool, the internal gateway server, the publisher route, and the background reconcilers. Authorization denials and rejected tokens are recorded on every surface.

The only gap is authentication itself, which happens in Thunder.

Implementation: #1493. Reference documentation: agent-manager-service/docs/audit-logging.md.