[Proposal] Custom traffic log format support

[Dimagidhp]

Summary

The gateway's traffic log line has a fixed shape today. This proposal adds two opt-in features:

• [traffic_logging.json_fields] lets you define the whole traffic log line, the same way [router.access_logs.json_fields] defines the access log. Each key is a field name. Each value is either a literal or a $ctx: expression.
• Latency time unit: every latency is available in microseconds (as today) and in decimal milliseconds, so you choose the unit for each field.

Existing configurations behave exactly as before. Implementation: #3846.

Feedback is welcome on the configuration and on the review questions at the end.

---

Problem

• The line can't be reshaped. Operators can add fields under properties and drop fields with exclude_fields. They can't rename fields or flatten the nested objects (api, operation, latencies, ...). The access log already allows this through json_fields.
• Latencies are in microseconds only (durationUs, backendLatencyUs, ...). Most log backends and dashboards use milliseconds, so every consumer has to convert.
• Lines are long, mostly because of the latencies object, and there is no way to keep only the fields a consumer needs.

Proposed configuration

1. [traffic_logging.json_fields]

[traffic_logging]
enabled = true

[traffic_logging.json_fields]
"@timestamp"  = "$ctx:request.timestamp"
correlationId = "$ctx:request.id"
method        = "$ctx:request.method"
status        = "$ctx:response.status"
api           = "$ctx:api.name + '/' + api.version"
durationMs    = "$ctx:latencies.durationMs"
backendMs     = "$ctx:latencies.backendLatencyMs"
env           = "prod"

This config writes one line like this per request:

{"@timestamp":"2026-10-21T09:12:33.482Z","api":"PetStore/v1","backendMs":12.874,"correlationId":"5f6b6e2a-2f38-4b7a-9c2f-6b6a2f384b7a","durationMs":15.321,"env":"prod","method":"GET","status":200}

Behaviour:

• When json_fields is set, it replaces the default line. Only these fields are written.
• A value is either a literal string or a $ctx: CEL expression. Expressions use the same variables as properties, plus the new ones listed below. Results keep their type: numbers stay numbers, and maps become objects.
• If properties is also set, it is ignored and a warning is logged at startup.
• masked_headers still applies. For example, $ctx:request.header['authorization'] logs ****.
• An expression that doesn't compile is dropped at startup with an error log. The other fields are still written.
• Data a request doesn't have, such as latencies on a request rejected before reaching the backend, is written as its zero value ("" or 0).
• Keys are flat and written in alphabetical order. "http.method" is a single key, not a nested object.
• In TOML, quote keys that contain characters other than letters, digits, _ and -, such as "@timestamp" and "http.method". An unquoted dot creates a nested table.

2. Latency time unit

Each latency is available in both units:

Microseconds (integer)	Milliseconds (decimal)
latencies.durationUs	latencies.durationMs
latencies.requestMediationLatencyUs	latencies.requestMediationLatencyMs
latencies.responseMediationLatencyUs	latencies.responseMediationLatencyMs
latencies.backendLatencyUs	latencies.backendLatencyMs

Milliseconds keep microsecond precision: 15321 µs becomes 15.321. You pick the unit for each field by choosing the variable. The default line's latencies object stays in microseconds.

Other units need no new variables, because a $ctx: expression can do the arithmetic: latencies.durationUs * 1000 for nanoseconds (still at microsecond precision), or latencies.durationMs / 1000.0 for seconds. CEL doesn't mix decimal and whole numbers, so latencies.durationMs / 1000 is rejected at startup; write 1000.0.

3. New $ctx: variables

These variables work in both json_fields and properties, except where noted:

Variable	Value
request.timestamp	Request time in UTC, RFC 3339 with milliseconds (same as the default line's timestamp)
client.ip, client.userAgent	Same as the default line's client object
latencies.*Us, latencies.*Ms	See the latency table above
errorType, error.<field>	Same as the default line's errorType and error fields: errorCode, errorMessage, wso2ErrorCode, type, direction, summary, policy, policyPhase, source, originalStatus, jsonRpcCode
request.body, response.body	json_fields only. They need [collector] body capture and are cut to max_payload_size. The request_body/response_body toggles aren't needed.

Existing variables (request.*, response.*, api.*, application.*, auth.*, metadata, ...) are unchanged.

4. exclude_fields with json_fields

For the default line, exclude_fields works as it does today. With json_fields:

Entry	Effect on json_fields
requestHeaders.<name>, responseHeaders.<name>	The header is removed from request.header / response.header
New: $ctx:<variable>, e.g. $ctx:error.policy	Every entry that uses the variable is dropped at startup, with a warning
New: $ctx:auth.property.<key>, $ctx:metadata.<key>	The key is removed from the map before expressions are evaluated
Any other name (latencies, api.name, ...)	No effect, because these name fields of the default line

exclude_fields = ["requestHeaders.cookie", "$ctx:auth.property.email"]

An unknown $ctx: variable logs a warning at startup, so a typo doesn't silently leave data in the log. $ctx: entries don't apply to properties; properties are still excluded by name (properties.<key>).

5. Helm

gateway:
  config:
    traffic_logging:
      enabled: true
      json_fields:
        "@timestamp": "$ctx:request.timestamp"
        durationMs: "$ctx:latencies.durationMs"
        env: prod

The chart renders this as [traffic_logging.json_fields] and quotes every key.

Backward compatibility

• The default line, properties and exclude_fields don't change unless json_fields is set.
• The new variables and $ctx: exclude entries are additions. Existing configurations stay valid as they are.

Review questions

1. Both set: when json_fields and properties are both set, properties is ignored with a startup warning. Should this be a startup error instead?
2. Exclusion scope: $ctx: exclude entries apply only to json_fields. properties keep name-based exclusion (properties.<key>), so there aren't two ways to do the same thing. Is that acceptable?
3. Default-line unit: there is no latency_unit setting for the default line. Its keys name the unit (durationUs), so changing the unit would break existing consumers, and json_fields already lets users choose. Is such a setting still needed?
4. Absent values: these are written as "" or 0. Should those fields be left out instead?