Skip to content

Standardize gateway JSONL records with event/_schema and millisecond timestamps#6485

Merged
lpcox merged 5 commits into
mainfrom
copilot/standardize-jsonl-log-format
May 25, 2026
Merged

Standardize gateway JSONL records with event/_schema and millisecond timestamps#6485
lpcox merged 5 commits into
mainfrom
copilot/standardize-jsonl-log-format

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 25, 2026

MCP Gateway logs had mixed record semantics (type used for both direction and record kind), no schema discriminator, and inconsistent timestamp precision across JSONL vs text logs. This change aligns gateway logs to the AWF-style base contract so cross-log parsing and aggregation can use one shape.

  • JSONL contract normalization (RPC + DIFC)

    • JSONLRPCMessage now emits:
      • timestamp (ISO 8601 UTC with milliseconds)
      • event (rpc_request / rpc_response)
      • _schema (rpc-message/v2)
    • JSONLFilteredItem now emits:
      • timestamp (same format)
      • event: "difc_filtered"
      • _schema: "difc-filtered/v2"
    • Removed overloaded type from JSONL records in favor of event.
  • Timestamp consistency across logs

    • Unified text log formatter (mcp-gateway.log and {serverID}.log) now uses the same millisecond-precision UTC layout as JSONL.
  • Parser/spec updates for downstream consumers

    • Updated in-repo workflow/spec docs that parse rpc-messages.jsonl to match the new discriminator (event == "difc_filtered"), including Repo Assist and DIFC audit/smoke workflows.
    • Updated top-level logging docs to explicitly document required JSONL fields and schema versions.
  • Test updates for schema and format expectations

    • Adjusted logger/MCP tests to assert event, _schema, and millisecond timestamp parsing instead of legacy type values.
type JSONLRPCMessage struct {
    Timestamp string `json:"timestamp"` // 2006-01-02T15:04:05.000Z07:00
    Event     string `json:"event"`     // rpc_request | rpc_response
    Schema    string `json:"_schema"`   // rpc-message/v2
    Direction string `json:"direction"` // IN | OUT
    // ...
}

Copilot AI changed the title [WIP] Standardize JSONL log record format Standardize gateway JSONL records with event/_schema and millisecond timestamps May 25, 2026
Copilot finished work on behalf of lpcox May 25, 2026 17:23
Copilot AI requested a review from lpcox May 25, 2026 17:23
@lpcox lpcox marked this pull request as ready for review May 25, 2026 17:33
Copilot AI review requested due to automatic review settings May 25, 2026 17:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR standardizes the MCP Gateway’s machine-readable JSONL logging contract (RPC + DIFC) by introducing explicit event and _schema discriminators and aligning timestamps (JSONL + text logs) to a consistent UTC ISO-8601-with-milliseconds format, enabling downstream parsers/aggregators to rely on a single record shape.

Changes:

  • Replaced overloaded JSONL type fields with event and added _schema versioning for RPC and DIFC-filter records.
  • Unified timestamp formatting across JSONL and text logs to millisecond-precision UTC.
  • Updated in-repo workflows/docs/tests to parse and assert the new JSONL fields (event, _schema) and timestamp layout.
Show a summary per file
File Description
README.md Documents the updated logging contract and timestamp format.
internal/server/difc_log.go Updates DIFC log comment to reflect difc_filtered event naming.
internal/mcp/connection_logging_test.go Updates MCP connection logging test assertions to event/_schema.
internal/logger/rpc_logger.go Adds RPCMessageType.JSONLEvent() mapping to standardized event strings.
internal/logger/rpc_logger_test.go Updates RPC logger tests to assert event/_schema.
internal/logger/jsonl_logger.go Implements JSONL record shape changes and millisecond timestamp formatting.
internal/logger/jsonl_logger_test.go Updates JSONL logger tests for new fields and timestamp parsing.
internal/logger/common.go Updates text log timestamp formatting to millisecond layout.
internal/logger/common_test.go Updates log line format tests for millisecond timestamps and parsing.
.github/workflows/smoke-proxy-github-script.md Updates DIFC event counting to match event:"difc_filtered".
.github/workflows/smoke-proxy-github-script.lock.yml Mirrors the smoke workflow update in the compiled/locked workflow.
.github/workflows/smoke-proxy-github-script.invalid.yml Mirrors the smoke workflow update in the generated invalid fixture.
.github/workflows/repo-assist.md Updates JSONL parsing logic to use event == difc_filtered.
.github/workflows/integrity-filtering-audit.md Updates audit guidance/greps to match the new JSONL discriminator.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 14/14 changed files
  • Comments generated: 2

Comment thread README.md Outdated
Comment thread internal/logger/jsonl_logger.go Outdated
@lpcox
Copy link
Copy Markdown
Collaborator

lpcox commented May 25, 2026

@copilot address review feedback

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot stopped work on behalf of lpcox due to an error May 25, 2026 17:38
@lpcox lpcox merged commit 0c9b621 into main May 25, 2026
14 checks passed
@lpcox lpcox deleted the copilot/standardize-jsonl-log-format branch May 25, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standardize JSONL log record format: consistent timestamp and event fields

3 participants