Skip to content

fix(logger): never let structured serialization throw into the caller - #6331

Merged
waleedlatif1 merged 4 commits into
stagingfrom
fix/logger-safe-json-serialization
Aug 6, 2026
Merged

fix(logger): never let structured serialization throw into the caller#6331
waleedlatif1 merged 4 commits into
stagingfrom
fix/logger-safe-json-serialization

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Context

Production structured logging went fully silent for hours across all five sim-production-us-east-1-app ECS tasks — INFO/WARN/ERROR all dropped to zero per 30 min, and a raw substring search for \"module\": returned nothing. Confirmed as genuine loss via AWS/Logs IncomingLogEvents (~25k → ~2-6k per 30 min), not a query artifact. Non-JSON output (Next.js internals, Better Auth, raw stack traces) kept shipping, and the socket service on the same cluster and same awslogs driver was unaffected — so it was localized to the app containers' own output, not the driver or the account.

This PR fixes one concrete, reproducible defect on that path. It is not claimed to be the whole root cause.

The bug

In the production JSON branch, Logger.log merged caller-supplied arguments into the entry with Object.assign and then called JSON.stringify, with no error handling on either step. If that metadata contained a cyclic reference, a BigInt, or a throwing getter, the call raised a TypeError out of logger.info(...) into the caller — the line was lost and whatever was being logged about was aborted.

Verified against the current staging logger under NODE_ENV=production:

RESULT: circular -> THREW: TypeError | JSON.stringify cannot serialize cyclic structures.
RESULT: bigint   -> THREW: TypeError | JSON.stringify cannot serialize BigInt.

Cyclic metadata is easy to hit unintentionally — errors carrying request/response/cause chains, ORM rows with back-references, socket handles.

The development path was never affected: the colorized branch routes objects through formatObject, which already has a try/catch. So this class of bug is invisible locally and only appears in production, where its signature is exactly structured logs vanish while raw stack traces keep shipping.

The fix

Entry construction and serialization now go through serializeEntry, which degrades instead of throwing:

  1. normal JSON.stringify
  2. retry with a replacer that tolerates cycles ([Circular]) and BigInt (stringified)
  3. minimal entry preserving timestamp/level/module/message, flagged serializationError: true

A logger must never be able to break its caller.

Tests

Three regression tests in packages/logger/src/index.test.ts covering cyclic, BigInt, and throwing-getter metadata. All three were confirmed to fail against the unfixed source (git stash on index.ts3 failed | 25 passed) and pass with it (28 passed).

bun run lint and tsc --noEmit are clean.

Follow-ups (not in this repo)

Alarms and ECS task definitions live in the private sibling CDK repo, not here. Two gaps found there and reported separately:

  • No alarm exists on AWS/Logs IncomingLogEvents for the app log group. Every metric filter over that group sees zero matches during a blackout and sits in OK — a silent alarm indistinguishable from healthy. A log-absence alarm with treatMissingData: BREACHING is specified in the report.
  • Both app and socket LogDrivers.awsLogs({ ... }) calls omit mode, so both default to blocking.

@vercel

vercel Bot commented Aug 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 6, 2026 8:32pm

Request Review

@cursor

cursor Bot commented Aug 6, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches the core production logging path used for observability; behavior changes only on previously failing payloads, but every log line now goes through new serialization logic.

Overview
Production JSON logging no longer throws back into callers when metadata or extra args are cyclic, contain BigInt, or use hostile getters/proxies/toJSON. The structured branch now routes through serializeEntry, which tries plain JSON.stringify, retries with a path-aware replacer ([Circular], stringified bigints), then emits a minimal line flagged serializationError: true.

withMetadata copies input through materializeMetadata so spreads and key reads degrade ([Unreadable], metadataError) instead of raising during child logger setup.

Regression tests cover cycles, BigInt, throwing getters, repeated non-cyclic refs, hostile toJSON, and hostile proxies.

Reviewed by Cursor Bugbot for commit 1a4b662. Configure here.

@greptile-apps

greptile-apps Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR makes production structured logging degrade safely when caller-supplied metadata cannot be serialized.

  • Adds tolerant handling for cyclic references and BigInt values.
  • Adds guarded metadata materialization and a minimal last-resort log entry.
  • Adds regression coverage for getters, proxies, toJSON, repeated references, cycles, and BigInt.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
packages/logger/src/index.ts Adds layered, non-throwing metadata materialization and structured-entry serialization fallbacks; the previously reported escape paths are addressed.
packages/logger/src/index.test.ts Adds focused regression tests covering hostile metadata and serialization edge cases.

Reviews (5): Last reviewed commit: "fix(logger): keep repeated references ou..." | Re-trigger Greptile

Comment thread packages/logger/src/index.ts
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread packages/logger/src/index.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 2463ff6. Configure here.

@waleedlatif1
waleedlatif1 force-pushed the fix/logger-safe-json-serialization branch from 2463ff6 to 22e6b9b Compare August 6, 2026 19:28
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 22e6b9b. Configure here.

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

In production the JSON branch merged caller-supplied arguments into the log
entry and stringified it with no error handling. A cyclic reference, a BigInt,
or a throwing getter in that metadata raised a TypeError out of `logger.info`
and friends: the line was lost and the caller's code path aborted.

Dev was unaffected — the colorized branch already routes objects through
`formatObject`, which catches — so this class of bug is invisible locally and
only surfaces in production, where it reads as structured logs disappearing
while raw stack traces keep shipping.

Build and serialize through `serializeEntry`, which falls back to a
cycle/BigInt-tolerant replacer and then to a minimal entry flagged with
`serializationError`.
@waleedlatif1
waleedlatif1 force-pushed the fix/logger-safe-json-serialization branch from 1a4b662 to 8d5863e Compare August 6, 2026 20:26
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 1a4b662. Configure here.

@waleedlatif1
waleedlatif1 merged commit 0ae09c1 into staging Aug 6, 2026
4 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/logger-safe-json-serialization branch August 6, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant