feat(workflows): richer observability for push notifications

[dmarchuk]

Problem

Push logging and metrics are much thinner than email, so when a customer's push notifications don't land they can't easily tell what happened. Three concrete gaps:

• Opaque failure logs. On any failure the workflow log dumps the raw FCM/APNs JSON error at the user (FCM send error. Status: 400. Body: {...}). Fine for us, useless for a customer.
• Mislabeled metrics. Push send metrics are emitted as metric_kind: 'other' even though a dedicated 'push' kind already exists (email correctly uses 'email').
• No failure visibility in ops. The only Prometheus signal was a per-platform "sent" counter. Nothing told us why sends were failing, how many were skipped, or how slow they were.

Push has real limits vs email: FCM/APNs respond synchronously and there is no delivery-receipt webhook like SES, so there's no honest delivered/opened/clicked. This focuses on what we can actually observe at send time.

Changes

A failure-reason normalizer (push-notification-errors.ts) maps FCM error codes / APNs reasons / HTTP status into a small bounded set — unregistered, invalid_token, auth_error, rate_limited, invalid_payload, provider_error, network_error, unknown — each with:

• a plain-language, action-guiding message (e.g. an auth_error tells the user which credential to check), and
• a severity: config problems a user must fix are errors, transient or expected ones (throttle, dead token, 5xx) are warnings.

Logging now uses it. Instead of a raw payload dump, the workflow log shows a readable one-liner at the right level. The raw provider response is kept at debug level for support. Success reads "accepted by FCM/APNs" rather than "sent" — honest, since we only know the provider accepted it, not that the device displayed it.

Metrics. Business metrics now use the correct metric_kind: 'push'. Ops metrics gain reason/skip/latency dimensions:

metric	labels	meaning
push_notification_failed_total	platform, reason	terminal failures, by normalized reason
push_notification_skipped_total	platform	recipient had no registered device token
push_notification_send_duration_ms	platform	provider request latency (histogram)

Existing push_notification_sent_total and push_notification_token_pruned_total are unchanged.

The channel send throws a typed PushSendError carrying the reason/level/platform, so the top-level loop logs once at the right severity and labels the failure metric without re-deriving anything or double-logging.

Note

Surfacing a push summary in the workflow Metrics tab (email has EmailMetricsSummary; push has no UI at all) is the intended immediate follow-up — this PR is the backend foundation it needs.

How did you test this code?

• New push-notification-errors.test.ts: parameterized over representative FCM and APNs error shapes, asserting the reason classification, the unregistered flag that drives token pruning, the raw code passthrough, and the severity mapping. Guards against a mapping regression silently breaking pruning or mislabeling metrics.
• Updated push-notification.service.test.ts to assert the metric_kind: 'push' fix.

Independent verification against this branch (master merged in for a current tree):

• Ran the normalizer + push service suites: 50/50 passing. tsc --noEmit and prettier --check clean on the changed files.
• Live end-to-end on a running local stack: registered a device token, triggered a workflow, and confirmed in the Invocations tab that a successful send now logs Push notification accepted by FCM. (the wording changes from "sent via" to "accepted by", since we only know the provider accepted the message, not that the device displayed it).

The richer provider-error messages (auth rejected, rate-limited, invalid payload) can't be forced against a live provider on demand, so those stay covered by the normalizer unit tests. CI covers the kafka/clickhouse-backed integration suites.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

No user-facing docs change — this improves the observability surface (workflow logs + internal metrics) with no API or config change.

🤖 Agent context

Autonomy: Human-driven (agent-assisted). Directed by the requester (session owner); left unassigned only because I couldn't verify their GitHub handle this session — a maintainer should set the DRI.

Authored with Claude Code. Two parallel research agents first mapped email's observability end-to-end (metrics/logging/SES webhook) as the target and the CDP metrics/logging infrastructure + current push state, which surfaced the constraints that shaped scope: app-metrics carry no label dimensions (fixed 6-field schema), so platform/reason granularity lives in Prometheus and logs rather than business metrics; and push has no delivery-receipt channel, so open/click parity with email isn't possible. Invoked the /writing-tests skill before writing the normalizer tests to keep them aimed at real regressions. Chose to keep this PR backend-only (fully testable here) and tee up the Metrics-tab UI as a separate follow-up rather than ship unverified frontend wiring.

🤖 Generated with Claude Code

---

Generated by Claude Code

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs

[greptile-apps]

_{Reviews (1): Last reviewed commit: "feat(messaging): richer observability fo..." | Re-trigger Greptile}

[mayteio]

Nothing major, solid PR 😎

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-15 12:49 UTC	Run
prod-us	✅ Deployed	2026-07-15 13:16 UTC	Run
prod-eu	✅ Deployed	2026-07-15 13:15 UTC	Run