Skip to content

fix(services): notify's run summary stops claiming a delivery that dead-lettered (#7747) - #7875

Merged
huangyiirene merged 1 commit into
mainfrom
claude/issue-7747-notify-acted-dead-letter
Aug 12, 2026
Merged

fix(services): notify's run summary stops claiming a delivery that dead-lettered (#7747)#7875
huangyiirene merged 1 commit into
mainfrom
claude/issue-7747-notify-acted-dead-letter

Conversation

@huangyiirene

Copy link
Copy Markdown
Collaborator

Fixes #7747

The finding, confirmed

Boot without the push channel registered, fire a flow whose notify node targets ['push'], and the two records an operator can read contradict each other:

record says
sys_notification_delivery status: 'dead', error: "channel 'push' not registered"
flow-run summary status: 'success', acted: 1

Nothing was delivered, and the surface built to answer "did this sweep actually do anything" (#4354) said it had.

Root cause — located, and sharper than the dispatch hypothesis

The dispatch guessed notify-node.ts:293 (metrics: { acted: Number(result.delivered) || 0 }) was the seam. That line is where the number is consumed; the defect is where it is minted.

MessagingService.emit() has two paths, and they disagreed about what EmitResult.delivered means:

  • Inline (P0)fanOut() has each channel's answer before emit() returns. An unregistered channel is already ok: false, so delivered was a real terminal count. This path never had the bug.
  • Outbox (P1, ADR-0030)enqueueDeliveries() returns once the (recipient × channel) rows are durable; the dispatcher sends and decides the outcome afterwards. delivered counted those enqueued rows anyway. Its own docstring said so — "here ok means 'accepted for delivery' (enqueued), not yet delivered" — and the pin at messaging-service.test.ts:311 wrote the conflation down verbatim: expect(result.delivered).toBe(2); // 2 enqueued (accepted).

So acted was minted before any send attempt, from a count labelled delivered that meant enqueued, and nothing ever revisits it — not stale by a moment, never revised at all. The dispatcher then dead-letters the push row (dispatcher.ts:210/:251), and the summary keeps saying acted: 1.

The fix

1. EmitResult separates the two counts (messaging-service.ts)

  • delivered now means a channel accepted the delivery — terminal and observed. Only the inline path can report it non-zero.
  • New enqueued carries the outbox path's accepted rows: durable, unsent, outcome pending on sys_notification_delivery.

2. The notify node counts only what was delivered toward acted (notify-node.ts)

When deliveries are merely enqueued it reports unmeasuredEffect instead — the qualifier a connector_action already uses for an effect the platform cannot count.

Why that and not the alternatives the dispatch offered:

  • Not a bare acted: 0 (option a). The codebase rules this out explicitly — "never acted: 0, which would claim it did nothing" (connector.zod.ts:613). It would trip the broken-sweep alert on every healthy outbox-backed notify, which is exactly what the #4354 comment at notify-node.ts:290 was defending against.
  • Not "keep acted: 1, surface the outcome alongside" (option b). acted_count is a column and a highlightField on sys_automation_run, and the documented alert reads it directly. Leaving the overstatement in the column and annotating it elsewhere leaves the lie where the operator actually looks.
  • unmeasured is the platform's existing word for this exact shape. The alert is selected > 0 AND acted = 0 AND unmeasured = 0, so a pending delivery suppresses the alert without asserting success — and formatRunSummaryLine already prints unmeasured= with the comment "acted=0 on a line that also says unmeasured=3 means 'cannot tell', not 'did nothing'". No fourth counter invented.

Node output gains enqueued next to delivered and notificationId, so the id has a stated reason to be followed into the delivery record.

Status stays success. The flow did everything it can do synchronously, and failing it would let a channel registered a moment later retroactively break the flow (messaging-service-plugin.ts:212 documents that late registration is supported). No synchronous wait on async delivery was added — notify must not block a flow on a downstream channel, so "delivered" is not a claim it is ever positioned to make. It simply stops making it.

Acceptance criterion

an operator reading the run summary must not conclude "delivered" when sys_notification_delivery says dead.

notify-delivery-outcome.integration.test.ts wires the real MessagingService + MemoryNotificationOutbox + NotificationDispatcher behind the notify node and asserts on the two durable records — the folded run summary and the outbox row — never on internal call counts. A fake that answers emit() in one shot cannot express the disagreement at all, because the defect lives in the seam between enqueue and dispatch.

Four cases, covering both directions so the fix can't degenerate into "unregistered channels are special":

case summary durable record
outbox + push unregistered acted: 0, unmeasured: 1 row dead, "channel 'push' not registered"
outbox + inbox registered acted: 0, unmeasured: 1 row pendingsuccess after tick
inline, delivered acted: 1, unmeasured: 0 channel received it
inline, push unregistered acted: 0, unmeasured: 0 (a measured zero — correctly alert-eligible)

Reverse-verified on origin/main: the durable-record assertions pass (the dead-letter is real) while the summary assertion fails with acted: 1, unmeasured: 0 — the finding exactly. 2 of 4 red before, 4/4 green after.

Pins

  • messaging-service.test.ts:311updated deliberately: delivered: 2 // 2 enqueued (accepted)enqueued: 2, delivered: 0. That comment was the conflation; the pin now states the distinction.
  • messaging-service.test.ts:102 (inline) — extended with enqueued: 0, keeping delivered terminal on both paths.
  • connector-nodes.test.ts:292unaffected and green. It pins connector accounting (acted: 1, unmeasured: 0), not notify's.
  • No other consumer exists: a repo-wide grep for EmitResult / .delivered finds only these tests and the notify node. The other two messaging.emit() callers (plugin-approvals, plugin-audit) await without reading the result. No docs reference notify.delivered or EmitResult.

Gates

gate result
build closure (--filter "@objectstack/service-automation...")
service-automation suite ✅ 77 files / 932 tests
service-messaging suite ✅ 18 files / 206 tests
pnpm check:docs-audit-scope
changeset ✅ patch × 2 (user-visible observability change)

packages/spec/src/** is not touched, so gen:schema / gen:docs do not apply. Engine-double gate is N/A — the new test adds no data-engine fake with update/delete verbs (it drives the real messaging stack; its only fake is a send-only channel).

Note for the backlog (not fixed here)

http-nodes.ts:139 reports { output: { deliveryId, enqueued: true }, metrics: { acted: 1 } } for an outbox-enqueued HTTP delivery — the same class of overstatement against sys_http_delivery, which has its own dead status. Left alone as out of scope for #7747; worth its own card.


Generated by Claude Code

…ad-lettered (#7747)

A stack booted without the `push` channel registered, running a flow whose
notify node targets `['push']`, produced two operator-facing records that
contradicted each other: `sys_notification_delivery` held `status: 'dead'`,
`error: "channel 'push' not registered"`, while the flow-run summary reported
`status: 'success', acted: 1`.

The seam is `EmitResult.delivered`. With the durable outbox in play (ADR-0030
P1), `emit()` returns as soon as the `(recipient x channel)` rows are enqueued
and the dispatcher decides the outcome afterwards — but `delivered` counted
those enqueued rows under a name that says they arrived, and `notify` fed the
number straight into `acted`. A count minted before any send attempt then
survived the dead-letter unrevised; nothing ever revisits it.

- `EmitResult` separates the two counts. `delivered` now means a channel
  ACCEPTED the delivery — terminal and observed, which only the inline (P0)
  fan-out can report. New `enqueued` carries the outbox path's accepted rows:
  durable, unsent, outcome pending on `sys_notification_delivery`.
- The notify node counts only delivered toward `acted`, and reports
  `unmeasuredEffect` when deliveries are merely enqueued — the qualifier a
  `connector_action` already uses for an effect the platform cannot count, and
  deliberately not a bare `acted: 0`, which would claim the run did nothing.
  The broken-sweep alert is `selected > 0 AND acted = 0 AND unmeasured = 0`, so
  a pending delivery suppresses the alert without asserting success. Node output
  gains `enqueued` next to `delivered` and `notificationId`.

The run still reports `success`: the flow did everything it can do
synchronously, and failing it would let a channel registered a moment later
retroactively break the flow. Notify must not block on a downstream channel, so
"delivered" is not a claim it is ever positioned to make — it simply stops
making it.

Tests wire the REAL MessagingService + NotificationDispatcher behind the notify
node and assert on the two durable records (folded run summary, outbox row),
not on call counts — the finding is that those records disagree. Reverse-
verified: on origin/main the durable assertions pass and the summary asserts
`acted: 1, unmeasured: 0`.

Pin updated deliberately: `messaging-service.test.ts` asserted
`delivered: 2 // 2 enqueued (accepted)` — the conflation written down — now
`enqueued: 2, delivered: 0`. `connector-nodes.test.ts:292` is unaffected (it
pins connector, not notify, accounting) and stays green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGwDLmaML1LtLmQ4F4Aq7z
@vercel

vercel Bot commented Aug 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Aug 12, 2026 1:41am

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/service-automation, @objectstack/service-messaging.

4 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/automation/flows.mdx (via @objectstack/service-automation)
  • content/docs/automation/webhooks.mdx (via @objectstack/service-messaging)
  • content/docs/kernel/services-checklist.mdx (via @objectstack/service-automation, @objectstack/service-messaging)
  • content/docs/plugins/packages.mdx (via @objectstack/service-automation, @objectstack/service-messaging)

2 release-owned page(s) also reference the affected code. These are read-only:

  • content/docs/releases/implementation-status.mdx (via @objectstack/service-automation, @objectstack/service-messaging)
  • content/docs/releases/v9.mdx (via @objectstack/service-automation)

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file tests tooling labels Aug 12, 2026
@huangyiirene
huangyiirene marked this pull request as ready for review August 12, 2026 01:57
@huangyiirene
huangyiirene added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit f28ef3b Aug 12, 2026
27 checks passed
@huangyiirene
huangyiirene deleted the claude/issue-7747-notify-acted-dead-letter branch August 12, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation size/m tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

finding: a notify flow-run summary reports success/acted:1 while the delivery dead-lettered — the dead-letter is absent from the run summary

2 participants