Skip to content

fix(orb): make self-host webhook-delivery loss visible to operators#1628

Merged
JSONbored merged 1 commit into
mainfrom
fix/orb-relay-delivery-observability
Jun 27, 2026
Merged

fix(orb): make self-host webhook-delivery loss visible to operators#1628
JSONbored merged 1 commit into
mainfrom
fix/orb-relay-delivery-observability

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Makes self-host webhook-delivery loss visible to operators — the failure class a self-host operator most needs to trust ("the container looks alive but reviews nothing"). The self-host Sentry forwarder only forwards level:"error"/"fatal" lines, so these losses were silent or warn-only:

  1. Push-path relay drop (retryFailedRelays, relay.ts): a relay event abandoned after its 1h TTL / 5-retry budget was logged at level:"warn" → invisible to Sentry. Escalated to level:"error" (orb_relay_events_dropped).
  2. Pull-path relay drop (pruneRelayPending, relay.ts): a pull-mode (NAT/tailnet) container offline past the 24h retention window lost its queued webhooks with no log at all — the push path was hardened but the functionally-identical pull path wasn't. Added a distinct orb_relay_pending_dropped at level:"error" (distinct event name so operators can tell pull-mode loss from push-mode loss).
  3. Boot relay-registration failure (server.ts): registerOrbRelayTarget returning "failed" was logged at info (no level) and the .catch was swallowed (() => {}). A failed registration means the central Orb never forwards this install's webhooks here — the container reviews nothing. Now "failed" logs at level:"error" (selfhost_orb_relay_register_failed) and the catch routes to captureError (already imported).

Surfaced by the 2026-06-27 review-pipeline hardening audit (observability) — consolidates the "self-host delivery integrity" cluster the audit's completeness-critic called out. (The companion retryFailedRelays not_forwardable retry-semantics fix is deferred to its own PR — it changes forwardOrbEvent's return type and ripples into the webhook handler.)

Scope

Validation

  • git diff --check · actionlint · typecheck
  • test:coverage — updated the exhausted-budget drop test to assert console.error + "level":"error"; extended the TTL-prune test to assert orb_relay_pending_dropped at error level. Both changes>0/changes===0 arms remain covered (the many no-stale enqueue/retry paths). server.ts is codecov-ignored (boot wiring).
  • test:workers · build:mcp · test:mcp-pack · ui:* · npm audit --audit-level=moderate

If any required check was skipped, explain why:

  • No migration/OpenAPI/cf-typegen: logging/escalation only.

Safety

  • No secrets/wallets/trust-scores exposed — logs carry only an event name + a dropped-count.
  • No public GitHub text change · auth/CORS N/A.

Notes

  • relay.ts runs on the cloud Orb (gittensory-api, auto-deploys on merge); the server.ts boot wiring runs on the self-host engine (ships on the next rebuild).

The self-host Sentry forwarder only forwards level:error/fatal, so delivery
loss was silent or warn-only — the failure a self-host operator most needs to
see (container looks alive, reviews nothing):

- retryFailedRelays: a relay event abandoned after its TTL/retry budget logged
  at level:warn -> escalate to level:error (orb_relay_events_dropped).
- pruneRelayPending: a pull-mode (tailnet) container offline past the 24h
  window lost queued webhooks with NO log -> add orb_relay_pending_dropped at
  level:error (distinct event from the push-path drop).
- server.ts boot: registerOrbRelayTarget 'failed' logged at info + a swallowed
  .catch -> 'failed' now logs level:error and the catch routes to captureError.

Consolidates the self-host delivery-integrity cluster from the 2026-06-27 audit.
@dosubot dosubot Bot added the size:S label Jun 27, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jun 27, 2026
@codecov

codecov Bot commented Jun 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.52%. Comparing base (a2ebeb1) to head (d9d9b81).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1628   +/-   ##
=======================================
  Coverage   95.52%   95.52%           
=======================================
  Files         204      204           
  Lines       22057    22059    +2     
  Branches     7967     7968    +1     
=======================================
+ Hits        21070    21072    +2     
  Misses        412      412           
  Partials      575      575           
Files with missing lines Coverage Δ
src/orb/relay.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit b1631fb into main Jun 27, 2026
19 checks passed
@JSONbored
JSONbored deleted the fix/orb-relay-delivery-observability branch June 27, 2026 23:38
JSONbored added a commit that referenced this pull request Jun 28, 2026
…the event slug (#1636)

forwardStructuredLogToSentry titled each issue with only obj.event and buried
the real failure in a 'log' context blob — so operators had to open each issue
to learn what broke. Now:
- title leads with the failure: "<event>: <message ?? error>" (e.g.
  orb_broker_unavailable: The operation was aborted due to timeout)
- index filterable tags for the dimensions operators search/group by (repo,
  installationId, pull, pr, project, kind, deliveryId), present values only
- fingerprint by event so recurrences collapse into one issue instead of
  fragmenting on the variable detail now in the title

Pairs with the source-side instrumentation (#1605/#1619/#1625/#1627/#1628):
those make failures reach Sentry; this makes each issue legible at a glance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant