fix(orb): make self-host webhook-delivery loss visible to operators#1628
Merged
Conversation
The self-host Sentry forwarder only forwards level:error/fatal, so delivery loss was silent or warn-only — the failure a self-host operator most needs to see (container looks alive, reviews nothing): - retryFailedRelays: a relay event abandoned after its TTL/retry budget logged at level:warn -> escalate to level:error (orb_relay_events_dropped). - pruneRelayPending: a pull-mode (tailnet) container offline past the 24h window lost queued webhooks with NO log -> add orb_relay_pending_dropped at level:error (distinct event from the push-path drop). - server.ts boot: registerOrbRelayTarget 'failed' logged at info + a swallowed .catch -> 'failed' now logs level:error and the catch routes to captureError. Consolidates the self-host delivery-integrity cluster from the 2026-06-27 audit.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1628 +/- ##
=======================================
Coverage 95.52% 95.52%
=======================================
Files 204 204
Lines 22057 22059 +2
Branches 7967 7968 +1
=======================================
+ Hits 21070 21072 +2
Misses 412 412
Partials 575 575
🚀 New features to boost your workflow:
|
8 tasks
JSONbored
added a commit
that referenced
this pull request
Jun 28, 2026
…the event slug (#1636) forwardStructuredLogToSentry titled each issue with only obj.event and buried the real failure in a 'log' context blob — so operators had to open each issue to learn what broke. Now: - title leads with the failure: "<event>: <message ?? error>" (e.g. orb_broker_unavailable: The operation was aborted due to timeout) - index filterable tags for the dimensions operators search/group by (repo, installationId, pull, pr, project, kind, deliveryId), present values only - fingerprint by event so recurrences collapse into one issue instead of fragmenting on the variable detail now in the title Pairs with the source-side instrumentation (#1605/#1619/#1625/#1627/#1628): those make failures reach Sentry; this makes each issue legible at a glance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes self-host webhook-delivery loss visible to operators — the failure class a self-host operator most needs to trust ("the container looks alive but reviews nothing"). The self-host Sentry forwarder only forwards
level:"error"/"fatal"lines, so these losses were silent or warn-only:retryFailedRelays,relay.ts): a relay event abandoned after its 1h TTL / 5-retry budget was logged atlevel:"warn"→ invisible to Sentry. Escalated tolevel:"error"(orb_relay_events_dropped).pruneRelayPending,relay.ts): a pull-mode (NAT/tailnet) container offline past the 24h retention window lost its queued webhooks with no log at all — the push path was hardened but the functionally-identical pull path wasn't. Added a distinctorb_relay_pending_droppedatlevel:"error"(distinct event name so operators can tell pull-mode loss from push-mode loss).server.ts):registerOrbRelayTargetreturning"failed"was logged at info (nolevel) and the.catchwas swallowed (() => {}). A failed registration means the central Orb never forwards this install's webhooks here — the container reviews nothing. Now"failed"logs atlevel:"error"(selfhost_orb_relay_register_failed) and the catch routes tocaptureError(already imported).Surfaced by the 2026-06-27 review-pipeline hardening audit (observability) — consolidates the "self-host delivery integrity" cluster the audit's completeness-critic called out. (The companion
retryFailedRelaysnot_forwardable retry-semantics fix is deferred to its own PR — it changesforwardOrbEvent's return type and ripples into the webhook handler.)Scope
relay.ts+server.ts+ the relay test).site//CNAME/Pages; followsCONTRIBUTING.md.Validation
git diff --check·actionlint·typechecktest:coverage— updated the exhausted-budget drop test to assertconsole.error+"level":"error"; extended the TTL-prune test to assertorb_relay_pending_droppedat error level. Bothchanges>0/changes===0arms remain covered (the many no-stale enqueue/retry paths).server.tsis codecov-ignored (boot wiring).test:workers·build:mcp·test:mcp-pack·ui:*·npm audit --audit-level=moderateIf any required check was skipped, explain why:
Safety
Notes
relay.tsruns on the cloud Orb (gittensory-api, auto-deploys on merge); theserver.tsboot wiring runs on the self-host engine (ships on the next rebuild).