fix(hybridcloud): Allow skip-on-failure for Bitbucket and GHE webhooks - #121057
Draft
vaind wants to merge 1 commit into
Draft
fix(hybridcloud): Allow skip-on-failure for Bitbucket and GHE webhooks#121057vaind wants to merge 1 commit into
vaind wants to merge 1 commit into
Conversation
A mailbox drain aborts on the first delivery failure unless the payload's provider is in `hybridcloud.webhookpayload.skip_on_failure_providers`, so a single wedged message blocks everything behind it for up to MAX_DELIVERY_AGE. Bitbucket Server mailboxes have been observed stuck for days behind one failing head payload. Add the three providers whose cell-side handlers tolerate reordering: - github_enterprise: GithubEnterpriseRequestParser subclasses GithubRequestParser and only overrides `provider`, so GHE runs literal subclasses of GitHub's webhook handlers yet never matched the exact-string allowlist. An unintended exclusion, not a decision. - bitbucket / bitbucket_server: each endpoint registers a single push handler (repo:push and repo:refs_changed) and inserts commits idempotently under `except IntegrityError: pass`. Bitbucket Server re-fetches each self-contained fromHash..toHash range from the API. Neither has a create/update/delete lifecycle that reordering could corrupt. Deliberately excluded: gitlab, vsts, jira_server and msteams are order-dependent. jira was not assessed and remains a follow-up. The registered default only governs environments without an explicit override; the production value lives in sentry-options-automator.
github-actions
Bot
added
the
Scope: Backend
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draining a webhook mailbox aborts on the first delivery failure unless the payload's provider is in
hybridcloud.webhookpayload.skip_on_failure_providers(deliver_webhooks.py:287,:563). Onlygithubwas listed, so a single wedged message blocks everything behind it for up toMAX_DELIVERY_AGE(3 days) — Bitbucket Server mailboxes have been seen holding tens of undelivered payloads for days behind one failing head. This addsgithub_enterprise,bitbucketandbitbucket_server.Why these three are safe
github_enterprise—GithubEnterpriseRequestParseroverrides onlyprovider(parsers/github_enterprise.py:20-21) and runs literal subclasses of GitHub's handlers (github_enterprise/webhook.py:144-183). The allowlist is an exact string match, so GHE never matched"github"— an unintended exclusion, not a risk decision.bitbucket/bitbucket_server— one push-only handler each (bitbucket/webhook.py:170,bitbucket_server/webhook.py:160), commits inserted idempotently underexcept IntegrityError: pass(:148-160,:134-148), per-organization mailboxes. Bitbucket Server re-fetches each self-containedfromHash..toHashrange from the API (:101-116). No create/update/delete lifecycle to reorder.Known caveat
Both Bitbucket handlers call
update_repo_data(bitbucket/webhook.py:83-108,bitbucket_server/webhook.py:44-51), a last-writer-wins refresh of the stored repo name and URL from the event body. A push carrying a pre-rename name, retried after a later push, can briefly write the stale name back. It self-heals on the next successful push and renames are rare — assessed as not disqualifying, but it is the one place reordering is observable for these providers.Why GitLab is not in this list
IssuesEventWebhook._handle_status_change(gitlab/webhooks.py:325-349) passes a delta,{"action": action}, intosync_status_inboundrather than a state snapshot, andget_resolve_sync_action(gitlab/issue_sync.py:262-277) maps it straight through —CLOSE → RESOLVE,REOPEN → UNRESOLVE— without re-reading the issue. A reordered CLOSE/REOPEN pair leaves the Sentry group resolved while the GitLab issue is open.vsts,jira_serverandmsteamsare order-dependent for similar reasons.jirais also not cleared, for a different reason: its verdict was UNCLEAR becausesync_status_inbounddispatches viaapply_async(mixins/issues.py:485), so mailbox ordering is already void before the handler runs.No production effect on its own
The registered default only governs environments without an explicit override, and
sentry-options-automatorpins this fleet-wide. getsentry/sentry-options-automator#9018 is the change that actually takes effect in prod; merging this one alone changes nothing on sentry.io.Draft until the in-flight PR adding monotonic guards to the GitHub and GitLab pull-request lifecycle writes lands — merge this after it.