fix(hybridcloud): Allow skip-on-failure for Jira, Jira Server, VSTS and MS Teams webhooks

[vaind]

Adds jira, jira_server, vsts and msteams to hybridcloud.webhookpayload.skip_on_failure_providers, which now lists every provider the integration middleware forwards through WebhookPayload. These four were left as follow-ups in #121057.

Why now

Before #122893, any claim of 60 or more records went through the parallel drain regardless of provider, so a strict provider's deep mailbox already got concurrent, reordered delivery with skip-on-failure inside each wave. Jira routinely cleared that depth, so everything below has been reachable in production for months, precisely when its mailbox was deep. #122893 fixed that: a strict claim now delivers on one thread, in order, stopping at the first retryable failure. That is correct, but it also removed the accidental throughput, and the per-integration Jira and VSTS mailboxes are backing up at serial speed. Since the deploy, Jira holds the deepest single mailbox of any provider, its pending count roughly doubled over two hours, and its oldest pending record is aging at wall-clock speed. VSTS peaked at the second-deepest mailbox in the same window. outcome:retry is rare for both: this is the one-thread ceiling, not head-blocking.

What the option exposes

Two webhooks for the same issue can be delivered concurrently or in reverse order, and a failed one is retried after the ones behind it. For every provider here the state that can go wrong is the same as for the providers already listed:

• Status sync never had mailbox order. Every handler calls sync_status_inbound, which enqueues a task via apply_async, and the task drops any event not strictly newer than the ExternalIssue provider-time watermark from fix(integrations): Ignore reordered and replayed inbound status webhooks #121084. Jira and Jira Server stamp issue.fields.updated; VSTS stamps System.ChangedDate. The sync_status_inbound.stale_event counter already fires for Jira in production.
• Assignee sync is last-writer-wins with no watermark for every provider, GitHub and GitLab included, until fix(integrations): Guard inbound assignee sync against reordered webhooks #121157 lands. Two reordered assignment changes for one issue can leave the older assignee in place. Jira and Jira Server apply the full issue.fields.assignee snapshot. VSTS was held back in fix(hybridcloud): Allow skip-on-failure for Bitbucket, GHE and GitLab webhooks #121057 as syncing from a delta, but Azure work items have a single assignee, so System.AssignedTo.newValue is the assignee after the change and an absent newValue is unassigned, the same snapshot Jira sends.

Provider-specific caveats

• jira / jira_server: reordered key moves. handle_issue_moved rekeys by exact old_key, so it is idempotent on redelivery, but two moves of one issue delivered in reverse leave the ExternalIssue at the intermediate key with no log or halt, and later lookups by key miss until the next move. A status or assignee webhook already carrying the new key can also run before a concurrent rekey commits and drop as no-affected-groups. No GitHub analogue, and Jira project moves are rare. feat(integrations): Report renames that moved no external issue #123480 makes a zero-row rekey observable, splitting a stranded link from the far more common move of an issue nobody linked. Note also that a unique-key conflict during a rekey merges the stale row's links into the survivor and deletes it, which is irreversible rather than a no-op. Jira Server's issue-id bucketing only engages above a volume gate and puts same-issue webhooks in the same bucket, so it is not a mitigation here.
• vsts: _get_done_statuses fails open to unresolve. On any ApiError, including a rate limit, it returns an empty set, which makes get_resolve_sync_action return UNRESOLVE for every state change, and the watermark then advances past the lost resolve. Jira propagates the same failure and no-ops; GitHub and GitLab make no API call. Sixteen threads enqueue sixteen times as many uncached states lookups against Azure DevOps per second, so a 429 burst on a deep mailbox would reopen issues in bulk. fix(vsts): Stop unresolving when the work item states lookup fails #123478 fixes this by raising instead, which leaves both the status and the watermark alone. It should deploy before vsts is added here.
• msteams: same-issue card actions can interleave. The cell forwards card actions, channel messages that mention the bot, and bot membership events; installationUpdate and personal messages stay synchronous. A card action is applied as the clicking user via the issue update API with no watermark, so resolve followed by unresolve on one issue in one backed-up claim can land in either order, and the refreshed card can disagree with the result. Self-correcting on the next click, and the double click that causes it is itself a symptom of the wedge. Channel mentions and teamMemberAdded only send a card; teamMemberRemoved deletes the integration, and retry backoff could already reorder it against a re-install. The wedge itself is a bug, fixed in fix(msteams): Answer a rejected card action with the API's status #123477: _issue_state_change hit an unbound response on any 4xx from the issue update and returned 500, so the record head-blocked the tenant's mailbox through ten attempts at hour-long backoff.

What to watch after the flip

Jira Cloud is the only provider here with an unbucketed per-integration mailbox and a cell-side rate limit of 100 requests per second on the webhook endpoint. A 429 is retryable, so watch outcome:retry for provider:jira. For VSTS, watch unresolve volume and vsts.get-done-states.failed.

Not covered

stripe and vercel also appear as strict WebhookPayload providers, but their forwarding lives in getsentry and was not assessed here.

No production effect on its own

sentry-options-automator pins this option; https://github.com/getsentry/sentry-options-automator/pull/9540 is the change that takes effect. Tests of the strict path now pin jira out of the allowlist instead of relying on the default.