Skip to content

fix(integrations): raise sync_status_outbound processing deadline to 30s - #120726

Draft
billyvg wants to merge 1 commit into
masterfrom
claude/loving-hypatia-3t9jjr
Draft

fix(integrations): raise sync_status_outbound processing deadline to 30s#120726
billyvg wants to merge 1 commit into
masterfrom
claude/loving-hypatia-3t9jjr

Conversation

@billyvg

@billyvg billyvg commented Jul 28, 2026

Copy link
Copy Markdown
Member

Fixes ProcessingDeadlineExceeded('execution deadline of 10 seconds exceeded by sentry.integrations.tasks.sync_status_outbound').

Root cause

sync_status_outbound never set processing_deadline_duration, so it inherited the 10 second default. That budget is too small for the work the task actually does:

# Step Observed
1 RPC get_integration ~130ms
2 RPC get_organization_integrations ~125ms
3 Proxy GET — fetch work item from Azure DevOps ~680ms
4 Proxy GET — fetch VSTS projects ~605ms
5 RPC get_integration_external_project ~125ms
6 Proxy PATCH — update work item deadline exceeded

Roughly 2.5s elapses before the PATCH is even issued, leaving under 7.5s for a request whose own HTTP timeout is 10s. Under ordinary network variance (DNS, TCP setup, proxy overhead) the PATCH regularly runs past the remaining budget. The stack trace confirms the deadline fires inside _new_conn()safe_create_connection(), while the TCP socket for the PATCH is still being established — i.e. the task is killed mid-connect, not stuck in application logic.

Because the activation is configured with:

retry_state { max_attempts: 5, on_attempts_exceeded: ON_ATTEMPTS_EXCEEDED_DISCARD, delay_on_retry: 300 }

sustained network slowness makes all 5 attempts fail identically and the sync is then silently discarded — VSTS users' work item statuses never sync, with no user-visible signal.

Fix

Set processing_deadline_duration=30 on the task. 30s covers the two proxied calls at their full 10s HTTP timeout plus RPC overhead, with headroom.

This also aligns the task with its sibling sync_assignee_outbound, which already uses a 30s deadline for the same shape of work:

# src/sentry/integrations/tasks/sync_assignee_outbound.py
@instrumented_task(
    name="sentry.integrations.tasks.sync_assignee_outbound",
    namespace=integrations_tasks,
    processing_deadline_duration=30,
    ...

What was ruled out

  • Raising the HTTP timeout in the VSTS client — a separate concern. The client's 10s timeout is reasonable; the bug is that the task deadline didn't leave room for it. Changing it would mask the mismatch rather than fix it, and would affect every VSTS caller.
  • Reducing the number of RPC / proxy calls — the pre-PATCH work is ~2.5s and is not itself the failure. Caching or batching those calls is a possible follow-up optimization, but it isn't the root cause and wouldn't reliably keep the task under 10s.
  • Retry/backoff tuning — retries already exist; they all fail the same way because each attempt gets the same insufficient deadline. More attempts don't help.
  • Treating ProcessingDeadlineExceeded as a halt in IntegrationEventLifecycle.__exit__ (src/sentry/integrations/utils/metrics.py) — noted in triage as an optional secondary improvement to stop generating spurious Sentry issues for infra timeouts. Deliberately not included here: this PR fixes the root cause only. Worth doing separately, since after this change a deadline breach is a genuine signal rather than routine noise.

Testing

The dev environment in the session where this was authored had no provisioned .venv / Postgres, so the integration task suite could not be executed locally. The change is a single decorator keyword argument that is validated against the instrumented_task signature (processing_deadline_duration: int | datetime.timedelta | None); ruff check and ruff format --check pass on the modified file. Please let CI run tests/sentry/integrations/tasks/.

🤖 Generated with Claude Code

https://claude.ai/code/session_019zHLGtYfgXd9JfY3Ee66nT


Generated by Claude Code

sync_status_outbound did not set processing_deadline_duration, so it
inherited the 10s default. The task makes 3 RPC calls plus 2 proxied
Azure DevOps requests (~2.5s) before issuing the work-item PATCH,
leaving under 7.5s for a request whose own HTTP timeout is 10s. Under
normal network variance the PATCH exceeded the remaining budget and the
task was killed with ProcessingDeadlineExceeded; with all 5 retries
failing the same way, the status sync was silently discarded.

Align with the sibling sync_assignee_outbound task, which already uses a
30s deadline.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019zHLGtYfgXd9JfY3Ee66nT
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants