Skip to content

fix(integrations): treat 5xx SCM responses as halts, not SLO failures - #120733

Draft
billyvg wants to merge 1 commit into
masterfrom
claude/loving-hypatia-lvd96t
Draft

fix(integrations): treat 5xx SCM responses as halts, not SLO failures#120733
billyvg wants to merge 1 commit into
masterfrom
claude/loving-hypatia-lvd96t

Conversation

@billyvg

@billyvg billyvg commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

Fixes a noisy instrumentation issue where transient third-party 5xx responses during stacktrace link resolution were creating Sentry issues via record_failure(create_issue=True).

Sentry issue: https://sentry.io/organizations/sentry/issues/7549081279/
Dispatched by: automated triage routine (Claude session https://claude.ai/code/session_01N4onAVtnECuj7KrXLyavhg)


Root-cause analysis

When Bitbucket's API returns a 503 Service Unavailable during a check_file call (stacktrace link resolution), the error path was:

  1. ProjectStacktraceLinkEndpoint.getget_stacktrace_configget_link (stacktrace_link.py)
  2. get_link calls install.get_stacktrace_link(...), catching only ApiError with code == 403 and re-raising everything else
  3. BitbucketIntegration.get_stacktrace_linkRepositoryIntegration.check_file in source_code_management/repository.py
  4. Inside check_file, client.check_file() raises ApiError(code=503) from the HTTP response
  5. The except ApiError as e: handler only halted on codes 404 and 400; the else: raise branch re-raised 5xx errors
  6. The re-raised exception propagated out of the with self.record_event(...).capture() as lifecycle: block
  7. IntegrationEventLifecycle.__exit__ called self.record_failure(exc_value, create_issue=True) — creating a Sentry issue for an external service being temporarily unavailable

This produced false-positive SLO failures for a condition entirely outside Sentry's control.


What the fix does

In RepositoryIntegration.check_file (src/sentry/integrations/source_code_management/repository.py), added a 5xx guard inside the except ApiError as e: handler, immediately after the existing 404/400 halt:

elif e.code is not None and 500 <= e.code < 600:
    # Transient third-party service errors (5xx) indicate temporary
    # unavailability of the external SCM provider, not a bug in Sentry.
    # Record as a halt rather than a failure to avoid creating spurious
    # Sentry issues for conditions we cannot control.
    lifecycle.record_halt(e)
    return None

record_halt (unlike record_failure) does not call sentry_sdk.capture_exception and does not create a Sentry issue by default. This matches how 404/400 responses are already handled — they represent expected or benign conditions that don't warrant an issue.

Returning None causes get_stacktrace_link to return None normally (success path), so neither the inner CHECK_FILE nor the outer GET_STACKTRACE_LINK lifecycle records a failure.


What was considered and ruled out

  • Fixing in get_link (stacktrace_link.py): That function only sees the exception after it has already escaped both lifecycle context managers (triggering two record_failure calls). Fixing there would suppress the duplicate re-raise but would not prevent the instrumentation noise at the source.
  • Fixing in get_stacktrace_link (repository.py): Similar problem — by the time the exception reaches get_stacktrace_link's outer try/except, the CHECK_FILE lifecycle has already recorded a failure. The fix needs to be at the point where the ApiError is first caught inside the lifecycle.
  • Simply downgrading a log level: The issue is create_issue=True in record_failure, not log level. Changing a log level would not prevent the Sentry issue from being created.
  • Also fixing ApiRetryError for non-GitLab providers: ApiRetryError (code=503, from requests.adapters.RetryError) is caught in a separate except clause before except ApiError. The existing TODO comment indicates that behavior is intentionally deferred. This fix only addresses plain ApiError 5xx responses from HTTP responses, which is the path the reported issue takes.

Tests

Added two tests to tests/sentry/integrations/bitbucket/test_client.py:

  • test_check_file_5xx_returns_none_and_records_halt: verifies that a 503 from Bitbucket causes check_file to return None and records HALTED (not FAILURE) in the lifecycle
  • test_get_stacktrace_link_5xx_returns_none: verifies end-to-end that no FAILURE outcome is recorded when the SCM provider is temporarily unavailable

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.


Generated by Claude Code

When a third-party SCM provider (e.g. Bitbucket) returns a 5xx response
during `check_file`, the ApiError previously propagated out of the
IntegrationEventLifecycle context manager, triggering `record_failure`
with `create_issue=True`. This caused spurious Sentry issues for
conditions that are transient and outside our control.

Fix: add a 5xx guard in `RepositoryIntegration.check_file`'s
`except ApiError` handler that calls `lifecycle.record_halt(e)` and
returns `None`, consistent with how 404/400 responses are already
handled.

Fixes https://sentry.io/organizations/sentry/issues/7549081279/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4onAVtnECuj7KrXLyavhg
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on e32c5ee in this run:

tests/sentry/integrations/gitlab/test_client.py::GitlabRefreshAuthTest::test_get_stacktrace_link_restricted_ip_addresslog
[gw1] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/integrations/gitlab/test_client.py:272: in test_get_stacktrace_link_restricted_ip_address
    with pytest.raises(ApiHostError):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   Failed: DID NOT RAISE <class 'sentry.shared_integrations.exceptions.ApiHostError'>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants