Skip to content

fix(silo): Reduce timeout and disable retries for synchronous proxy requests#116043

Draft
sentry[bot] wants to merge 2 commits into
masterfrom
seer/fix/silo-blocking-proxy-timeout
Draft

fix(silo): Reduce timeout and disable retries for synchronous proxy requests#116043
sentry[bot] wants to merge 2 commits into
masterfrom
seer/fix/silo-blocking-proxy-timeout

Conversation

@sentry
Copy link
Copy Markdown
Contributor

@sentry sentry Bot commented May 21, 2026

This PR addresses a "Blocking Operation" performance issue (SENTRY-5PX9) observed on the /api/0/organizations/{organization_id_or_slug}/issues/ endpoint.

Problem:
The ratelimit.__call__ middleware was blocking for ~2.4 seconds due to a synchronous HTTP call made by CellSiloClient.proxy_request() to an internal Cell Silo endpoint. This blocking was exacerbated by a long default timeout and inappropriate retry logic for synchronous proxy operations.

Changes:

  1. Reduced CellSiloClient timeout: The default timeout for CellSiloClient has been reduced from 30 seconds to 10 seconds. This prevents a single unresponsive Cell Silo from blocking a Control Silo request thread for an excessively long duration.
  2. Disabled retries for proxy_request: The proxy_request method now explicitly disables retries before making the super()._request() call. Synchronous proxy requests within a Django request cycle should fail fast rather than retrying with backoff, which would compound the blocking time and worsen user experience.

These changes ensure that synchronous cross-silo proxy calls are more resilient to slow Cell Silo responses and do not excessively block the Control Silo's request processing.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

Fixes SENTRY-5PX9

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label May 21, 2026
@shayna-ch shayna-ch self-assigned this May 21, 2026
@shayna-ch shayna-ch added the Trigger: getsentry tests Once code is reviewed: apply label to PR to trigger getsentry tests label May 21, 2026
Comment thread src/sentry/silo/client.py Outdated
Co-authored-by: sentry-warden[bot] <258096371+sentry-warden[bot]@users.noreply.github.com>
@github-actions github-actions Bot removed the Trigger: getsentry tests Once code is reviewed: apply label to PR to trigger getsentry tests label May 21, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on ac98529 in this run:

tests/sentry/middleware/integrations/parsers/test_jira.py::JiraRequestParserTest::test_get_response_routing_to_cell_sync_retry_errorslog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/middleware/integrations/parsers/test_jira.py:117: in test_get_response_routing_to_cell_sync_retry_errors
    assert len(responses.calls) == 6
E   assert 1 == 6
E    +  where 1 = len(<responses.CallList object at 0x7f92a145f0e0>)
E    +    where <responses.CallList object at 0x7f92a145f0e0> = responses.calls

@shayna-ch shayna-ch added the Trigger: getsentry tests Once code is reviewed: apply label to PR to trigger getsentry tests label May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Trigger: getsentry tests Once code is reviewed: apply label to PR to trigger getsentry tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant