fix(testutils): Make _snuba_pool lazy for xdist per-worker URL isolation#112797
Draft
fix(testutils): Make _snuba_pool lazy for xdist per-worker URL isolation#112797
Conversation
In xdist, each worker starts its own Snuba container on a different port. The module-level _snuba_pool was created eagerly at import time — before pytest_configure ran and set the per-worker SENTRY_SNUBA URL — so all workers shared the default pool pointing at the wrong port. Changes: - snuba.py: Replace the eager module-level pool with a _SnubaPool proxy that rebuilds the pool on first use (or when the URL changes). Reads os.environ["_SENTRY_SNUBA_POOL_URL"] so it survives Django override_settings() resets that restore settings.SENTRY_SNUBA. - pytest/xdist.py: Rewrite get_snuba_url() to read env vars at call time (not import time) and parse PYTEST_XDIST_WORKER dynamically. Also switch Redis DB allocation to round-robin so > 7 workers don't crash. - pytest/sentry.py: Write the per-worker URL into both settings and os.environ["_SENTRY_SNUBA_POOL_URL"] in pytest_configure and pytest_runtest_setup, and force a pool rebuild when the URL changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In xdist shuffled runs each worker gets its own Snuba container on a different port (1230, 1231, 1232). The module-level
_snuba_poolinsnuba.pywas created eagerly at import time — beforepytest_configureran and setsettings.SENTRY_SNUBAto the per-worker URL. All workers therefore shared the default pool pointing at port 1218, causingConnection refusederrors under load.A secondary failure mode:
override_settings()restores the originalsettings.SENTRY_SNUBAvalue when its context exits, silently reverting any URL override made duringconfigure_for_worker.Changes
src/sentry/utils/snuba.py_snuba_pool(aurllib3HTTPConnectionPool) with a_SnubaPoollazy proxy class.os.environ["_SENTRY_SNUBA_POOL_URL"](if set) orsettings.SENTRY_SNUBAon first use, and rebuilds the inner pool whenever the URL changes.urlopen) is unchanged; no callers updated.src/sentry/testutils/pytest/xdist.pyget_snuba_url()to readPYTEST_XDIST_WORKERandSNUBA_PORT_BASEat call time rather than import time, so the correct URL is returned after the worker's env is set up.src/sentry/testutils/pytest/sentry.pyconfigure_for_worker(xdist) andpytest_runtest_setup, write the per-worker Snuba URL into bothsettings.SENTRY_SNUBAandos.environ["_SENTRY_SNUBA_POOL_URL"]. The env var survivesoverride_settings()resets that would otherwise revert the URL.Test plan
Verified by running the
shuffle-tests-across-shardsworkflow with 16 shards × 3 xdist workers: Snuba connection errors disappeared after this change.