redis-py 8 hangs on Python 3.10/3.11 — blocks the redis v8 upgrade (#6764)

[masenf]

This is the blocker for #6764 (support redis v8). #7019 attempted the widening (redis >=6.4,<8.0 → <9.0) and backed it out; the bound is still <8.0 on main.

Rewritten 2026-09-01. The original version of this issue was filed mid-investigation and was both too narrow and wrong in one claim: it said the fixture leak fix in #7019 "removes the hang". It does not — that fix removed one orphaned task, and three separate hangs remained.

The blocker

With REFLEX_REDIS_URL set, three redis-backed tests hang indefinitely on Python 3.10 and 3.11 under redis-py 8.1.0. They pass on 3.12, 3.13 and 3.14.

Test	Where it hangs
tests/units/test_app.py::test_initialize_with_state	in the test body, awaiting app.state_manager.get_state(...) — parked in the event-loop selector with timeout = -1
tests/units/istate/manager/test_redis.py::test_modify_oplock	async-fixture finalizer teardown
tests/units/test_state.py::test_on_load_internal_supersedes_previous_navigation	event-loop / runner teardown

The first one is why this blocked the upgrade. It is not a teardown artifact — it hangs on get_state, a normal runtime path. Two of the three are teardown, which is easy to dismiss as test-only; this one is not.

Reflex supports 3.10 and 3.11 (requires-python = ">=3.10,<4.0"), so this is a supported-version issue.

A/B that isolates it to redis-py 8

Same interpreter, same machine, same command, nothing else running, against a flushed redis:

py3.11 + redis-py 7.4.1  →  0 hangs, 120s
py3.11 + redis-py 8.1.0  →  3 hangs (60s pytest-timeout each), 300s

Not a pre-existing 3.11 problem, and not environment noise. (15 unrelated failures appear in both runs — pyi-golden tests needing a working ruff, and two IPv6-dependent test_processes tests. Those are local-environment artifacts, not part of this.)

Also worth recording, because it misled me at first: an early measurement suggested 3.11 was broadly ~5× slower with redis. That was wrong — it was contamination from concurrent test runs sharing one redis instance. Measured clean, the suite is ~2 min on both 3.11 and 3.14, and tests/units/istate/manager/test_redis.py is 0.78s on both. There is no general slowdown; there are exactly three discrete hangs.

How to reproduce

uv sync -p 3.11                      # or 3.10
uv pip install pytest-timeout
redis-server --daemonize yes --port 6379 --save ''
export REFLEX_REDIS_URL=redis://localhost:6379
# temporarily widen the bound in pyproject.toml to <9.0 and re-lock
uv run pytest tests/units -p no:randomly --timeout=60 --timeout-method=signal -q

Use --timeout-method=signal, not thread. The hanging tests are async, so with thread the stack dump contains only pytest's own frames and never names the test; signal raises inside the test, names it, and continues so you see all three in one run.

How it presents in CI (important for triage)

The unit-tests workflow does not go red — it is cancelled. Run unit tests w/ redis sits ~26 minutes on ubuntu 3.10/3.11 until the 30-minute job timeout kills the run, while 3.12–3.14 finish the same step in 1–2 minutes.

Because the conclusion is cancelled, it reads as "nothing failed" on the checks API and in the check_suite.completed webhook. Anyone retrying this upgrade should treat cancelled as not-green; it is how this hid twice during #7019.

A related, still-unexplained detail

One of the teardown hangs was instrumented at the exact hang point (asyncio.runners._cancel_all_tasks), with an orphaned StateManagerRedis whose _lock_task was left running:

[dump] _cancel_all_tasks: 1 task(s)
[dump]   name='reflex_ensure_task|StateManagerRedis._lock_task=_subscribe_lock_updates|...'
[dump] AFTER 15s: 1 task(s) still not done
[dump]   STUCK cancelling=1
  File "reflex/utils/tasks.py", line 43, in _run_forever
    await coro_function(*args, **kwargs)

cancelling=1 means the cancellation was delivered, yet the task is awaiting a newly created future — something swallowed the CancelledError and issued a fresh await, so it never dies.

It does not appear to be our code:

• _run_forever (reflex/utils/tasks.py) explicitly re-raises asyncio.CancelledError and RuntimeError before its suppress_exceptions handling.
• _subscribe_lock_updates (reflex/istate/manager/redis.py) is a plain async with self.redis.pubsub() + async for _ in pubsub.listen(), with no handling on the cancel path.

And it does not reproduce in isolation. A minimal script cancelling a task blocked in pubsub.listen() inside async with, using the same Redis.from_url(..., retry_on_error=[RedisError]) client Reflex builds, cancels cleanly on both 3.11 and 3.14 under 8.1.0. Cancelling the orphaned _lock_task directly, rather than via _cancel_all_tasks during loop close, also cancels cleanly. The hang needs the loop-shutdown path specifically.

Unverified hypothesis, needs confirming before anyone acts on it: redis-py 8.0 raised the default retry policy from 3 to 10 attempts with exponential jitter backoff. If pubsub teardown re-awaits through that retry machinery while the loop is closing, each retry could reset the cancellation.

Already done in #7019 — do not redo

• Response-type narrowing in istate/manager/redis.py and utils/token_manager.py (bytes | str → bytes | None at the call sites, since Reflex builds its clients without decode_responses). redis-py 8's stricter command overloads require it. Verified pyright-clean against both 7.4.1 and 8.1.0, so it is merged and the eventual upgrade does not need it again.
• A genuine resource leak fixed in tests/units/conftest.py: StateManager.create() was called before branching on the fixture param, so with REFLEX_REDIS_URL set a live StateManagerRedis (with a running _lock_task) was built and then discarded unclosed for the in_process/disk params. Worth having regardless of redis version — but note it did not fix this issue.
• redis filtered out of the backend outdated-dependency check in .github/workflows/check_outdated_dependencies.yml, with a pointer here, so it stops reporting as a perpetually-stale row. Remove it from that filter when the bound widens.

What needs doing

• Reproduce the three hangs per the recipe above on 3.10/3.11
• Root-cause test_initialize_with_state first — it is the one on a real runtime path (get_state) and decides whether this is a user-facing bug or only a teardown/test concern
• Confirm or eliminate the retry-backoff hypothesis, e.g. by constructing the client with retry=Retry(NoBackoff(), 0)
• Establish what changed in 3.12 that makes it survive (candidates: Task.uncancel()/cancelling() semantics, the _cancel_all_tasks implementation, or the eager task factory)
• Decide the fix: constrain the retry policy on the clients built in reflex/utils/prerequisites.py, shield cleanup in _run_forever, or carry an upstream fix
• If upstream, file against redis-py with the minimal reproducer
• Widen the bound to <9.0, remove redis from the outdated-check filter, and close support redis v8 #6764

Worth doing alongside, separately

This failure mode cost ~26 min of runner time per affected job and surfaced as cancelled rather than failure. A per-test timeout in the unit suite (pytest-timeout with a generous default) would turn any future hang into a fast, correctly-attributed failure with a stack instead of a silent 30-minute cancellation. That is CI hygiene, not part of this bug.

[linear-code]

ENG-11996