Skip to content

Flaky unit test: GC-scheduled google-genai aclose tasks trip fail_on_leaked_tasks in unrelated tests #6789

Description

@hassannaftabb

Bug Description

pytest --unit intermittently errors at teardown of tests/test_plugin_google_realtime.py::test_output_streams_close_on_generation_complete:

ERROR at teardown of test_output_streams_close_on_generation_complete
E   Failed: Test leaked tasks:
E
E   Task Name   : Task-7047
E   Coroutine   : BaseApiClient.aclose
E   Location    : /opt/venv/lib/python3.12/site-packages/google/genai/_api_client.py:2213
E   State       : pending
E
E   Task Name   : Task-7048
E   Coroutine   : AsyncClient.aclose

The leaked coroutines belong to google-genai, not to the test. _make_session in that module already documents the mechanism:

Closed on exit so the genai http clients are released here instead of by AsyncClient.__del__, which schedules aclose() on whatever event loop is running when the collector happens to reach them.

That mitigation only covers the client this test builds. When the garbage collector finalizes any other genai client later in the run, AsyncClient.__del__ schedules aclose() on whichever loop is live at that moment. The autouse fail_on_leaked_tasks fixture in tests/conftest.py takes the non-concurrent path for this test and diffs asyncio.all_tasks() around the test body, so a GC-scheduled task that lands inside that window is attributed to this test even though nothing in it created the client.

This is a test-infrastructure flake, not a framework bug, but it can fail an otherwise clean unit-tests job.

Expected Behavior

pytest --unit is deterministic. A task scheduled by garbage-collector finalization of a client owned by another test should not be attributed to whichever test happens to be running.

Two directions, whichever you prefer:

  1. Treat GC-scheduled genai closes as ignorable in _is_ignorable_task (it already special-cases async_generator_athrow tasks created by GC finalization of async generators, which is the same class of problem).
  2. Close genai clients deterministically wherever they are constructed in the suite, so __del__ never has to schedule anything.

Reproduction Steps

# main @ 3568970, Linux, in a container so tests/test_room.py can start livekit-server
docker run -d --name lk --network host \
  -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":/app -w /app \
  -e UV_PROJECT_ENVIRONMENT=/opt/venv -e PYTHONPATH=/app python:3.12-slim sleep infinity
docker exec lk bash -c "pip install -q uv && uv sync --all-extras --dev --frozen && apt-get update -qq && apt-get install -y -qq docker.io"

# run the full unit suite repeatedly; it does not fail every time
for i in 1 2 3 4 5 6; do docker exec lk uv run --no-sync pytest --unit -q | tail -1; done

# observed on one branch of mine: 4 consecutive runs with the error, then 2 clean,
# with no code change between them. Passes in isolation every time:
docker exec lk uv run --no-sync pytest tests/test_plugin_google_realtime.py --unit -q

It only reproduces in a full-suite run, never when the module runs alone, which fits GC timing rather than anything in the test.

Operating System

Linux (python:3.12-slim container, 24 cores) on a Windows 11 host via Docker Desktop

Models Used

N/A — no model is contacted. The test monkeypatches GOOGLE_API_KEY and cancels the connect loop before any websocket is opened; the leaked tasks come from the genai HTTP client being finalized.

Package Versions

livekit-agents 1.6.9 (repo main @ 3568970)
livekit 1.1.14
google-genai 2.11.0
pytest 9.0.3
pytest-asyncio 1.4.0
pytest-asyncio-concurrent 0.5.2
python 3.12.13

Additional Context

Separately, pytest --audio_eot is at 8 failed / 75 passed on main in the same environment, and no workflow in .github/workflows/ selects that category. Happy to open that as its own issue if it isn't already tracked under the turn-detection test work in #6715.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions