Skip to content

Adopt asyncio.TaskGroup for structured concurrency #765

@vdusek

Description

@vdusek

Summary

Once Python 3.10 support is dropped, several places in the codebase can benefit from asyncio.TaskGroup (introduced in Python 3.11) for proper structured concurrency.

High Priority

src/apify/request_loaders/_apify_request_list.py:145-161 — Remote URL fetching

Current code uses create_task + add_done_callback + gather. The callbacks spawn fire-and-forget asyncio.create_task calls inside lambdas — those secondary tasks are never awaited. If create_requests_from_response fails, the error is silently lost.

Wins:

  • Eliminates fire-and-forget tasks that can silently swallow errors.
  • All tasks (fetch + process) are managed under one scope.
  • Errors propagate properly via ExceptionGroup.

Medium Priority

src/apify/_actor.py:1187 — Reboot listener dispatch

asyncio.gather fires persist-state and migrating listeners concurrently. If one listener raises, the others are left running unmanaged.

Wins:

  • All listener tasks are cancelled on first failure instead of being left dangling.
  • Produces an ExceptionGroup with all errors, improving debuggability.

Low Priority

src/apify/events/_apify_event_manager.py:76 — Background WebSocket task

A long-running create_task with manual cancel + suppress(CancelledError) in __aexit__. Could be replaced by a TaskGroup owned by the context manager, but the task must outlive __aenter__, so it would require restructuring the class.

Wins:

  • Cleaner lifecycle management (no manual cancel/suppress boilerplate).
  • Requires class restructuring, so benefit-to-effort ratio is lower.

Not Applicable

src/apify/scrapy/_async_thread.py:103-113 — Shutdown tasks

Cancels pre-existing tasks and awaits them with gather(*tasks, return_exceptions=True). TaskGroup is for spawning new tasks, not for wrangling already-running ones. Leave as-is.

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions