Skip to content

A snippet for running multiple, concurrent invocations of a Python function

License

Notifications You must be signed in to change notification settings

alexwlchan/concurrently

Repository files navigation

concurrently

I often use the following pattern in Python:

for task in get_tasks_to_do():
    perform(task)

Tasks run one after the other: task 1 runs to completion, then task 2 runs to completion, then task 3 runs to completion, and so on until everything is done.

This is fine for certain classes of task, but if perform() is heavily I/O bound, it's unnecessarily slow. If I could run multiple instances of perform() concurrently, the overall process would complete much faster. Task 1 could start, make a network request, then task 2 could start while task 1 is waiting.

I wrote my recipe for concurrent processing in a blog post in 2019, which explains how it works – but the code was a little cumbersome to use. This repo is a tidied up (and tested!) version of that code.

It allows me to write code like:

from concurrently import concurrently

for (input, output) in concurrently(handler=perform, inputs=get_tasks_to_do()):
    print(input, output)

It yields both the input and the output, because results may not come in the original order of inputs.

I would recommend using this code instead of the code in the original blog post.

Usage

Copy and paste the file concurrently.py into your project.

You can see examples in the examples directory.

License

MIT.

About

A snippet for running multiple, concurrent invocations of a Python function

Resources

License

Stars

Watchers

Forks

Languages