Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ThreadPoolTaskRunner concurrency by copying max_workers on duplication #13943

Merged
merged 1 commit into from
Jun 11, 2024

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented Jun 11, 2024

Fixes issue with ThreadPoolTaskRunner concurrency by copying max_workers on task runner duplication. Now the max_workers setting is respected, and users can increase the amount of concurrent execution if desired.

Example

Execute 100 tasks concurrently (this would execute in batches previously):

from prefect import task, flow
from prefect.task_runners import ThreadPoolTaskRunner


@task
def sleepy_task(i):
    import time

    print(f"Sleepy task {i}")
    time.sleep(10)
    return i


@flow(task_runner=ThreadPoolTaskRunner(max_workers=100))
def test_flow():
    return sleepy_task.map(range(100))


if __name__ == "__main__":
    test_flow()

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.

For documentation changes:

  • This pull request includes redirect settings in mint.json for files that are removed or renamed.

For new functions or classes in the Python SDK:

  • This pull request includes helpful docstrings.
  • If a new Python file was added, this pull request contains a stub page in the Python SDK docs and an entry in docs/mint.json navigation.

@desertaxle desertaxle requested a review from a team as a code owner June 11, 2024 20:45
@desertaxle desertaxle added the fix A fix for a bug in an existing feature label Jun 11, 2024
@desertaxle desertaxle merged commit fa90e61 into main Jun 11, 2024
28 checks passed
@desertaxle desertaxle deleted the fix-thread-pool-tr-concurrency branch June 11, 2024 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix A fix for a bug in an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants