[SDK/Python] Parallelism Flag Not Respected #11116
Labels
area/language-host
Runtime that executes user programs
impact/performance
Something is slower than expected
kind/bug
Some behavior is incorrect or out of spec
language/python
resolution/fixed
This issue was fixed
size/S
Estimated effort to complete (1-2 days).
Milestone
What happened?
Python programs do not respect the
--parallel
flag. While the--parallel
flag is plumbed through to the language host, and even into the the SDK atpulumi/runtime/settings.py
, the value is never used.Certain rpc calls generate blocking futures that the runtime offloads to a worker thread. For example, any use of
run_in_executor
throws the future into a worker thread. (Here's where we make that call.)The default executor sets the max number of worker threads using a fixed value scaling based on the number of CPUs on the machine. This is the value that's currently employed.
This issue is to replace the default executor with one that respects the
--parallel
flag.Steps to reproduce
Run the following program. Even though parallelism is unbounded by default, the maximum number of concurrently created resources is limited.
Expected Behavior
All resources should be created concurrently.
Actual Behavior
Only about 20 resources are created at a time.
Output of
pulumi about
Additional context
I'm currently working on a fix. This issue is also a program for the Automation API. I'm going to open a separate issue for that since I'm planning to separate the PRs for the respective fixes.
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
The text was updated successfully, but these errors were encountered: