[Automation API] Support parallel execution of nodejs inline programs #5449
Assignees
Labels
area/automation-api
kind/enhancement
Improvements or new features
language/javascript
resolution/fixed
This issue was fixed
Milestone
Comments
|
To be clear for nodejs users, the Automation API allows one concurrent update per node process for the time being. If you want to run multiple updates in parallel, they will need to be in separate node processes. |
2 tasks
pulumi/pulumi or @pulumi/pulumi ? |
|
https://nodejs.org/api/async_context.html#new-asynclocalstorage could probably solve this now, similar to how we use AsyncLocal in dotnet. |
This was referenced Aug 29, 2022
Merged
3 tasks
Does this still relevant ? |
|
It's not still relevant, automation API should correctly run updates in parallel now. |
justinvp
added a commit
that referenced
this issue
Oct 10, 2023
This change removes the SxS check in the Node.js Automation API since it's blocking a customer scenario and isn't needed anymore now that #5449 has been addressed with #10568. The check was originally added in #7349 to provide a helpful error message when multiple versions of `@pulumi/pulumi` were used with inline programs, which could cause clashes with global state. Since then, we've changed how state is stored to allow parallel execution of multiple inline programs with #10568. However, the SxS checks were not removed as part of that change. A customer recently hit the error associated with the SxS check. They are creating a Next.js program that runs Pulumi operations as inline programs. Next.js ends up loading modules multiple times in a way that confuses the SxS checker, causing the error to be thrown, even though it wouldn't have been a problem.
justinvp
added a commit
that referenced
this issue
Oct 10, 2023
This change removes the SxS check in the Node.js Automation API since it's blocking a customer scenario and isn't needed anymore now that #5449 has been addressed with #10568. The check was originally added in #7349 to provide a helpful error message when multiple versions of `@pulumi/pulumi` were used with inline programs, which could cause clashes with global state. Since then, we've changed how state is stored to allow parallel execution of multiple inline programs with #10568. However, the SxS checks were not removed as part of that change. A customer recently hit the error associated with the SxS check. They are creating a Next.js program that runs Pulumi operations as inline programs. Next.js ends up loading modules multiple times in a way that confuses the SxS checker, causing the error to be thrown, even though it wouldn't have been a problem.
github-merge-queue bot
pushed a commit
that referenced
this issue
Oct 11, 2023
This change removes the SxS check in the Node.js Automation API since it's blocking a customer scenario and isn't needed anymore now that #5449 has been addressed with #10568. The check was originally added in #7349 to provide a helpful error message when multiple versions of `@pulumi/pulumi` were used with inline programs, which could cause clashes with global state. Since then, we've changed how state is stored to allow parallel execution of multiple inline programs with #10568. However, the SxS checks were not removed as part of that change. A customer recently hit the error associated with the SxS check. They are creating a Next.js program that runs Pulumi operations as inline programs. Next.js ends up loading modules multiple times in a way that confuses the SxS checker, causing the error to be thrown, even though it wouldn't have been a problem. Fixes #14128
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The Pulumi nodejs runtime implementation uses quite a bit of global state: https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/runtime/settings.ts#L188
We'll need to refactor all usage of global state out of the node SDK to allow Automation API to execute concurrent programs on different stacks that use the inline/hosted execution model.
One consequence of this is that multiple versions of pulumi can cause problems due to conflicting module level global state. Automation API users should pin to a single version of pulumi/pulumi via yarn
resolutions: https://classic.yarnpkg.com/en/docs/selective-version-resolutions/The text was updated successfully, but these errors were encountered: