Start policy packs in parallel #14495
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently we start up and thus compile policy packs one by one. When multiple policy packs need to be loaded, this increases the start up time substantially. In previous tests plugins took ~1-3s to start up, so having multiple of these the time adds up quickly. Loading them in parallel will help reduce the startup time here.
Changelog[uncommitted] (2023-11-20)Features
|
Frassle
reviewed
Nov 13, 2023
Yeah good point. I pushed up a commit that avoids doing the installation step in parallel, where npm/pip installing is being run, but that still loads the policies in a goroutine. This won't result in much of a performance improvement when policies need to be installed, though that should be the much less common case, as I'd expect users to usually have the policies cached locally already, and they only need to be run. |
|
Yeh that seems like a good balance to start with. |
Frassle
reviewed
Nov 13, 2023
I'm actually struggling to find any tests that load even a single policy pack, so I'm gonna look into adding tests. |
|
I added a few tests here, though they are all integration tests. I felt like that made most sense, but I'd be curious to hear if anyone has other ideas on how to test this. |
tgummerer
force-pushed
the
tg/start-plugins-parallel
branch
from
dba5acf
to
fbf0bb4
Compare
Frassle
changed the title
Frassle
approved these changes
Nov 16, 2023
tgummerer
force-pushed
the
tg/start-plugins-parallel
branch
from
1f9004f
to
0709a79
Compare
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
We have utf-8 characters in the command line, and should be supporting testing those. Explicity specify utf-8 encoding when reading the integration test files, so we don't fail on some platform where the default encoding might not allow those characters.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 30, 2023
### Features - [cli/config] Include config values from ESC in `pulumi config` [#14560](#14560) - [cli/config] Add commands for managing stack environments [#14628](#14628) - [cli/config] Add a command to create an ESC environment from stack config [#14634](#14634) - [sdk/go] add optional display name and tag fields to project templates [#14587](#14587) - [cli/plugin] Load policy packs in parallel on startup to reduce startup time [#14495](#14495) - [sdkgen/{go,nodejs,python}] Resource methods with plain: true outputs can now return plain values without an Output wrapper. In particular, this feature enables resource methods to serve as explicit provider factories by returning preconfigured explicit providers. [#13592](#13592) ### Bug Fixes - [auto/go] Fix a datarace in cloning git repos. [#14643](#14643) - [auto/go] Fixes event stream lag on windows runtime [#14659](#14659) - [engine] Engine now correctly handles any resource name. [#14107](#14107) - [engine] Fix a panic in cancellation. [#14612](#14612) - [engine] Fix root directory passed to langauge plugins when starting pulumi in a subfolder. [#14684](#14684) - [sdkgen] Schemas now validate that 'urn' and 'id' are not used as resource output properties. [#14637](#14637) - [sdkgen] Fixes marshalling the "plain" flag from object or resource properties [#14648](#14648) - [programgen/nodejs] Fix generated readFile function so that it includes the encoding and returns a string [#14633](#14633) - [sdkgen/{dotnet,go,nodejs,python}] No longer writing out name and project from alias definitions into SDKs, only type [#14625](#14625) - [sdk/go] Fix optional handling on nested props [#14629](#14629) - [sdkgen/go] Fixes plain and optional properties for generated types for Go SDKs using generics [#14616](#14616) - [sdkgen/go] Generate non-plain type variants for types used as inputs inside unions [#14679](#14679) - [sdk/python] Introduces RuntimeError when we detect a cycle upon adding dependencies to the graph. Additionally adds "PULUMI_ERROR_ON_DEPENDENCY_CYCLES" as a new environment variable to control this behavior. Set to `False` to return to the previous behavior, which could potentially re-introduce infinite hangs for some programs. [#14597](#14597)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently we start up and thus compile policy packs one by one. When multiple policy packs need to be loaded, this increases the start up time substantially. In previous tests plugins took ~1-3s to start up, so having multiple of these the time adds up quickly.
Loading them in parallel will help reduce the startup time here.
Fixes #14454
Checklist
make tidyto update any new dependenciesmake lintto verify my code passes the lint checkgofumptmake changelogand committed thechangelog/pending/<file>documenting my change