-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: Enable async component builds, synchronous Yarn execution #10716
Conversation
Changelog[uncommitted] (2022-09-14) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
One question, but I don't feel strongly about it.
@@ -152,8 +153,15 @@ func (e *Environment) PathExists(p string) bool { | |||
return err == nil | |||
} | |||
|
|||
var YarnInstallMutex sync.Mutex |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was initially slightly hesitant about publicly exporting this, but I think it's fine.
We've used yarn
pretty pervasively from early on, but npm
has evolved quite a bit since then. I've always wondered if we should consider moving away from yarn
and use standard npm
instead. (I wonder if npm
even has this timeout issue). I guess if we do ever get around to switching to npm
, we could still keep this publicly exported variable, we just wouldn't be using it anymore, since we would be running npm
rather than yarn
(unless there's a need for a similar mutex for npm
and we co-opt this mutex for that purpose as well).
To that end, should we use a more general name, like NodeInstallMutex
, or is it better to keep it yarn-specific since right now it really is only for any yarn
command? (I don't feel strongly either way and am OK with YarnInstallMutex
; I just wanted to raise the question in case anyone else has a stronger opinion).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it should be Yarn specific, we're working around a specific behavior with the mutex & timeout that yarn
implements. If npm, pnpm, or other packagers require a similar lock, we should separate them out into individual mutexes so that we can still parallelize.
bors merge |
Build succeeded: |
…umi/pulumi#10713 pulumi/pulumi#10714 pulumi/pulumi#10715 pulumi/pulumi#10716 pulumi/pulumi#10718 pulumi/pulumi#10719 pulumi/pulumi#10721 pulumi/pulumi#10722 pulumi/pulumi#10723 pulumi/pulumi#10724 10691: Bring back auto-install plugins r=Zaid-Ajaj a=Zaid-Ajaj <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This PR brings back pulumi/pulumi#10530 (cc `@AaronFriel)` Fixes pulumi/pulumi#10521 Fixes pulumi/pulumi#10216 After debugging for ages, it turns out that the [package](https://github.com/checkly/pulumi-checkly/tree/main/sdk/nodejs) that caused the p0/revert of the original PRs were faulty in the sense that they didn't use latest codegen to generate the nodejs package. The generated package.json had ```json { "pulumi": { "resource": true, "pluginDownloadURL": "https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2" } } ``` And couldn't auto install the third-party plugin because: - Doesn't have a `name` which is now going to be required for third-party packages when the package name is not ``@pulumi/{something}`(see` change in pulumi-language-nodejs/main.go) - When `pulumi.name` is missing, `pulumi-language-nodejs/main.go` tried to "derive" the plugin name by simply removing the "`@"` sign from the package name but that is still is wrong because the plugin name was "checkly" and the derived plugin name was "checkly/pulumi" hence why a `packageJson.pulumi.name` is now required - Had `pluginDownloadURL` property instead of `server` which is not read anywhere by Pulumi except from the original schema.json itself, not the generated package.json file where it should be transformed to `server` - Third-party package used a very old version of tfgen v3.19.0 which is from February this year. I've confirmed that after installing the plugin via the install script added to the package _and_ adding a name to the `pulumi` section that the error no longer shows since now ## Checklist <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Service, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. --> 10702: Prep for real 3.40.0 release r=AaronFriel a=AaronFriel This turns the "safeties off" after validating the bors workflow and ensuring everything up to the dispatch commands succeeds. It also fixes an error in `get-next-version` that resulted in the "vNext" PR pulumi/pulumi#10701 to calculate the wrong "next version", which was corrected by hand. (The branch name and commit history is evidence of the error.) 10704: ci: Add make changelog shortcut for creating changelog entries r=AaronFriel a=AaronFriel 10713: ci: Fix Windows flake with longer timeout r=AaronFriel a=AaronFriel This fixes Windows tests failing like this due to very short (1 second) timeouts. https://github.com/pulumi/pulumi/actions/runs/3054396239/jobs/4926314390 10714: ci: Reduce test duration by avoiding costly subtests in loop r=AaronFriel a=AaronFriel The loops in this file create millions of subtests, the majority of which do very little work. This reduces the time to execute these tests to a couple seconds. 10715: ci: Remove v0.10.0 compatibility test r=AaronFriel a=AaronFriel Removes a _very old_ and ostensibly disabled test. The condition in this test was actually incorrect: ```go func skipIfNotNode610(t *testing.T) { nodeVer, err := getNodeVersion() if err != nil && nodeVer.Major == 6 && nodeVer.Minor == 10 { t.Skip("Skipping 0.10.0 compat tests, because current node version is not 6.10.X") } } ``` This only skips **if** the node version is vMajor = 6 and vMinor == 10. I discovered this test from bisecting which test was installing a very old Pulumi library and running node-gyp to link an even older gRPC library. 10716: ci: Enable async component builds, synchronous Yarn execution r=AaronFriel a=AaronFriel The `make test_build` command is blocking for all integration tests, adding roughly 2 minutes of execution time to each. It runs a series of Node, Python, and Go CLI commands to configure providers. These build steps use a file lock to ensure only one runs at a time, and additionally parallelizes the three build steps. This moves the setup step into an asynchronous action performed on-demand, which enables greater concurrency and allows integration tests that perform _no_ setup to skip most of the costly build process altogether. The second commit, dependent on the first, adds a global mutex in the sdk library for running `yarn` commands. The Yarn CLI itself has a lock that prohibits concurrent execution, however it times out. On concurrent executions then, we get spurious test failures because `yarn` failed to execute waiting for a separate execution. These changes improve test reliability and reduce integration test execution time by roughly 2 minutes across all jobs. 10718: ci: Skip refresh on tests that should fail fast r=AaronFriel a=AaronFriel These tests performed unnecessary refresh steps. Happened to catch these while optimizing build times. The impact on execution times was negligible. 10719: ci: Use yarn link before install, ensure Pulumi latest is used r=AaronFriel a=AaronFriel These cut execution time on myriad node tests by 5 to 30 seconds each. Linking ``@pulumi/pulumi`` first, before `yarn install`, enables yarn to skip downloading and extracting the node SDK before replacing it with a symlink. 10721: ci: Increase bors timeout r=AaronFriel a=AaronFriel The default timeout is one hour, which our CI runs can exceed. This should give us a buffer while work is done to reduce execution time to a reasonable threshold. 10722: ci: Improve test parallelism, enabled by pulumi/pulumi#10263 r=AaronFriel a=AaronFriel These tests no longer mutate environment variables, instead using `e.SetEnvVars` to only set env vars on child processes since pulumi/pulumi#10263. 10723: ci: Reduce test duration with parallel subtests r=AaronFriel a=AaronFriel This change parallelizes subtests. Although there are only ~200 combinations tested, they are fully parallelizable and this reduces execution time for these tests from 80-100s to 10-15s each, reducing execution time for the package by about a minute or a little more. Each parent test is compute bound and already run in parallel, so the reduction isn't additive. 10724: ci: Enable robust retries on tests r=AaronFriel a=AaronFriel This adds a retry script to `./scripts/retry` which takes any shell command and retries it up to `PULUMI_TEST_RETRIES` times. On each iteration of the script, parallelism and variation is reduced. Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com> Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
The
make test_build
command is blocking for all integration tests, adding roughly 2 minutes of execution time to each. It runs a series of Node, Python, and Go CLI commands to configure providers. These build steps use a file lock to ensure only one runs at a time, and additionally parallelizes the three build steps.This moves the setup step into an asynchronous action performed on-demand, which enables greater concurrency and allows integration tests that perform no setup to skip most of the costly build process altogether.
The second commit, dependent on the first, adds a global mutex in the sdk library for running
yarn
commands. The Yarn CLI itself has a lock that prohibits concurrent execution, however it times out. On concurrent executions then, we get spurious test failures becauseyarn
failed to execute waiting for a separate execution.These changes improve test reliability and reduce integration test execution time by roughly 2 minutes across all jobs.