pulumi convert fails when it can't find resource plugins
#10216
Labels
area/codegen
SDK-gen, program-gen, convert
kind/bug
Some behavior is incorrect or out of spec
resolution/fixed
This issue was fixed
Milestone
Comments
Zaid-Ajaj
added
kind/bug
3 tasks
bors bot
added a commit
that referenced
this issue
Sep 14, 2022
10691: Bring back auto-install plugins r=Zaid-Ajaj a=Zaid-Ajaj <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This PR brings back #10530 (cc `@AaronFriel)` Fixes #10521 Fixes #10216 After debugging for ages, it turns out that the [package](https://github.com/checkly/pulumi-checkly/tree/main/sdk/nodejs) that caused the p0/revert of the original PRs were faulty in the sense that they didn't use latest codegen to generate the nodejs package. The generated package.json had ```json { "pulumi": { "resource": true, "pluginDownloadURL": "https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2" } } ``` And couldn't auto install the third-party plugin because: - Doesn't have a `name` which is now going to be required for third-party packages when the package name is not ``@pulumi/{something}`(see` change in pulumi-language-nodejs/main.go) - When `pulumi.name` is missing, `pulumi-language-nodejs/main.go` tried to "derive" the plugin name by simply removing the "`@"` sign from the package name but that is still is wrong because the plugin name was "checkly" and the derived plugin name was "checkly/pulumi" hence why a `packageJson.pulumi.name` is now required - Had `pluginDownloadURL` property instead of `server` which is not read anywhere by Pulumi except from the original schema.json itself, not the generated package.json file where it should be transformed to `server` - Third-party package used a very old version of tfgen v3.19.0 which is from February this year. I've confirmed that after installing the plugin via the install script added to the package _and_ adding a name to the `pulumi` section that the error no longer shows since now ## Checklist <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Service, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com>
bors bot
added a commit
that referenced
this issue
Sep 14, 2022
10691: Bring back auto-install plugins r=Zaid-Ajaj a=Zaid-Ajaj <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This PR brings back #10530 (cc `@AaronFriel)` Fixes #10521 Fixes #10216 After debugging for ages, it turns out that the [package](https://github.com/checkly/pulumi-checkly/tree/main/sdk/nodejs) that caused the p0/revert of the original PRs were faulty in the sense that they didn't use latest codegen to generate the nodejs package. The generated package.json had ```json { "pulumi": { "resource": true, "pluginDownloadURL": "https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2" } } ``` And couldn't auto install the third-party plugin because: - Doesn't have a `name` which is now going to be required for third-party packages when the package name is not ``@pulumi/{something}`(see` change in pulumi-language-nodejs/main.go) - When `pulumi.name` is missing, `pulumi-language-nodejs/main.go` tried to "derive" the plugin name by simply removing the "`@"` sign from the package name but that is still is wrong because the plugin name was "checkly" and the derived plugin name was "checkly/pulumi" hence why a `packageJson.pulumi.name` is now required - Had `pluginDownloadURL` property instead of `server` which is not read anywhere by Pulumi except from the original schema.json itself, not the generated package.json file where it should be transformed to `server` - Third-party package used a very old version of tfgen v3.19.0 which is from February this year. I've confirmed that after installing the plugin via the install script added to the package _and_ adding a name to the `pulumi` section that the error no longer shows since now ## Checklist <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Service, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com>
abhinav
pushed a commit
to pulumi/pulumi-dotnet
that referenced
this issue
Jan 11, 2023
…umi/pulumi#10713 pulumi/pulumi#10714 pulumi/pulumi#10715 pulumi/pulumi#10716 pulumi/pulumi#10718 pulumi/pulumi#10719 pulumi/pulumi#10721 pulumi/pulumi#10722 pulumi/pulumi#10723 pulumi/pulumi#10724 10691: Bring back auto-install plugins r=Zaid-Ajaj a=Zaid-Ajaj <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This PR brings back pulumi/pulumi#10530 (cc `@AaronFriel)` Fixes pulumi/pulumi#10521 Fixes pulumi/pulumi#10216 After debugging for ages, it turns out that the [package](https://github.com/checkly/pulumi-checkly/tree/main/sdk/nodejs) that caused the p0/revert of the original PRs were faulty in the sense that they didn't use latest codegen to generate the nodejs package. The generated package.json had ```json { "pulumi": { "resource": true, "pluginDownloadURL": "https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2" } } ``` And couldn't auto install the third-party plugin because: - Doesn't have a `name` which is now going to be required for third-party packages when the package name is not ``@pulumi/{something}`(see` change in pulumi-language-nodejs/main.go) - When `pulumi.name` is missing, `pulumi-language-nodejs/main.go` tried to "derive" the plugin name by simply removing the "`@"` sign from the package name but that is still is wrong because the plugin name was "checkly" and the derived plugin name was "checkly/pulumi" hence why a `packageJson.pulumi.name` is now required - Had `pluginDownloadURL` property instead of `server` which is not read anywhere by Pulumi except from the original schema.json itself, not the generated package.json file where it should be transformed to `server` - Third-party package used a very old version of tfgen v3.19.0 which is from February this year. I've confirmed that after installing the plugin via the install script added to the package _and_ adding a name to the `pulumi` section that the error no longer shows since now ## Checklist <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Service, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. --> 10702: Prep for real 3.40.0 release r=AaronFriel a=AaronFriel This turns the "safeties off" after validating the bors workflow and ensuring everything up to the dispatch commands succeeds. It also fixes an error in `get-next-version` that resulted in the "vNext" PR pulumi/pulumi#10701 to calculate the wrong "next version", which was corrected by hand. (The branch name and commit history is evidence of the error.) 10704: ci: Add make changelog shortcut for creating changelog entries r=AaronFriel a=AaronFriel 10713: ci: Fix Windows flake with longer timeout r=AaronFriel a=AaronFriel This fixes Windows tests failing like this due to very short (1 second) timeouts. https://github.com/pulumi/pulumi/actions/runs/3054396239/jobs/4926314390 10714: ci: Reduce test duration by avoiding costly subtests in loop r=AaronFriel a=AaronFriel The loops in this file create millions of subtests, the majority of which do very little work. This reduces the time to execute these tests to a couple seconds. 10715: ci: Remove v0.10.0 compatibility test r=AaronFriel a=AaronFriel Removes a _very old_ and ostensibly disabled test. The condition in this test was actually incorrect: ```go func skipIfNotNode610(t *testing.T) { nodeVer, err := getNodeVersion() if err != nil && nodeVer.Major == 6 && nodeVer.Minor == 10 { t.Skip("Skipping 0.10.0 compat tests, because current node version is not 6.10.X") } } ``` This only skips **if** the node version is vMajor = 6 and vMinor == 10. I discovered this test from bisecting which test was installing a very old Pulumi library and running node-gyp to link an even older gRPC library. 10716: ci: Enable async component builds, synchronous Yarn execution r=AaronFriel a=AaronFriel The `make test_build` command is blocking for all integration tests, adding roughly 2 minutes of execution time to each. It runs a series of Node, Python, and Go CLI commands to configure providers. These build steps use a file lock to ensure only one runs at a time, and additionally parallelizes the three build steps. This moves the setup step into an asynchronous action performed on-demand, which enables greater concurrency and allows integration tests that perform _no_ setup to skip most of the costly build process altogether. The second commit, dependent on the first, adds a global mutex in the sdk library for running `yarn` commands. The Yarn CLI itself has a lock that prohibits concurrent execution, however it times out. On concurrent executions then, we get spurious test failures because `yarn` failed to execute waiting for a separate execution. These changes improve test reliability and reduce integration test execution time by roughly 2 minutes across all jobs. 10718: ci: Skip refresh on tests that should fail fast r=AaronFriel a=AaronFriel These tests performed unnecessary refresh steps. Happened to catch these while optimizing build times. The impact on execution times was negligible. 10719: ci: Use yarn link before install, ensure Pulumi latest is used r=AaronFriel a=AaronFriel These cut execution time on myriad node tests by 5 to 30 seconds each. Linking ``@pulumi/pulumi`` first, before `yarn install`, enables yarn to skip downloading and extracting the node SDK before replacing it with a symlink. 10721: ci: Increase bors timeout r=AaronFriel a=AaronFriel The default timeout is one hour, which our CI runs can exceed. This should give us a buffer while work is done to reduce execution time to a reasonable threshold. 10722: ci: Improve test parallelism, enabled by pulumi/pulumi#10263 r=AaronFriel a=AaronFriel These tests no longer mutate environment variables, instead using `e.SetEnvVars` to only set env vars on child processes since pulumi/pulumi#10263. 10723: ci: Reduce test duration with parallel subtests r=AaronFriel a=AaronFriel This change parallelizes subtests. Although there are only ~200 combinations tested, they are fully parallelizable and this reduces execution time for these tests from 80-100s to 10-15s each, reducing execution time for the package by about a minute or a little more. Each parent test is compute bound and already run in parallel, so the reduction isn't additive. 10724: ci: Enable robust retries on tests r=AaronFriel a=AaronFriel This adds a retry script to `./scripts/retry` which takes any shell command and retries it up to `PULUMI_TEST_RETRIES` times. On each iteration of the script, parallelism and variation is reduced. Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com> Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
3 tasks
bors bot
added a commit
that referenced
this issue
Mar 21, 2023
12441: Move automatic plugin install to engine r=Frassle a=Frassle <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> We added auto-downloading of plugins in #10530, to resolve #5619 and #10216. While we added this to the general plugin logic it was in fact only intended (and only really useful) for provider plugins. After the introduction of this we actually got a few confused questions by people missing language plugins asking why "pulumi plugin install language nodejs" didn't work as the error message had told them to run that. At any rate, while putting the auto-download logic in GetPluginInfo/Path worked for official pulumi plugins, it didn't work for any third-party plugins because the interface to GetPluginInfo doesn't take the extra download url information as a parameter. Third party plugins ended up just trying to download from github.com/pulumi/pulumi-name and hitting 404 and then falling back to get.pulumi.com and 404'ing there as well. This also made GetPluginInfo a potentially expensive call, as it didn't just do a filesystem lookup but possibly multiple attempts to download and extract a file as well. Further because the workspace plugin logic is a public API used by the Go automation API and by downstream code such as tfbridge there wasn't any sensible reworking of it that would have allowed the passing of the extra download url data without significant breaking changes or large amounts of code duplication. Instead this PR simply moves the auto-download logic out of the workspace code and up into the engine where we still have access to the download url (and checksum data eventually) and can correctly build a full `PluginSpec` to download with. N.B This doesn't add any code to the converter to replace the fix for #10216 because the converter now uses `schema.PluginLoader` which always calls to install a plugin before loading its schema. Interestingly this also misses the plugin download url so convert probably doesn't work for third party plugins that are currently missing. But fixing that will be more involved than this fix as we'd need to change all the schema interfaces which also has massive downstream ramifications. Fixes #11535 Fixes #10882 ## Checklist <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Service, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Fraser Waters <fraser@pulumi.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
Running
pulumi convertwill fail if you don't have resource plugins already available in the plugin cacheSteps to reproduce
pulumi new azure-yamlpulumi plugin rm --allpulumi convert --language csharp --out distObserve the following error:
Expected Behavior
Expected
pulumi convertto run successfully and install the required resource plugins on demandActual Behavior
Fails with an error message instead
Versions used
Pulumi v3.36.0
Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
The text was updated successfully, but these errors were encountered: