Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Passing a provider for an MLC's children via pulumi.Provider results in error #12430

Closed
justinvp opened this issue Mar 14, 2023 · 2 comments · Fixed by #12433
Closed

Passing a provider for an MLC's children via pulumi.Provider results in error #12430

justinvp opened this issue Mar 14, 2023 · 2 comments · Fixed by #12433
Assignees
@abhinav
Labels
area/sdks Pulumi language SDKs customer/feedback Feedback from customers customer/lighthouse Lighthouse customer bugs impact/regression Something that used to work, but is now broken kind/bug Some behavior is incorrect or out of spec language/go p1 Bugs severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Milestone
0.86

Comments

@justinvp
Copy link
Member

justinvp commented Mar 14, 2023 •
edited

The following Go program previews as expected using v3.56.0 of the Go SDK:

main.go:

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v5/go/aws"
	"github.com/pulumi/pulumi-awsx/sdk/go/awsx/ec2"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		provider, err := aws.NewProvider(ctx, "myprovider", nil)
		if err != nil {
			return err
		}

		_, err = ec2.NewVpc(ctx, "myvpc", nil, pulumi.Provider(provider))
		if err != nil {
			return err
		}

		return nil
	})
}

go.mod:

module myproject

go 1.18

require (
	github.com/pulumi/pulumi-aws/sdk/v5 v5.31.0
	github.com/pulumi/pulumi-awsx/sdk v1.0.2
	github.com/pulumi/pulumi/sdk/v3 v3.56.0
)

Output:

$ pulumi preview
Previewing update (dev)

     Type                                          Name             Plan
 +   pulumi:pulumi:Stack                           goproviderr-dev  create
 +   ├─ pulumi:providers:aws                       myprovider       create
 +   └─ awsx:ec2:Vpc                               myvpc            create
 +      └─ aws:ec2:Vpc                             myvpc            create
 +         ├─ aws:ec2:InternetGateway              myvpc            create
 +         ├─ aws:ec2:Subnet                       myvpc-public-2   create
 +         │  ├─ aws:ec2:RouteTable                myvpc-public-2   create
 +         │  │  ├─ aws:ec2:Route                  myvpc-public-2   create
 +         │  │  └─ aws:ec2:RouteTableAssociation  myvpc-public-2   create
 +         │  ├─ aws:ec2:Eip                       myvpc-2          create
 +         │  └─ aws:ec2:NatGateway                myvpc-2          create
 +         ├─ aws:ec2:Subnet                       myvpc-public-1   create
 +         │  ├─ aws:ec2:RouteTable                myvpc-public-1   create
 +         │  │  ├─ aws:ec2:RouteTableAssociation  myvpc-public-1   create
 +         │  │  └─ aws:ec2:Route                  myvpc-public-1   create
 +         │  ├─ aws:ec2:Eip                       myvpc-1          create
 +         │  └─ aws:ec2:NatGateway                myvpc-1          create
 +         ├─ aws:ec2:Subnet                       myvpc-private-1  create
 +         │  └─ aws:ec2:RouteTable                myvpc-private-1  create
 +         │     ├─ aws:ec2:RouteTableAssociation  myvpc-private-1  create
 +         │     └─ aws:ec2:Route                  myvpc-private-1  create
 +         ├─ aws:ec2:Subnet                       myvpc-public-3   create
 +         │  ├─ aws:ec2:RouteTable                myvpc-public-3   create
 +         │  │  ├─ aws:ec2:RouteTableAssociation  myvpc-public-3   create
 +         │  │  └─ aws:ec2:Route                  myvpc-public-3   create
 +         │  ├─ aws:ec2:Eip                       myvpc-3          create
 +         │  └─ aws:ec2:NatGateway                myvpc-3          create
 +         ├─ aws:ec2:Subnet                       myvpc-private-3  create
 +         │  └─ aws:ec2:RouteTable                myvpc-private-3  create
 +         │     ├─ aws:ec2:RouteTableAssociation  myvpc-private-3  create
 +         │     └─ aws:ec2:Route                  myvpc-private-3  create
 +         └─ aws:ec2:Subnet                       myvpc-private-2  create
 +            └─ aws:ec2:RouteTable                myvpc-private-2  create
 +               ├─ aws:ec2:RouteTableAssociation  myvpc-private-2  create
 +               └─ aws:ec2:Route                  myvpc-private-2  create


Resources:
    + 35 to create

However, upgrading the pulumi SDK to v3.57.1 (github.com/pulumi/pulumi/sdk/v3 v3.57.1) results in the following error during preview:

$ pulumi preview
Previewing update (dev)

     Type                     Name             Plan       Info
 +   pulumi:pulumi:Stack      goproviderr-dev  create     1 error
 +   └─ pulumi:providers:aws  myprovider       create


Diagnostics:
  pulumi:pulumi:Stack (goproviderr-dev):
    error: an unhandled error occurred: program failed:
    waiting for RPCs: rpc error: code = Unknown desc = plugins that can construct components must support secrets

The error message is deceiving... it's coming from:

pulumi/sdk/go/common/resource/plugin/provider_plugin.go

Line 1250 in 477e0c9

return ConstructResult{}, fmt.Errorf("plugins that can construct components must support secrets")

We've seen this before in #11316. When passing an AWS provider to an MLC, the engine is trying to use that provider to construct the MLC. But the AWS provider doesn't have an implementation of Construct for the MLC, and it's failing earlier when trying to call Construct on the AWS provider because the AWS provider currently says it doesn't support secrets.

Workaround

A workaround is to pass the provider via pulumi.Providers (plural) rather than pulumi.Provider (singular). The plural variant is intended for passing providers for a component's children.
@justinvp justinvp added kind/bug Some behavior is incorrect or out of spec area/sdks Pulumi language SDKs p1 Bugs severe enough to be the next item assigned to an engineer language/go impact/regression Something that used to work, but is now broken labels Mar 14, 2023
@justinvp justinvp added this to the 0.86 milestone Mar 14, 2023
@justinvp
Copy link
Member Author

This is related to #11520

I think we want the following behavior: If a provider is passed to an MLC with pulumi.Provider and the pkg of the provider:

  • matches the pkg of the resource, then that provider should be used for the MLC's Construct call.
  • does not match the pkg of the resource, then that provider should be used as if it were passed to pulumi.Providers for use by the component's children

@justinvp justinvp added customer/lighthouse Lighthouse customer bugs customer/feedback Feedback from customers labels Mar 14, 2023
@abhinav abhinav mentioned this issue Mar 14, 2023
Merged
abhinav added a commit that referenced this issue Mar 14, 2023
@abhinav
sdk/go: Don't use Provider for type if package doesn't match
9a5e733 
In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430
@abhinav
Copy link
Contributor

abhinav commented Mar 14, 2023

Fixing in #12433.
Proposed fix verified locally against the repro above.

abhinav added a commit that referenced this issue Mar 14, 2023
@abhinav
sdk/go: Don't use Provider for type if package doesn't match
273787c 
In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430
bors bot added a commit that referenced this issue Mar 15, 2023
@bors @Frassle
@abhinav @justinvp
Merge #12426 #12433 #12435
985c81c 
12426: filestate/internal: Use stack reference, not name r=abhinav a=abhinav

filestate backend currently operates exclusively with stack names.
All its internal pass around just the stack name, and nothing else.
This makes it a bit difficult to add project support to the backend.

This is a refactor in advance of adding project support,
changing the internals of filestate to pass a stack reference around.
It inspects the reference directly for all its operations.

Note: This contains no behavioral changes.
Name and FullyQualifiedName currently both return just the stack name.
In a future change, once project name is incorporated into the object,
FullyQualifiedName will be able to return `organization/$project/$name`.

Change extracted from #12134


12433: sdk/go: Don't use Provider for type if package doesn't match r=abhinav a=abhinav

In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430


12435: Pin CI to Node 19.7.x to avoid hitting assert in 19.8.0 r=abhinav a=justinvp

Node 19.8.0 was released ~5 hours ago and we're hitting an internal assert. Pin to 19.7.x to unblock CI until we understand what's going on with 19.8.0.

Part of #12434

Co-authored-by: Fraser Waters <fraser@pulumi.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
bors bot added a commit that referenced this issue Mar 15, 2023
@bors @abhinav @justinvp
Merge #12433 #12435
629f11c 
12433: sdk/go: Don't use Provider for type if package doesn't match r=abhinav a=abhinav

In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430


12435: Pin CI to Node 19.7.x to avoid hitting assert in 19.8.0 r=abhinav a=justinvp

Node 19.8.0 was released ~5 hours ago and we're hitting an internal assert. Pin to 19.7.x to unblock CI until we understand what's going on with 19.8.0.

Part of #12434

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
bors bot added a commit that referenced this issue Mar 15, 2023
@bors @abhinav
Merge #12433
3defb3a 
12433: sdk/go: Don't use Provider for type if package doesn't match r=abhinav a=abhinav

In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430


Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
bors bot added a commit that referenced this issue Mar 15, 2023
@bors @abhinav
Merge #12433
73bc578 
12433: sdk/go: Don't use Provider for type if package doesn't match r=Frassle a=abhinav

In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430


Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
bors bot added a commit that referenced this issue Mar 15, 2023
@bors @abhinav
Merge #12433
2806cac 
12433: sdk/go: Don't use Provider for type if package doesn't match r=Frassle a=abhinav

In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430


Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
bors bot added a commit that referenced this issue Mar 15, 2023
@bors @abhinav
Merge #12433
e72faa3 
12433: sdk/go: Don't use Provider for type if package doesn't match r=Frassle a=abhinav

In #12296, we started correctly interpreting the Provider argument
per other SDKs. but this introduced a regression:
the Provider field is now set for resources with mismatched types.

This results in a scenario where a provider foo for package X
is passed to a resource bar with package Y,
with the intent of plumbing it to bar's descendants,
but bar attempts to incorrectly use the provider directly.

    foo := NewXProvider()
    bar := NewYThing("bar", Provider(foo))
    // ...
    baz := NewXThing("baz", Parent(bar)) // should use foo

This worked previously, but with #12296, this fails
because NewYThing attempts to use Provider foo directly.

To fix this, we need to prevent NewYThing from using a provider
that does not match the package that it belongs to.
We had to make a similar change to the Python SDK in #12292.

https://github.com/pulumi/pulumi/blob/477e0c97660bdbeb28a142057899792d52e63a0b/sdk/python/lib/pulumi/resource.py#L925-L927

The regression test is specific to remote component resources
per #12430, but the issue would be caused even for normal component
resources before this.

Resolves #12430


Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
@bors bors bot closed this as completed in ef6036a Mar 15, 2023
@pulumi-bot pulumi-bot added the resolution/fixed This issue was fixed label Mar 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@abhinav abhinav

Labels
area/sdks Pulumi language SDKs customer/feedback Feedback from customers customer/lighthouse Lighthouse customer bugs impact/regression Something that used to work, but is now broken kind/bug Some behavior is incorrect or out of spec language/go p1 Bugs severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Projects
None yet
Milestone
0.86
Development

Successfully merging a pull request may close this issue.

sdk/go: Don't use Provider for type if package doesn't match pulumi/pulumi
3 participants
@abhinav @justinvp @pulumi-bot