Python Helm charts no longer deploy into EKS cluster #1565

benesch · 2021-05-06T06:27:27Z

Since #1539, when deploying a Helm v3 chart in Python, the resources in that chart are no longer deployed to provider specified on the chart. This means that e.g. attempting to deploy a chart to an EKS cluster in Python silently deploys it to whatever is in your kubeconfig instead.

Steps to reproduce

Assuming eks.cluster is an EKS cluster resource, run a program like:

k8s.helm.v3.Chart(
    "some-helm-chart",
    k8s.helm.v3.ChartOpts(
        chart="whatever",
        # ...
    ),
    opts=pulumi.ResourceOptions(provider=eks.cluster.provider),
)

The chart's resources will get deployed to whatever kubectl is configured to use, rather than the EKS cluster.

Likely cause

This is almost certainly fallout from #1539. v3.0.0 does not exhibit this bug, but v3.1.0 (which includes #1539) does.

cc @lukehoban @lblackstone

The text was updated successfully, but these errors were encountered:

lukehoban · 2021-05-06T08:18:41Z

@benesch I just tried this - and I was not able to reproduce the issue.

With this program and requirements.txt:

from pulumi import ResourceOptions
from pulumi_kubernetes import Provider
from pulumi_kubernetes.helm.v3 import Chart, LocalChartOpts

prov = Provider("p", context="docker-desktop")

Chart("foo", LocalChartOpts(path="../foo", namespace="lager"))
Chart("foo2", LocalChartOpts(path="../foo", namespace="lager2"), ResourceOptions(provider=prov))

pulumi>=3.0.0,<4.0.0
pulumi-kubernetes==3.1.0

And with my current context set to a cluster that is not reachable, but with a local docker-desktop context that is available.

An update gives me the expected:

Updating (dev)

View Live: https://app.pulumi.com/lukehoban/pychart/dev/updates/6

     Type                                     Name         Status                  Info
 +   pulumi:pulumi:Stack                      pychart-dev  **creating failed**     1 error
 +   ├─ kubernetes:helm.sh/v3:Chart           foo2         created                 
 +   │  ├─ kubernetes:core/v1:ServiceAccount  lager2/foo2  created                 
 +   │  ├─ kubernetes:core/v1:Service         lager2/foo2  created                 
 +   │  └─ kubernetes:apps/v1:Deployment      lager2/foo2  created                 
 +   ├─ pulumi:providers:kubernetes           p            created                 
 +   └─ kubernetes:helm.sh/v3:Chart           foo          created                 
 +      ├─ kubernetes:core/v1:ServiceAccount  lager/foo    **creating failed**     1 error
 +      ├─ kubernetes:core/v1:Service         lager/foo    **creating failed**     1 error
 +      └─ kubernetes:apps/v1:Deployment      lager/foo    **creating failed**     1 error

Do you have any more details on your repro for this?

lblackstone · 2021-05-06T23:18:23Z

I was able to repro with the following program:

import pulumi
import pulumi_aws as aws
import pulumi_eks as eks
import os
from pulumi import ResourceOptions
from pulumi_kubernetes.helm.v3 import Chart, ChartOpts, FetchOpts

base_name = "demo"
profile = os.environ["AWS_PROFILE"]

# Create an AWS provider instance using the named profile creds
# and current region.
uswest2 = aws.Provider("uswest2", region="us-west-2", profile=profile)

kubeconfig_opts = eks.KubeconfigOptionsArgs(profile_name=profile)
myekscluster = eks.Cluster(
    base_name,
    provider_credential_opts=kubeconfig_opts,
    opts=pulumi.ResourceOptions(provider=uswest2)
)

Chart("nginx", ChartOpts(
    chart="nginx",
    values={"service": {"type": "ClusterIP"}},
    fetch_opts=FetchOpts(
        repo="https://charts.bitnami.com/bitnami"
    )
), ResourceOptions(provider=myekscluster.provider))

pulumi>=3.0.0,<4.0.0
pulumi-aws>=4.0.0,<5.0.0
pulumi-eks>=0.30.0
pulumi-kubernetes==3.1.0

Here are the relevant parts of the state file:

Chart

{
    "urn": "urn:pulumi:dev::eks-py-test::kubernetes:helm.sh/v3:Chart::nginx",
    "custom": false,
    "type": "kubernetes:helm.sh/v3:Chart",
    "outputs": {
        "resources": {
            "apps/v1/Deployment:nginx": {
                "4dabf18193072939515e22adb298388d": "5cf8f73096256a8f31e491e813e4eb8e",
                "id": "default/nginx",
                "packageVersion": "",
                "urn": "urn:pulumi:dev::eks-py-test::kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::nginx"
            },
            "v1/ConfigMap:nginx-server-block": {
                "4dabf18193072939515e22adb298388d": "5cf8f73096256a8f31e491e813e4eb8e",
                "id": "default/nginx-server-block",
                "packageVersion": "",
                "urn": "urn:pulumi:dev::eks-py-test::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::nginx-server-block"
            },
            "v1/Service:nginx": {
                "4dabf18193072939515e22adb298388d": "5cf8f73096256a8f31e491e813e4eb8e",
                "id": "default/nginx",
                "packageVersion": "",
                "urn": "urn:pulumi:dev::eks-py-test::kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::nginx"
            }
        }
    },
    "parent": "urn:pulumi:dev::eks-py-test::pulumi:pulumi:Stack::eks-py-test-dev",
    "aliases": [
        "urn:pulumi:dev::eks-py-test::kubernetes:helm.sh/v2:Chart::nginx"
    ]
},

Deployment (Chart sub-resource)

"parent": "urn:pulumi:dev::eks-py-test::kubernetes:helm.sh/v3:Chart::nginx",
"provider": "urn:pulumi:dev::eks-py-test::pulumi:providers:kubernetes::default_3_1_0::2077f3de-aa61-4ebf-8d8c-ce6739f0b7ae",

lblackstone · 2021-05-06T23:24:17Z

Also confirmed that it works as expected with pulumi_kubernetes==3.0.0.

Here's the updated Deployment state with the 3.0.0 provider:

"parent": "urn:pulumi:dev::eks-py-test::kubernetes:helm.sh/v3:Chart::nginx",
"provider": "urn:pulumi:dev::eks-py-test::eks:index:Cluster$pulumi:providers:kubernetes::demo-provider::c99dc918-8274-4f42-b90c-3daf6ca53bc3",

benesch · 2021-05-07T04:07:17Z

Thanks for piecing together a standalone repro, @lblackstone!

lukehoban · 2021-05-07T06:02:01Z

Debugging this - I'm seeing that the Chart object looks like this when the child Deployment is being created (reading the deployment's parent):

 {'_transformations': [], '_name': 'nginx', '_providers': {<pulumi.output.Output object at 0x10e3f2280>: <pulumi.output.Output object at 0x10e3eac40>}, '_protect': False, '_aliases': [<pulumi.output.Output object at 0x10e3f26a0>], 'urn': <pulumi.output.Output object at 0x10e3f2a90>, 'id': None, 'resources': <pulumi.output.Output object at 0x10e3f2e80>}

Note that the key in the _providers map is an Output, even though the type of _providers and implementation of get_provider assume it is a str. See for example https://github.com/pulumi/pulumi/blob/master/sdk/python/lib/pulumi/resource.py#L792:L808.

It seems that something - most likely related to multi-language components? - is populating the providers map incorrectly, which is leading to the parent providers inheritance not working. This results in the child believing there is no provider to inherit from the parent for kubernetes - even though there is.

benesch · 2021-05-07T06:07:08Z

A bit of a shot in the dark, but I filed pulumi/pulumi#6693 a few weeks back about other RemoteComponent-related weirdness. Using eks.Cluster from Python definitely seems to have a bunch of weird edge cases.

lukehoban · 2021-05-07T06:23:16Z

The issue appears to be that myekscluster.provider is an Output, but the implementation of the Resource base class does not correctly handle Output[Provider] arguments. In particular this line causes an propeperty on the Output[Provider] to be accessed, which is itself an Output[str], which is then used as the key for the dict entry.

https://github.com/pulumi/pulumi/blob/master/sdk/python/lib/pulumi/runtime/resource.py#L618

Either Resource needs to handle Output[Provider] inputs, or multi-lang components need to ensure that true Provider instances are returned as outputs instead of Output[Provider].

lblackstone · 2021-05-07T17:57:22Z

~~I think this regression was caused by the changes to fix pulumi/pulumi-eks#555~~

~~I'll go through all of the related changes and make sure Outputs are being handled properly.~~

Edit: This wasn't the problem.

lblackstone · 2021-05-07T20:02:48Z

After some more digging, I realized that my repro program contains a type error. In the Python, .NET, and Go SDKs, the provider attribute is typed as an Output rather than a prompt value as in the NodeJS SDK.

Everything works as expected when I unwrap the provider Output with an apply:

import pulumi
import pulumi_aws as aws
import pulumi_eks as eks
import os
from pulumi import ResourceOptions
from pulumi_kubernetes.helm.v3 import Chart, ChartOpts, FetchOpts

base_name = "demo"
profile = os.environ["AWS_PROFILE"]

# Create an AWS provider instance using the named profile creds
# and current region.
uswest2 = aws.Provider("uswest2", region="us-west-2", profile=profile)

kubeconfig_opts = eks.KubeconfigOptionsArgs(profile_name=profile)
myekscluster = eks.Cluster(
    base_name,
    provider_credential_opts=kubeconfig_opts,
    opts=pulumi.ResourceOptions(provider=uswest2)
)

myekscluster.provider.apply(lambda p: chart(p))


def chart(provider):
    Chart("nginx", ChartOpts(
        chart="nginx",
        values={"service": {"type": "ClusterIP"}},
        fetch_opts=FetchOpts(
            repo="https://charts.bitnami.com/bitnami"
        )
    ), ResourceOptions(provider=provider))

lblackstone · 2021-05-07T20:21:23Z

I narrowed down the change in behavior between v3.0.0 and v3.1.0 to this: https://github.com/pulumi/pulumi-kubernetes/pull/1539/files?file-filters%5B%5D=.py#diff-f2b60d028397820398a9fa1ebca34ef73fe594525259f511f39bfce01ef24e9fL177-L178

With this change reverted, the Output<Provider> implicitly resolves even though the types don't match.

lblackstone · 2021-05-07T21:26:10Z

@benesch We're currently investigating a few options to fix this. As a workaround in the meantime, you can create a new Provider instance using the EKS cluster's kubeconfig, and it will work as you'd expect:

import pulumi
import pulumi_aws as aws
import pulumi_eks as eks
import os
from pulumi import ResourceOptions
from pulumi_kubernetes.helm.v3 import Chart, ChartOpts, FetchOpts
from pulumi_kubernetes import Provider, ProviderArgs

base_name = "demo"
profile = os.environ["AWS_PROFILE"]

# Create an AWS provider instance using the named profile creds
# and current region.
uswest2 = aws.Provider("uswest2", region="us-west-2", profile=profile)

kubeconfig_opts = eks.KubeconfigOptionsArgs(profile_name=profile)
myekscluster = eks.Cluster(
    base_name,
    provider_credential_opts=kubeconfig_opts,
    opts=pulumi.ResourceOptions(provider=uswest2)
)

provider = Provider("k8s", ProviderArgs(
    kubeconfig=myekscluster.kubeconfig
))
Chart("nginx", ChartOpts(
    chart="nginx",
    values={"service": {"type": "ClusterIP"}},
    fetch_opts=FetchOpts(
        repo="https://charts.bitnami.com/bitnami"
    )
), ResourceOptions(provider=provider))

benesch · 2021-05-07T23:34:39Z

Thanks, @lblackstone. For now we're happy to stick on v3.0.0, but I'll try out your workaround if we need to upgrade.

lblackstone · 2021-05-10T20:48:08Z

pulumi/pulumi#7012 tracks making the cluster.provider work for SDKs other than NodeJS.

leezen · 2021-06-08T17:08:25Z

Closing as this is now tracked in pulumi/pulumi#7012.

benesch · 2021-07-07T03:00:06Z

I don't mean to sound ungrateful, but I'd like to advocate for reopening this issue until pulumi/pulumi#7012 is fixed, or there are some guardrails put in place here! While I understand that the aforementioned issue is tracking the root cause, it's not discoverable for folks who are just looking to understand why pulumi_eks and pulumi_kubernetes are not playing well together.

In particular, pulumi/pulumi#3383 means the bug presents as silently deploying into whatever cluster is currently active in the user's kubecfg. This is the kind of bug that can result in taking down prod. (We were lucky, and it only took down our staging environment, but that's only because I happened to have staging as my active cluster, not prod.)

Since pulumi/pulumi#7012 looks like it's not a quick fix, it'd be great to get some some assertions in the SDK to at least prevent disaster, or at the very least some warnings in the docs.

@lblackstone, thanks for the workaround, but I don't think it'll work for us, since swapping the provider out like that results in Pulumi attempting to replace all the resources in the cluster. Tried to work around this with provider aliases but looks like those aren't wired up yet (pulumi/pulumi#3979).

benesch · 2021-07-07T04:07:40Z

Here's my workaround for now:

class LazyResource:
    def __init__(self, resource):
        self.resource = pulumi.Output.from_input(resource)

    @property
    def urn(self):
        return self.resource.apply(lambda r: r.urn)

    @property
    def id(self):
        return self.resource.apply(lambda r: r.id)

    @property
    def __class__(self):
        # Bypass https://github.com/pulumi/pulumi/blob/b7d403204/sdk/python/lib/pulumi/resource.py#L460-L462.
        return pulumi.Resource


class LazyProvider(LazyResource):
    def __init__(self, package, resource):
        super().__init__(resource)
        self.package = package

    @property
    def __class__(self):
        return pulumi.ProviderResource

cluster = eks.Cluster(...)
_provider = cluster.provider
cluster.__dict__["provider"] = LazyProvider("kubernetes", _provider)

eliskovets · 2021-11-11T21:34:46Z

Hi guys,
With my limited undestanding of pulumi, even if the issue is related to pulumi/pulumi#7012 , the changes here seems to be an issue to me and it hasn't been reverted, still exist in the 3.9.0 version.

@benesch Could you please tell me if you still use 3.0.0 to mitigate this issue or found another workaround? How do you pass the kubernetes provider to the rest of Chart resources?

benesch · 2021-11-21T02:12:36Z

@eliskovets we've been using the workaround provided by @lblackstone in #1565 (comment) to good effect for a while now.

benesch added the kind/bug Some behavior is incorrect or out of spec label May 6, 2021

lukehoban added the priority/P1 label May 6, 2021

lukehoban self-assigned this May 6, 2021

leezen added this to the 0.56 milestone May 7, 2021

lblackstone assigned lblackstone and unassigned lukehoban May 7, 2021

lblackstone added kind/bug Some behavior is incorrect or out of spec emergent An issue that was added to the current milestone after planning and removed kind/bug Some behavior is incorrect or out of spec priority/P1 labels May 7, 2021

leezen modified the milestones: 0.56, 0.57 May 17, 2021

leezen removed this from the 0.57 milestone Jun 8, 2021

leezen added the resolution/duplicate This issue is a duplicate of another issue label Jun 8, 2021

leezen closed this as completed Jun 8, 2021

benesch mentioned this issue Aug 5, 2021

Remove provider from cluster result pulumi/pulumi-eks#602

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python Helm charts no longer deploy into EKS cluster #1565

Python Helm charts no longer deploy into EKS cluster #1565

benesch commented May 6, 2021 •

edited

Loading

lukehoban commented May 6, 2021

lblackstone commented May 6, 2021

lblackstone commented May 6, 2021

benesch commented May 7, 2021

lukehoban commented May 7, 2021

benesch commented May 7, 2021

lukehoban commented May 7, 2021

lblackstone commented May 7, 2021 •

edited

Loading

lblackstone commented May 7, 2021

lblackstone commented May 7, 2021

lblackstone commented May 7, 2021

benesch commented May 7, 2021

lblackstone commented May 10, 2021

leezen commented Jun 8, 2021 •

edited

Loading

benesch commented Jul 7, 2021

benesch commented Jul 7, 2021

eliskovets commented Nov 11, 2021

benesch commented Nov 21, 2021

Python Helm charts no longer deploy into EKS cluster #1565

Python Helm charts no longer deploy into EKS cluster #1565

Comments

benesch commented May 6, 2021 • edited Loading

Steps to reproduce

Likely cause

lukehoban commented May 6, 2021

lblackstone commented May 6, 2021

lblackstone commented May 6, 2021

benesch commented May 7, 2021

lukehoban commented May 7, 2021

benesch commented May 7, 2021

lukehoban commented May 7, 2021

lblackstone commented May 7, 2021 • edited Loading

lblackstone commented May 7, 2021

lblackstone commented May 7, 2021

lblackstone commented May 7, 2021

benesch commented May 7, 2021

lblackstone commented May 10, 2021

leezen commented Jun 8, 2021 • edited Loading

benesch commented Jul 7, 2021

benesch commented Jul 7, 2021

eliskovets commented Nov 11, 2021

benesch commented Nov 21, 2021

benesch commented May 6, 2021 •

edited

Loading

lblackstone commented May 7, 2021 •

edited

Loading

leezen commented Jun 8, 2021 •

edited

Loading