Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CustomResource panic when changing name field in Tekton Pipeline #1970

Closed
jsravn opened this issue Apr 19, 2022 · 5 comments · Fixed by #2987
Closed

CustomResource panic when changing name field in Tekton Pipeline #1970

jsravn opened this issue Apr 19, 2022 · 5 comments · Fixed by #2987
Assignees
Labels
impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed

Comments

@jsravn
Copy link

jsravn commented Apr 19, 2022

What happened?

I'm trying to modify a Tekton Pipeline via kubernetes.apiextensions.CustomResource. If I try changing a step name, I get a panic.

pulumi up
Previewing update (extended-test):
     Type                                       Name                         Plan     Info
     pulumi:pulumi:Stack                        extended-test-extended-test           44 messages     └─ kubernetes:tekton.dev/v1beta1:Pipeline  test-pipeline                         1 error
 
Diagnostics:
  kubernetes:tekton.dev/v1beta1:Pipeline (test-pipeline):
    error: transport is closing
 
  pulumi:pulumi:Stack (extended-test-extended-test):
    panic: fatal: An assertion has failed: path: [spec tasks 0 taskSpec spec]  |  v: <nil>  | old: <nil>  |  oldInput: <nil>
    goroutine 308 [running]:
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.27.0/go/common/util/contract/failfast.go:23
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0x80, {0x2204c9c, 0x8}, {0xc003129cc8, 0x203000, 0x203000})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.27.0/go/common/util/contract/assert.go:33 +0xf4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00312b398, {0xc003422c80, 0x1db06a0, 0x8}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2887 +0x188
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc003219540, {0xc003219540, 0x4, 0x4}, 0x2, 0x46acb9, 0x1d55fe0, 0x1f79ea0, 0x1)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00312b398, {0xc003219540, 0x1e4e5a0, 0x4}, {0x1e8cb00, 0xc002ce5800}, {0x1e8cb00, 0xc002bea930}, {0x1e8cb00, 0xc000aad620}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2910 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc003219540, {0xc003219540, 0x3, 0x4}, 0x203000, 0x203000, 0xc003219540, 0x1d5c9e0, 0x1)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00312b398, {0xc003219540, 0x8, 0x4}, {0x1e8cb00, 0xc002ce57d0}, {0x1e8cb00, 0xc002bea900}, {0x1e8cb00, 0xc000aad5f0}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2910 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchArrayToDiff(0x0, {0xc00321cee0, 0x2, 0x2}, {0xc00149ce30, 0x1, 0xc00312a920}, {0xc0024123c0, 0x1, 0x1}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3020 +0x23e
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00312b398, {0xc00321cee0, 0x0, 0x2}, {0x1d5c9e0, 0xc00316f188}, {0x1d5c9e0, 0xc002416168}, {0x1d5c9e0, 0xc0014724e0}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2917 +0x405
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc00149cf00, {0xc00149cf00, 0x1, 0x1}, 0xc00312ae80, 0xc00149cec0, 0x1e4e5a0, 0xc00316f1b8, 0x0)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00312b398, {0xc00149cf00, 0x1e8cb00, 0x1}, {0x1e8cb00, 0xc003d90900}, {0x1e8cb00, 0xc000aad7a0}, {0x1e8cb00, 0xc000aad5c0}, ...)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2910 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0x7, {0x0, 0x0, 0x0}, 0x20, 0x1db06a0, 0x70000c002d09401, 0xffffffffffffffff, 0x0)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.convertPatchToDiff(0xc003d908a0, 0xc000aad6e0, 0x1e8cb00, 0xc002cd1bf0, {0xc00321cec0, 0x2, 0x2})
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2815 +0x105
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*kubeProvider).Diff(0xc000162180, {0x2546728, 0xc00348b380}, 0xc0030f4310)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:1541 +0x1cf9
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x2546728, 0xc00348b380}, {0x2095b60, 0xc0030f4310})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.27.0/proto/go/provider.pb.go:2611 +0x78
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x2546728, 0xc000477680}, {0x2095b60, 0xc0030f4310}, 0xc0031313e0, 0xc00113d320)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x425
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x2130340, 0xc000162180}, {0x2546728, 0xc000477680}, 0xc00147f500, 0xc000814e40)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.27.0/proto/go/provider.pb.go:2613 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003c9500, {0x2586c48, 0xc0006031e0}, 0xc002ae7560, 0xc0008f8510, 0x3962a48, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf
    google.golang.org/grpc.(*Server).handleStream(0xc0003c9500, {0x2586c48, 0xc0006031e0}, 0xc002ae7560, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2a
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x294

Steps to reproduce

     const testPipeline = new kubernetes.apiextensions.CustomResource(
      "test-pipeline",
      {
        apiVersion: "tekton.dev/v1beta1",
        kind: "Pipeline",
        metadata: {},
        spec: {
          description: "test pipeline",
          tasks: [
            {
              name: "do-something-again",
              taskSpec: {
                steps: [
                  {
                    name: "do-something",
                    image: "busybox",
                    script: "/bin/whoami",
                  },
                ],
              },
            },
          ],
        },
      },
      {
        parent: this,
      }
    );
  1. Have tekton CRDs installed
  2. Run pulumi up to apply
  3. Change step name from "do-something" to "do-something-else"
  4. Run pulumi up to apply

Expected Behavior

It should successfully change the step name.

Actual Behavior

It panics.

Versions used

pulumi about
CLI          
Version      3.28.0
Go Version   go1.17.8
Go Compiler  gc

Plugins
NAME        VERSION
gcp         6.17.0
gcp         6.17.0
kubernetes  3.18.1
kubernetes  3.18.1
nodejs      unknown
random      4.4.2
random      4.4.2

Host     
OS       nixos
Version  21.11 (Porcupine)
Arch     x86_64

This project is written in nodejs (/nix/store/46g0dmf6rcpikbzs22y7w4amyg0ciksi-nodejs-16.14.2/bin/node v16.14.2)

Additional context

Using plain kubectl edit I can change the step name without issue.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@jsravn jsravn added the kind/bug Some behavior is incorrect or out of spec label Apr 19, 2022
@jsravn jsravn changed the title CustomResource assertion failure when changing name field in Tekton Pipeline CustomResource panic when changing name field in Tekton Pipeline Apr 19, 2022
@guineveresaenger
Copy link
Contributor

Thanks for filing this issue! Well take a look as soon as we can.

@jsravn
Copy link
Author

jsravn commented Jun 6, 2022

Seems to happen when I change other fields as well, it's not clear what exactly triggers this bug. For example, I made this change to a Pipeline today:

modified   genesis/src/index.ts
@@ -693,6 +693,10 @@ const cdPipeline = new kubernetes.apiextensions.CustomResource(
               name: "builder-image",
               value: "$(params.builder-image):$(params.git-revision)",
             },
+            {
+              name: "stack-name",
+              value: "$(params.stack-name)",
+            },
           ],
           runAfter: ["npm-install"],
           workspaces: [
@@ -705,7 +709,9 @@ const cdPipeline = new kubernetes.apiextensions.CustomResource(
             params: [
               {
                 name: "builder-image",
-                type: "string",
+              },
+              {
+                name: "stack-name",
               },
             ],
             workspaces: [

In other words, I simply added a param item. But now it always fails with a panic when trying to do a pulumi up:

Diagnostics:
        pulumi:pulumi:Stack ([genesis-genesis.ci](http://genesis-genesis.ci/)):
          panic: fatal: An assertion has failed: path: [spec tasks 4 taskSpec spec]  |  v: <nil>  | old: <nil>  |  oldInput: <nil>
          goroutine 348 [running]:
          [github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)](http://github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...))
          	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.30.0/go/common/util/contract/failfast.go:23
          [github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf](http://github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf)(0x80, {0x2207a62, 0x8}, {0xc003149cc8, 0x203000, 0x203000})
          	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.30.0/go/common/util/contract/assert.go:33 +0xf4
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchValueToDiff(0xc00314b398, {0xc00358c580, 0xc003149f08, 0x8}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2898 +0x188
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchMapToDiff(0xc0032f3200, {0xc0032f3200, 0x4, 0x4}, 0x1, 0x1, 0xc00322c080, 0x1, 0x1)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3000 +0x3f4
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchValueToDiff(0xc00314b398, {0xc0032f3200, 0x1e507a0, 0x4}, {0x1e8ed60, 0xc0032f7f80}, {0x1e8ed60, 0xc003201c20}, {0x1e8ed60, 0xc003365080}, ...)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2921 +0x2b0
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchMapToDiff(0xc0032f3200, {0xc0032f3200, 0x3, 0x4}, 0x203000, 0x203000, 0xc0032f3200, 0x1d5eac0, 0x1)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3000 +0x3f4
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchValueToDiff(0xc00314b398, {0xc0032f3200, 0x8, 0x4}, {0x1e8ed60, 0xc0032f7ef0}, {0x1e8ed60, 0xc003201bc0}, {0x1e8ed60, 0xc003364ff0}, ...)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2921 +0x2b0
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchArrayToDiff(0x0, {0xc0032eed20, 0x2, 0x2}, {0xc0032f1680, 0x5, 0x110}, {0xc0031e7200, 0x5, 0x8}, ...)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3031 +0x23e
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchValueToDiff(0xc00314b398, {0xc0032eed20, 0xfbdcc818e97db27, 0x2}, {0x1d5eac0, 0xc003584408}, {0x1d5eac0, 0xc0031e3830}, {0x1d5eac0, 0xc003362ca8}, ...)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2928 +0x405
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchMapToDiff(0xc0032ff8a0, {0xc0032ff8a0, 0x1, 0x1}, 0xc00314ae80, 0xc0032ff830, 0x1e507a0, 0xc003584438, 0x0)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3000 +0x3f4
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchValueToDiff(0xc00314b398, {0xc0032ff8a0, 0x1e8ed60, 0x1}, {0x1e8ed60, 0xc0033919b0}, {0x1e8ed60, 0xc0033653b0}, {0x1e8ed60, 0xc003364840}, ...)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2921 +0x2b0
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter)).addPatchMapToDiff(0x7, {0x0, 0x0, 0x0}, 0x20, 0x1db2780, 0x20000c0033a6301, 0xffffffffffffffff, 0x0)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3000 +0x3f4
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.convertPatchToDiff](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.convertPatchToDiff)(0xc003391980, 0xc0033652f0, 0x1e8ed60, 0xc0032f7a10, {0xc0032eece0, 0x2, 0x2})
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2826 +0x105
          [github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*kubeProvider](http://github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*kubeProvider)).Diff(0xc000137380, {0x2549a08, 0xc003306ff0}, 0xc00278fc00)
          	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:1552 +0x1cf9
          [github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1](http://github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1)({0x2549a08, 0xc003306ff0}, {0x2098820, 0xc00278fc00})
          	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.30.0/proto/go/provider.pb.go:2596 +0x78
          [github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1](http://github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1)({0x2549a08, 0xc00285cea0}, {0x2098820, 0xc00278fc00}, 0xc00281d280, 0xc0029f21f8)
          	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x425
          [github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler](http://github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler)({0x21356c0, 0xc000137380}, {0x2549a08, 0xc00285cea0}, 0xc00283b560, 0xc00054bba0)
          	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.30.0/proto/go/provider.pb.go:2598 +0x138
          [google.golang.org/grpc.(*Server](http://google.golang.org/grpc.(*Server)).processUnaryRPC(0xc00015cc40, {0x2589f68, 0xc0006024e0}, 0xc0032ab200, 0xc0001444e0, 0x3966c48, 0x0)
          	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf
          [google.golang.org/grpc.(*Server](http://google.golang.org/grpc.(*Server)).handleStream(0xc00015cc40, {0x2589f68, 0xc0006024e0}, 0xc0032ab200, 0x0)
          	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2a
          [google.golang.org/grpc.(*Server](http://google.golang.org/grpc.(*Server)).serveStreams.func1.2()
          	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98
          created by [google.golang.org/grpc.(*Server](http://google.golang.org/grpc.(*Server)).serveStreams.func1
          	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x294
       
        kubernetes:tekton.dev/v1beta1:Pipeline (cd):
          error: error reading from server: EOF

@jsravn
Copy link
Author

jsravn commented Jun 6, 2022

Okay, I think I figured out the cause.

After applying the Pipeline, the resource in Kubernetes ends up with a null spec field inside of TaskSpec:

    taskSpec:
      metadata: {}
      params:
      - name: builder-image
        type: string
      spec: null
      steps:
      - image: $(params.builder-image)
<snip>

Unsure where this is coming from - it's an optional field and should be omitted if empty (https://github.com/tektoncd/pipeline/blob/7de70f1c095a12c10e81932ab3b0e2268dd80ec5/pkg/apis/pipeline/v1beta1/pipeline_types.go#L149). Could be a Tekton bug that is causing this to get set to null somewhere rather than being omitted.

Anyways, it seems the kubernetes differ can't handle this properly. It encounters "spec: null" and ends up passing a nil object to the diff logic for both old and new value, and the assertion blows up. Seems to also only happen when modifying a Task with a taskSpec field present.

A workaround is to specify an empty spec: {} for the taskSpec. But the differ should also ignore such nil objects I think.

@eliskovets
Copy link

eliskovets commented Jun 8, 2022

I'm facing same issue with podTemplate in elasticsearch CR object:

podTemplate:
      metadata:
        creationTimestamp: nul

Error:

    panic: fatal: An assertion has failed: path: [spec nodeSets 0 podTemplate metadata creationTimestamp]  |  v: <nil>  | old: <nil>  |  oldInput: <nil>
    goroutine 2976 [running]:
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.33.2/go/common/util/contract/failfast.go:23
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0x80, {0x221d689, 0x8}, {0xc004ecd768, 0x1e99f80, 0xc002bd1e30})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.33.2/go/common/util/contract/assert.go:33 +0xf4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc004ecf398, {0xc00608a980, 0x100000001d6ae20, 0x8}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2887 +0x188
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc00608a980, {0xc00608a980, 0x5, 0x8}, 0x203001, 0x203001, 0xc00608a980, 0x3, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2998 +0x65a
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc004ecf398, {0xc00608a980, 0xc004ecdf08, 0x8}, {0x1ea25c0, 0xc0060910e0}, {0x1ea25c0, 0xc00307fce0}, {0x1ea25c0, 0xc002964690}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2910 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc0060a2200, {0xc0060a2200, 0x4, 0x4}, 0x2, 0xffffffffffffffff, 0x1d6ae20, 0x1f902c0, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc004ecf398, {0xc0060a2200, 0xcb40, 0x4}, {0x1ea25c0, 0xc0060910b0}, {0x1ea25c0, 0xc00307fcb0}, {0x1ea25c0, 0xc002964660}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2910 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc0060a2200, {0xc0060a2200, 0x3, 0x4}, 0x203001, 0x203001, 0xc0060a2200, 0xaaaaaaaaaaaaaa, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc004ecf398, {0xc0060a2200, 0x8, 0x4}, {0x1ea25c0, 0xc006090f30}, {0x1ea25c0, 0xc00307fb30}, {0x1ea25c0, 0xc0029644e0}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2910 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchArrayToDiff(0x0, {0xc00608e980, 0x2, 0x2}, {0xc00608a900, 0x6, 0xc004ece920}, {0xc003165980, 0x6, 0x8}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3020 +0x23e
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc004ecf398, {0xc00608e980, 0x45aca8e2535d7604, 0x2}, {0x1d718e0, 0xc0060872d8}, {0x1d718e0, 0xc003059ef0}, {0x1d718e0, 0xc002972c78}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2917 +0x405
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc006093470, {0xc006093470, 0x1, 0x1}, 0xc004ecee80, 0x40d0b4, 0x0, 0x7f1b1647cf40, 0x0)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc004ecf398, {0xc006093470, 0x1dc57a0, 0x1}, {0x1ea25c0, 0xc006090f00}, {0x1ea25c0, 0xc002976d50}, {0x1ea25c0, 0xc0029643c0}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2910 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0x2, {0x0, 0x0, 0x0}, 0x20, 0x1dc57a0, 0x30000c0060a8101, 0xffffffffffffffff, 0x0)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2989 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.convertPatchToDiff(0xc002a04690, 0xc002976c90, 0x1ea25c0, 0xc006090390, {0xc00608e960, 0x2, 0x2})
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2815 +0x105
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*kubeProvider).Diff(0xc000739e00, {0x2568da8, 0xc002782de0}, 0xc00315b3b0)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:1535 +0x1cf9
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x2568da8, 0xc002782de0}, {0x20ad2c0, 0xc00315b3b0})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.33.2/proto/go/provider.pb.go:2604 +0x78
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x2568da8, 0xc0026df9e0}, {0x20ad2c0, 0xc00315b3b0}, 0xc0026fff00, 0xc0026ee360)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x406
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x214a960, 0xc000739e00}, {0x2568da8, 0xc0026df9e0}, 0xc0027129c0, 0xc0002352c0)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.33.2/proto/go/provider.pb.go:2606 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006e3340, {0x25a9388, 0xc0004d64e0}, 0xc0026c46c0, 0xc0003217a0, 0x398dd48, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf
    google.golang.org/grpc.(*Server).handleStream(0xc0006e3340, {0x25a9388, 0xc0004d64e0}, 0xc0026c46c0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2a
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x294

Config:

CLI
Version      3.33.2
Go Version   go1.17.10
Go Compiler  gc

Plugins
NAME        VERSION
aws         4.38.1
eks         0.35.0
kubernetes  3.19.3
python      unknown

Host
OS       debian
Version  11.3
Arch     x86_64

NAME        VERSION
pip         21.3.1
pulumi-eks  0.35.0
setuptools  52.0.0

@EvanBoyle EvanBoyle added the impact/panic This bug represents a panic or unexpected crash label Jul 3, 2023
@awopenphone
Copy link

awopenphone commented Dec 1, 2023

I have a error reading from server: EOF I can contribute to this that is bugging me now. This resource worked fine a day ago.


     Type                              Name                       Plan       Info
     pulumi:pulumi:Stack               dev-eks-dev-charts-system             28 messages
 +   ├─ kubernetes:core/v1:Namespace   ingress                    create     
     └─ kubernetes:helm.sh/v3:Release  ingress-nginx                         1 error

Diagnostics:
  pulumi:pulumi:Stack (dev-eks-dev-charts-system):
    panic: interface conversion: interface {} is map[interface {}]interface {}, not map[string]interface {}
    goroutine 373 [running]:
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.excludeNulls({0x1029eec00?, 0x140061ff1d0?})
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/helm_release.go:1402 +0x2d0
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.excludeNulls({0x1029ecc80?, 0x140061eeb70?})
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/helm_release.go:1412 +0x214
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.mergeMaps(0x140061eeb70, 0x140061eedb0?, 0x0)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/helm_release.go:1389 +0xe4
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.decodeRelease(0x102319e51?, {0x14002e06200, 0x80})
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/helm_release.go:316 +0x338
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.(*helmReleaseProvider).Check(0x140016b8200, {0x102e151b0, 0x140061e9b60}, 0x14001968060)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/helm_release.go:359 +0x3ac
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.(*kubeProvider).Check(0x14000616480, {0x102e151b0, 0x140061e9b60}, 0x14001968060)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:1295 +0xfb8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Check_Handler.func1({0x102e151b0, 0x140061e9b60}, {0x102c1d940?, 0x14001968060})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.92.0/proto/go/provider_grpc.pb.go:557 +0x74
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x102e151b0, 0x140061e8db0}, {0x102c1d940, 0x14001968060}, 0x14004530020, 0x140003e4150)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x2e8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Check_Handler({0x102d760c0?, 0x14000616480}, {0x102e151b0, 0x140061e8db0}, 0x14002e00000, 0x140008f5700)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.92.0/proto/go/provider_grpc.pb.go:559 +0x12c
    google.golang.org/grpc.(*Server).processUnaryRPC(0x14000170780, {0x102e233e0, 0x14000c94340}, 0x14002d4e000, 0x140008ff7a0, 0x104601830, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1376 +0xb44
    google.golang.org/grpc.(*Server).handleStream(0x14000170780, {0x102e233e0, 0x14000c94340}, 0x14002d4e000, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1753 +0x7c4
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:998 +0x84
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 86
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:996 +0x154

  kubernetes:helm.sh/v3:Release (ingress-nginx):
    error: error reading from server: EOF
const _ingressNamespace = new k8s.core.v1.Namespace(
  'ingress',
  { metadata: { name: 'ingress' } },
  { provider: k8sProvider },
)

const _ingressNginxValues = new pulumi.asset.FileAsset('charts/ingress-nginx/values.yaml')
const _ingressNginxRelease = new helm.v3.Release(
  'ingress-nginx',
  {
    name: 'ingress-nginx',
    namespace: _ingressNamespace.metadata.name,
    chart: 'ingress-nginx',
    repositoryOpts: { repo: 'https://kubernetes.github.io/ingress-nginx' },
    version: '4.8.3',
    valueYamlFiles: [_ingressNginxValues],
  },
  { provider: k8sProvider, dependsOn: [_ingressNamespace] },
)

blampe added a commit that referenced this issue May 9, 2024
Users can run into a panic because we invoke `addPatchValueToDiff` like
so:

```go
pc.addPatchValueToDiff(append(path, k), v, old[k], newInput[k], oldInput[k], inArray)
```

These are `map[string]any` types and will return `nil` when the `[k]`
lookup returns `nil` _or_ when the key `k` is missing.

Then in `addPatchValueToDiff` we assert:
```go
contract.Assertf(v != nil || old != nil || oldInput != nil || newInput != nil)
```

but we don't have enough information to distinguish the "all values were
`nil`" case from the "one value was `nil` and the other was missing"
case. The assertion is only valid if all keys were present, hence we
panic when a key is removed when it previously had a value of `nil`.

The easiest fix is to relax the assertion (b886cfb) -- allow the
all-`nil` case since it can be valid. The downside is a poorer user
experience, because we don't have enough information to say whether the
update is an add or a delete; we could call it an update (odd) or a
no-op (lossy). The upside is it's a much smaller change with less risk.

Alternatively, we can have a better user experience by making
`addPatchValueToDiff` aware of whether keys were missing. This PR does
this by introducing a `missing` sentinel to distinguish from `nil`
values. This is riskier, but fortunately our test coverage is
comprehensive.

A failing unit test was added, along with another test to cover a small
bit of logic we weren't testing.

Fixes #1970.
@pulumi-bot pulumi-bot added the resolution/fixed This issue was fixed label May 9, 2024
lumiere-bot bot added a commit to coolguy1771/home-ops that referenced this issue May 24, 2024
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [@pulumi/kubernetes](https://pulumi.com)
([source](https://togithub.com/pulumi/pulumi-kubernetes)) | dependencies
| minor | [`4.11.0` ->
`4.12.0`](https://renovatebot.com/diffs/npm/@pulumi%2fkubernetes/4.11.0/4.12.0)
|

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.

---

### Release Notes

<details>
<summary>pulumi/pulumi-kubernetes (@&#8203;pulumi/kubernetes)</summary>

###
[`v4.12.0`](https://togithub.com/pulumi/pulumi-kubernetes/blob/HEAD/CHANGELOG.md#4120-May-21-2024)

[Compare
Source](https://togithub.com/pulumi/pulumi-kubernetes/compare/v4.11.0...v4.12.0)

##### Added

- Added a new Helm Chart v4 resource.
([pulumi/pulumi-kubernetes#2947)
- Added support for deletion propagation policies (e.g. Orphan).
([pulumi/pulumi-kubernetes#3011)
- Server-side apply conflict errors now include the original field
manager's name.
([pulumi/pulumi-kubernetes#2983)

##### Changed

- Pulumi will now wait for DaemonSets to become ready.
([pulumi/pulumi-kubernetes#2953)
- The Release resource's merge behavior for `valueYamlFiles` now more
closely matches Helm's behavior.
([pulumi/pulumi-kubernetes#2963)

##### Fixed

- Helm Chart V3 previews no longer fail when the cluster is unreachable.
([pulumi/pulumi-kubernetes#2992)
- Fixed a panic that could occur when a missing field became `null`.
([pulumi/pulumi-kubernetes#1970)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Renovate
Bot](https://togithub.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNzEuMSIsInVwZGF0ZWRJblZlciI6IjM3LjM3MS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJ0eXBlL21pbm9yIl19-->

Co-authored-by: lumiere-bot[bot] <98047013+lumiere-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants