Replacing failed but no changes to be done on next run

[avdv]

What happened?

I was running pulumi up and observed an error while replacing a k8s secret:

Updating (infra):
     Type                          Name                              Status                   Info
     pulumi:pulumi:Stack           infra  **failed**               1 error
 +-  └─ kubernetes:core/v1:Secret  solver-svc-acct    **replacing failed**     [diff: ~data]; 1 error


Diagnostics:
  kubernetes:core/v1:Secret (solver-svc-acct):
    error: resource cluster-ns/solver-svc-acct was not successfully created by the Kubernetes API server : Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 0

  pulumi:pulumi:Stack (infra):
    error: update failed

Outputs:
   ...

Resources:
    +-1 replaced
    14 unchanged

Duration: 15s

This happened due to passing non-base64 encoded data to the data argument for core.v1.Secret's constructor.

After fixing the problem, running pulumi up again results in no changes to be done:

Previewing update (infra):
     Type                 Name                              Plan     
     pulumi:pulumi:Stack  infra           


Resources:
    16 unchanged

The resource was indeed not created in the k8s cluster, but seems like Pulumi's state is saying otherwise.

BTW, running pulumi refresh deleted the secret from the state and I could run pulumi up again.

Expected Behavior

I would expect that Pulumi does not register the resource as created, if there was an error.

And hence I would expect the resource would be created the next time running pulumi up.

Steps to reproduce

new k8s.core.v1.Secret(
    "solver-svc-acct",
    {
      metadata: {
        name: "solver-svc-acct",
        namespace: "cluster-ns",
      },
      type: "Opaque",
      data: {
        "key":  << use some invalid base64 data first, then encoded base64 >> ,
      },
    }
  );

Output of pulumi about

CLI          
Version      3.55.0
Go Version   go1.20
Go Compiler  gc

Plugins
NAME    VERSION
nodejs  unknown

Host     
OS       nixos
Version  22.11 (Raccoon)
Arch     x86_64

This project is written in nodejs: executable='/nix/store/dnhhyhfql3cs9jmrhd3a05620higxy0i-nodejs-18.14.2/bin/node' version='v18.14.2'

Current Stack: infra

TYPE                                       URN
kubernetes:core/v1:Secret                  urn:pulumi:infra::project::kubernetes:core/v1:Secret::solver-svc-acct
...

Found no pending operations associated with infra

Backend        
Name           x1
URL            gs://da-cn-pulumi-stacks
User           claudio
Organizations  


### Additional context

_No response_

### Contributing

Vote on this issue by adding a 👍 reaction. 
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already). 

[kpitzen]

Hi @avdv - thank you for opening this issue. We very much appreciate your clear reproduction, and will look into this as soon as we can - I expect this to be a relatively straightforward fix regarding how the provider reports success back to the engine, but it will take a bit more in-depth investigation to be sure.

Thanks again!

[lblackstone]

I wasn't able to reproduce this with the v4 provider, so I suspect this may have been fixed by #2445.