Spurious diffs after updating resource with SSA followed by CSA

[henry-fn]

What happened?

1. Applied with CSA (in the past, this is an old resource)
2. Moved to using SSA and have been for months
3. A user who does not have the needed permissions to run preview for SSA (since SSA requires full permissions to do the apply) desires to preview changes. This led to them reasonably using a CSA preview (kubernetes:enableServerSideApply: "false")
4. The CSA preview shows spurious diffs in the metadata field
   • Specifically the fields that have been added since the move to pulumi + SSA (metadata.annotations pulumi.com/patchFieldManager and pulumi.com/patchForce and metadata.labels...)
   • This does not go away with a refresh

ATTEMPTED (none of these fix it):

1. Deleted the old kubectl-client-side-apply field manager via kubectl patch ... and refreshed
2. Deleted the resource in k8s and re-created it

LEADING TO WORKAROUND:

I then poked around the state file and noticed that there is a diff between the outputs.__inputs field and the .inputs field, it seems that outputs.__inputs is not updated on a refresh as I would expect. I figured a hard refresh would fix it (deleting the state object and importing)

WORKAROUND (until new fields are added again):

1. Delete the state object
2. pulumi import the object back so it has the exact same URN
3. (optional) refresh
4. everything is fixed and there are no more diffs in CSA or SSA

Expected Behavior

Assuming:

• The state in k8s matches the desired state by pulumi
• Refresh has been run since the most recent changes

Then:

previewing with kubernetes:enableServerSideApply: "false" set should not provide a diff

Steps to reproduce

1. Take an existing deployment that has kubernetes:enableServerSideApply: "false" set
2. Turn on kubernetes:enableServerSideApply: "true"
3. Change some metadata annotations or labels
4. Apply
5. Set kubernetes:enableServerSideApply: "false"
6. Run a preview (with or without refreshing)
7. You should see a diff

Output of pulumi about

CLI
Version 3.66.0
Go Version go1.20.3
Go Compiler gc

Plugins
NAME VERSION
python unknown

Host
OS darwin
Version 13.3.1
Arch arm64

This project is written in python: executable='/opt/homebrew/bin/python3' version='3.10.9
'

Current Stack: ...

TYPE URN
...

Found no pending operations associated with dev

Backend
Name ...
URL gs://...
User henry.peteet
Organizations

Dependencies:
NAME VERSION
iterm2 2.7.0
keyrings.google-artifactregistry-auth 1.1.0
pep440 0.1.2
pip 23.0.0
pyobjc 8.5.1
setuptools 65.6.3
wheel 0.38.4

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[danielrbradley]

Hi @henry-fn thanks for reporting this.

Do you have an example program you can share which demonstrates this behaviour - would reproduce the issue if deploying using SSA then running a CSA preview consistently shows this?

[danielrbradley]

Adding a little more background on this. Moving between SSA and CSA isn't currently a well-tested use-case. The current expectation is that having switched to using SSA, it's going to be used from that point forward.

[henry-fn]

That makes sense, I will try to get a sample up and running.

For some background we want to be able to run CSA previews (never up commands, those will all be SSA) so that users / less privileged bots can preview changes without having to have full permissions to do everything they are proposing (a real limit with k8s SSA). This is because we want to only give deployment permissions to our CD system + oncall engineers. Non-oncall engineers will still be able to propose changes, but those changes will have to be reviewed before being merged into main and deployed via the CD system. Having a preview is itself a very helpful tool in the review process to make sure unintended changes aren't being made, but we can't safely run a SSA preview on unreviewed code.

We realize that there may be some diffs between the 2 systems regarding fields that are actually shared via SSA, but we would expect to see no diffs for fields managed entirely by pulumi.

[lblackstone]

I think that this is fixed with the changes made in #2445, which are included in the v4 provider release. I'm going to close this out, and we can re-open if you are able to repro against the v4 code.