Issue with DaemonSetPatch failing with invalid value error message

[zbuchheit]

What happened?

When attempting to use DaemonSetPatch on pulumi-kubernetes v4.5.5 I am hitting an error DaemonSet.apps "kube-proxy" is invalid: [spec.template.metadata.labels: Invalid value: map[string]string(nil): selector does not match template labels, spec.selector: Invalid value: "null": field is immutable] with annotations for both field manager and force patch set.

Example

Repo with observed behavior

1. Create a DaemonSetPatch resource using any v4 package before v4.5.2
2. After successful resource creation, change a field and run pulumi up to trigger an update. It will fail, but the pulumi-kubernetes manager will take ownership fields previously managed by kubectl-client-side-apply
3. Additional runs will result in the same error regardless of version.

Output of pulumi about

Version      3.92.0
Go Version   go1.21.3
Go Compiler  gc

Plugins
NAME        VERSION
go          unknown
kubernetes  4.0.0

Host     
OS       darwin
Version  14.0
Arch     arm64

This project is written in go: executable='/opt/homebrew/bin/go' version='go version go1.21.0 darwin/arm64'

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/zbuchheit-pulumi-corp
User           zbuchheit-pulumi-corp
Organizations  zbuchheit-pulumi-corp
Token type     personal

Dependencies:
NAME                                        VERSION
github.com/pulumi/pulumi-kubernetes/sdk/v4  4.0.0
github.com/pulumi/pulumi/sdk/v3             3.92.0

Additional context

There isn't a way for the kubernetes provider to correctly unclaim fields inadvertently taken control by the Patch resource in the past versions. So once encountered, the error must be resolved through ignoreChanges, creating a new fieldManager, or manually manipulating the kubernetes manifest.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[zbuchheit]

It appears the root cause of issue was fixed by v4.5.2 which prevents patch resources from taking ownership of unexpected fields via field management.

However, without a way for the kubernetes provider to correctly unclaim the fields it inadvertently has taken control of, it will require manual intervention or painful workarounds to resolve.

Any new resource created fresh from v4.5.2 onward will work, but a patch resource that was created when the bug was present in an older version is at risk for encountering this issue going forward.

[zbuchheit]

Documenting Solutions for Users Encountering This Issue

Disclaimer

To avoid re-encountering the original issue, it's crucial that you are using a version of the pulumi-kubernetes package after 4.5.2. These solutions are applicable and effective only with versions post-4.5.2, where the root cause of the field management conflict has been addressed.

Ensure pulumi.com/patchForce is Set: To guarantee that the new resource manager effectively takes ownership of the correct fields from the problematic manager, make sure to set the pulumi.com/patchForce annotation. This step is crucial for the proper resolution of the field management conflict.

Proposed Workarounds

1. Resource Re-creation:

   • Use pulumi state delete to remove the problematic Patch resource.
   • Recreate the resource. Ensure the pulumi.com/patchFieldManager annotation is not set, or rename it to avoid conflicts.

2. Automatic Replacement:

   • Set replaceOnChanges to "metadata.annotations" and use retainOnDelete.
   • Modify pulumi.com/patchFieldManager annotation or add it to create a new resource manager and replace the old patch resource.

3. Ignore Specific Changes:

   • Implement ignoreChanges set to "spec.template.metadata" and "spec.selector" to include these fields in API calls and avoid conflicts.

4. Include Missing Fields:

   • Explicitly add fields indicated as missing by the patch manager in your resource definition.

[zbuchheit]

resolved with #2719