How does drift/overlap in changes outside of Pulumi get resolved when apply updates and refreshing?

[sheldonhull]

Trying to figure out the behavior with an update on pod annotations here and a conflict that I think was caused due to a manual edit I wasn't aware of. Not sure if it's a red herring though.

• Deploy
• I edit pod annotation to i shouldn't change this
• I run a pipeline that does a diff with refresh
• I run up
• Pod annotation doesn't apply fix, state file doesn't seem to show the changed value pulled in either.

Should it fix it?

And last caveat:

• Deploy without pod annotation
• Someone goes in and adds pod annotation manually
• I run pipeline thinking it's going to deploy all the new tags I enabled that someone already added manually
• Refresh/diff.
• Pod annotation doesn't apply fix, state file doesn't seem to show the changed value pulled in either.

Trying to make sure I wrap my head around the way the state file, refresh, drift work so I can communicate this a bit better to team.

Possibly Related

• Refresh not working as expected on some resources pulumi-kubernetes#1525

[AaronFriel]

This may need to be documented better, but I think this is expected - changes to a resource made outside of Pulumi cause it not to be tracked. IIRC, the granularity here is a specific key/value. If I change "replicas: 1" to "replicas: 2" in a deployment, I believe that only the replicas field becomes untracked by Pulumi.

@viveklak is that your understanding of the Kubernetes provider?

[rawkode]

I believe this is the expected, and correct, behaviour too; especially with managedFields and server side apply becoming the default behaviour in 1.24.

[sheldonhull]

I believe this is the expected, and correct, behaviour too; especially with managedFields and server side apply becoming the default behaviour in 1.24.

Is there any issue or docs on this so I can better understand what this means? I need to make sure I have clarity for team on what change will correctly redeploy.

[sheldonhull]

Is there a way to disable this behavior @rawkode @AaronFriel. This is problematic to me as the source of truth = pulumi/infra-as-code. If someone deviates from this I want the pulumi stack to align with what's in code, not silently start ignoring it. To me this violates the intention of managing my infra as code, as now potentially someone could tweak a configmap, tag, or something else and I'd never align it without forcing a destroy and rebuild.

[sheldonhull]

Follow-up. I was running watch with --refresh. Once I stopped --refresh for the watch command any changes to the underlying configmap data was correctly detected and resulted in the Deployment, Role and Config map all properly updating.

[viveklak]

Can you try your use-case with https://www.pulumi.com/registry/packages/kubernetes/api-docs/provider/#enabledryrun_nodejs enabled on the provider? Part of this is related to a reliance on kubectl.kubernetes.io/last-applied-configuration to determine the last applied state. The above flag should use server-side diff functionality.

[nimbinatus]

Note that GitHub Discussions doesn't let us mark a thread response as an answer, so we're marking the top of the thread that contains the answer 😄