Deployments are being replaced unnecessarily

[barclayadam]

What happened?

I have recently noticed in a number of our recent Pulumi runs that K8S resources are being unnecessarily replaced instead of updated, but I do not understand what is triggering this behaviour.

Steps to reproduce

When running preview (this is using Automation API) where the only that has changed is the contents of a Secret I can see in the Pulumi web app:

Changes:
 
    Type                            Name                                    Status    Info
    pulumi:pulumi:Stack             s365-postgres-production-deploy-ae-aen            
~   ├─ pulumi:providers:kubernetes  aks                                     updated   [diff: ~version]
    └─ sig365:DataCentre            UAE North                                         
++     └─ kubernetes:core:Secret    pgbouncer-config                        replaced  
 
Resources:
    +-replace 2
    ~ update 1
    11 unchanged
 
Duration: 2s

Note that the summary includes 2 resources to be replaced but does not output the 2nd one, the Deployment, only the Secret.

Looking at the Diff tab the Deployment replacement is listed (almost all is highlighted in red up until spec.replicas when the diff view becomes green, before reverting to red for last status):

sig365:DataCentre$sig365:aks:DockerDeployment$kubernetes:apps/v1:Deployment (pgbouncer-aks-deployment)
++ kubernetes:apps/v1:Deployment (create-replacement)
    [id=pgbouncer/pgbouncer]
    [urn=urn:pulumi:production-deploy-ae-aen::s365-postgres::sig365:DataCentre$sig365:aks:DockerDeployment$kubernetes:apps/v1:Deployment::pgbouncer-aks-deployment]
    __fieldManager     : "pulumi-kubernetes-dd802cd0"
    __initialApiVersion: "apps/v1"
    __inputs           : {
        apiVersion: "apps/v1"
        kind      : "Deployment"
        metadata  : {
            labels   : {
                app.kubernetes.io/managed-by: "pulumi"
            }
            name     : "pgbouncer"
            namespace: "pgbouncer"
        }
        spec      : [secret]
    }
    metadata           : {
        annotations      : {
            deployment.kubernetes.io/revision               : "1"
            kubectl.kubernetes.io/last-applied-configuration: [secret]
        }
        creationTimestamp: "2022-11-28T12:49:11Z"
        generation       : 1
        managedFields    : [
            [0]: {
                apiVersion: "apps/v1"
                fieldsType: "FieldsV1"
                fieldsV1  : {
                    f:metadata: {
                        f:annotations: {
                            .                                                 : {}
                            f:kubectl.kubernetes.io/last-applied-configuration: {}
                        }
                        f:labels     : {
                            .                             : {}
                            f:app.kubernetes.io/managed-by: {}
                        }
                    }
                    f:spec    : {
                        f:progressDeadlineSeconds: {}
                        f:replicas               : {}
                        f:revisionHistoryLimit   : {}
                        f:selector               : {}
                        f:strategy               : {
                            f:rollingUpdate: {
                                .               : {}
                                f:maxSurge      : {}
                                f:maxUnavailable: {}
                            }
                            f:type         : {}
                        }
                        f:template               : {
                            f:metadata: {
                                f:annotations: {
                                    .                     : {}
                                    f:prometheus.io/port  : {}
                                    f:prometheus.io/scrape: {}
                                }
                                f:labels     : {
                                    .                       : {}
                                    f:app                   : {}
                                    f:app.kubernetes.io/name: {}
                                }
                            }
                            f:spec    : {
                                f:automountServiceAccountToken : {}
                                f:containers                   : {
                                    k:{"name":"exporter"} : {
                                        .                         : {}
                                        f:env                     : {
                                            .                 : {}
                                            k:{"name":"PodId"}: {
                                                .          : {}
                                                f:name     : {}
                                                f:valueFrom: {
                                                    .         : {}
                                                    f:fieldRef: {}
                                                }
                                            }
                                        }
                                        f:image                   : {}
                                        f:imagePullPolicy         : {}
                                        f:name                    : {}
                                        f:ports                   : {
                                            .                                        : {}
                                            k:{"containerPort":9127,"protocol":"TCP"}: {
                                                .              : {}
                                                f:containerPort: {}
                                                f:name         : {}
                                                f:protocol     : {}
                                            }
                                        }
                                        f:resources               : {}
                                        f:terminationMessagePath  : {}
                                        f:terminationMessagePolicy: {}
                                        f:volumeMounts            : {
                                            .                                         : {}
                                            k:{"mountPath":"/etc/pgbouncer-exporter/"}: {
                                                .          : {}
                                                f:mountPath: {}
                                                f:name     : {}
                                                f:readOnly : {}
                                            }
                                        }
                                    }
                                    k:{"name":"pgbouncer"}: {
                                        .                         : {}
                                        f:env                     : {
                                            .                                 : {}
                                            k:{"name":"ASPNETCORE_URLS"}      : {
                                                .      : {}
                                                f:name : {}
                                                f:value: {}
                                            }
                                            k:{"name":"Kubernetes__AppName"}  : {
                                                .          : {}
                                                f:name     : {}
                                                f:valueFrom: {
                                                    .         : {}
                                                    f:fieldRef: {}
                                                }
                                            }
                                            k:{"name":"Kubernetes__Namespace"}: {
                                                .          : {}
                                                f:name     : {}
                                                f:valueFrom: {
                                                    .         : {}
                                                    f:fieldRef: {}
                                                }
                                            }
                                            k:{"name":"Kubernetes__PodId"}    : {
                                                .          : {}
                                                f:name     : {}
                                                f:valueFrom: {
                                                    .         : {}
                                                    f:fieldRef: {}
                                                }
                                            }
                                            k:{"name":"Kubernetes__PodName"}  : {
                                                .          : {}
                                                f:name     : {}
                                                f:valueFrom: {
                                                    .         : {}
                                                    f:fieldRef: {}
                                                }
                                            }
                                        }
                                        f:image                   : {}
                                        f:imagePullPolicy         : {}
                                        f:lifecycle               : {
                                            .        : {}
                                            f:preStop: {
                                                .     : {}
                                                f:exec: {
                                                    .        : {}
                                                    f:command: {}
                                                }
                                            }
                                        }
                                        f:livenessProbe           : {
                                            .                    : {}
                                            f:failureThreshold   : {}
                                            f:initialDelaySeconds: {}
                                            f:periodSeconds      : {}
                                            f:successThreshold   : {}
                                            f:tcpSocket          : {
                                                .     : {}
                                                f:port: {}
                                            }
                                            f:timeoutSeconds     : {}
                                        }
                                        f:name                    : {}
                                        f:ports                   : {
                                            .                                        : {}
                                            k:{"containerPort":5432,"protocol":"TCP"}: {
                                                .              : {}
                                                f:containerPort: {}
                                                f:name         : {}
                                                f:protocol     : {}
                                            }
                                        }
                                        f:readinessProbe          : {
                                            .                    : {}
                                            f:failureThreshold   : {}
                                            f:initialDelaySeconds: {}
                                            f:periodSeconds      : {}
                                            f:successThreshold   : {}
                                            f:tcpSocket          : {
                                                .     : {}
                                                f:port: {}
                                            }
                                            f:timeoutSeconds     : {}
                                        }
                                        f:resources               : {}
                                        f:securityContext         : {
                                            .                         : {}
                                            f:allowPrivilegeEscalation: {}
                                            f:capabilities            : {
                                                .     : {}
                                                f:drop: {}
                                            }
                                        }
                                        f:terminationMessagePath  : {}
                                        f:terminationMessagePolicy: {}
                                        f:volumeMounts            : {
                                            .                               : {}
                                            k:{"mountPath":"/etc/pgbouncer"}: {
                                                .          : {}
                                                f:mountPath: {}
                                                f:name     : {}
                                                f:readOnly : {}
                                            }
                                        }
                                    }
                                }
                                f:dnsPolicy                    : {}
                                f:nodeSelector                 : {}
                                f:restartPolicy                : {}
                                f:schedulerName                : {}
                                f:securityContext              : {}
                                f:shareProcessNamespace        : {}
                                f:terminationGracePeriodSeconds: {}
                                f:topologySpreadConstraints    : {
                                    .                                                                             : {}
                                    k:{"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"DoNotSchedule"}: {
                                        .                  : {}
                                        f:labelSelector    : {}
                                        f:maxSkew          : {}
                                        f:topologyKey      : {}
                                        f:whenUnsatisfiable: {}
                                    }
                                }
                                f:volumes                      : {
                                    .                           : {}
                                    k:{"name":"exporterconfig"} : {
                                        .       : {}
                                        f:name  : {}
                                        f:secret: {
                                            .            : {}
                                            f:defaultMode: {}
                                            f:secretName : {}
                                        }
                                    }
                                    k:{"name":"pgbouncerconfig"}: {
                                        .       : {}
                                        f:name  : {}
                                        f:secret: {
                                            .            : {}
                                            f:defaultMode: {}
                                            f:secretName : {}
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
                manager   : "pulumi-resource-kubernetes.exe"
                operation : "Update"
                time      : "2022-11-28T12:49:11Z"
            }
            [1]: {
                apiVersion : "apps/v1"
                fieldsType : "FieldsV1"
                fieldsV1   : {
                    f:metadata: {
                        f:annotations: {
                            f:deployment.kubernetes.io/revision: {}
                        }
                    }
                    f:status  : {
                        f:availableReplicas : {}
                        f:conditions        : {
                            .                       : {}
                            k:{"type":"Available"}  : {
                                .                   : {}
                                f:lastTransitionTime: {}
                                f:lastUpdateTime    : {}
                                f:message           : {}
                                f:reason            : {}
                                f:status            : {}
                                f:type              : {}
                            }
                            k:{"type":"Progressing"}: {
                                .                   : {}
                                f:lastTransitionTime: {}
                                f:lastUpdateTime    : {}
                                f:message           : {}
                                f:reason            : {}
                                f:status            : {}
                                f:type              : {}
                            }
                        }
                        f:observedGeneration: {}
                        f:readyReplicas     : {}
                        f:replicas          : {}
                        f:updatedReplicas   : {}
                    }
                }
                manager    : "kube-controller-manager"
                operation  : "Update"
                subresource: "status"
                time       : "2022-11-28T12:49:42Z"
            }
        ]
        resourceVersion  : "89359222"
        uid              : "14f95766-dc7a-46ef-a591-9521aa5f160e"
    }
    spec               : {
        ciphertext          : "[secret]"
        replicas            : 2
        revisionHistoryLimit: 1
        selector            : {
            matchLabels: {
                app: "pgbouncer"
            }
        }
        strategy            : {
            rollingUpdate: {
                maxUnavailable: 0
            }
        }
        template            : {
            metadata: {
                annotations: {
                    prometheus.io/port  : "9127"
                    prometheus.io/scrape: "true"
                }
                labels     : {
                    app                   : "pgbouncer"
                    app.kubernetes.io/name: "pgbouncer"
                }
            }
            spec    : {
                automountServiceAccountToken : false
                containers                   : [
                    [0]: {
                        env            : [
                            [0]: {
                                name : "ASPNETCORE_URLS"
                                value: "http://*:80/"
                            }
                            [1]: {
                                name     : "Kubernetes__AppName"
                                valueFrom: {
                                    fieldRef: {
                                        fieldPath: "metadata.labels['app.kubernetes.io/name']"
                                    }
                                }
                            }
                            [2]: {
                                name     : "Kubernetes__Namespace"
                                valueFrom: {
                                    fieldRef: {
                                        fieldPath: "metadata.namespace"
                                    }
                                }
                            }
                            [3]: {
                                name     : "Kubernetes__PodName"
                                valueFrom: {
                                    fieldRef: {
                                        fieldPath: "metadata.name"
                                    }
                                }
                            }
                            [4]: {
                                name     : "Kubernetes__PodId"
                                valueFrom: {
                                    fieldRef: {
                                        fieldPath: "metadata.uid"
                                    }
                                }
                            }
                        ]
                        image          : "edoburu/pgbouncer:1.17.0"
                        imagePullPolicy: "Always"
                        lifecycle      : {
                            preStop: {
                                exec: {
                                    command: [
                                        [0]: "/bin/sh"
                                        [1]: "-c"
                                        [2]: "killall -s -INT 1 && sleep 120s"
                                    ]
                                }
                            }
                        }
                        livenessProbe  : {
                            initialDelaySeconds: 60
                            periodSeconds      : 10
                            tcpSocket          : {
                                port: 5432
                            }
                        }
                        name           : "pgbouncer"
                        ports          : [
                            [0]: {
                                containerPort: 5432
                                name         : "postgres"
                            }
                        ]
                        readinessProbe : {
                            failureThreshold   : 6
                            initialDelaySeconds: 20
                            periodSeconds      : 10
                            tcpSocket          : {
                                port: 5432
                            }
                        }
                        securityContext: {
                            allowPrivilegeEscalation: false
                            capabilities            : {
                                drop: [
                                    [0]: "all"
                                ]
                            }
                        }
                        volumeMounts   : [
                            [0]: {
                                mountPath: "/etc/pgbouncer"
                                name     : "pgbouncerconfig"
                                readOnly : true
                            }
                        ]
                    }
                    [1]: {
                        env            : [
                            [0]: {
                                name     : "PodId"
                                valueFrom: {
                                    fieldRef: {
                                        fieldPath: "metadata.uid"
                                    }
                                }
                            }
                        ]
                        image          : "spreaker/prometheus-pgbouncer-exporter:2.1.1"
                        imagePullPolicy: "Always"
                        name           : "exporter"
                        ports          : [
                            [0]: {
                                containerPort: 9127
                                name         : "metrics"
                            }
                        ]
                        volumeMounts   : [
                            [0]: {
                                mountPath: "/etc/pgbouncer-exporter/"
                                name     : "exporterconfig"
                                readOnly : true
                            }
                        ]
                    }
                ]
                nodeSelector                 : {
                    kubernetes.io/os: "linux"
                }
                shareProcessNamespace        : true
                terminationGracePeriodSeconds: 120
                topologySpreadConstraints    : [
                    [0]: {
                        labelSelector    : {
                            matchLabels: {
                                app: "pgbouncer"
                            }
                        }
                        maxSkew          : 1
                        topologyKey      : "kubernetes.io/hostname"
                        whenUnsatisfiable: "DoNotSchedule"
                    }
                ]
                volumes                      : [
                    [0]: {
                        name  : "pgbouncerconfig"
                        secret: {
                            secretName: "pgbouncer-config"
                        }
                    }
                    [1]: {
                        name  : "exporterconfig"
                        secret: {
                            secretName: "pgbouncer-exporter-config"
                        }
                    }
                ]
            }
        }
    }
    status             : {
        availableReplicas : 2
        conditions        : [
            [0]: {
                lastTransitionTime: "2022-11-28T12:49:42Z"
                lastUpdateTime    : "2022-11-28T12:49:42Z"
                message           : "Deployment has minimum availability."
                reason            : "MinimumReplicasAvailable"
                status            : "True"
                type              : "Available"
            }
            [1]: {
                lastTransitionTime: "2022-11-28T12:49:11Z"
                lastUpdateTime    : "2022-11-28T12:49:42Z"
                message           : "ReplicaSet "pgbouncer-5dddd5bdf9" has successfully progressed."
                reason            : "NewReplicaSetAvailable"
                status            : "True"
                type              : "Progressing"
            }
        ]
        observedGeneration: 1
        readyReplicas     : 2
        replicas          : 2
        updatedReplicas   : 2
    }

Additionally, the Secret is listed, as below as a replacement. We have only updated a single value within the secret so would not expect a replacement

Expected Behavior

Existing Deployment and Secret is updated

Actual Behavior

Deployment and Secret are replaced

Output of pulumi about

CLI
Version      3.49.0
Go Version   go1.19.3
Go Compiler  gc

Plugins
NAME             VERSION
azure            4.37.0
azure-native     1.84.0
azuread          5.26.1
cloudflare       3.6.0
docker           3.0.0
docker-buildkit  0.1.17
kubernetes       3.23.0
newrelic         4.18.0
nodejs           unknown
random           4.0.0
tls              4.0.0

Host
OS       Microsoft Windows 11 Pro
Version  10.0.22000 Build 22000
Arch     x86_64

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[barclayadam]

Another example of a Deployment that has been replaced with no actual changes with diff in Pulumi being version small and clearly only inputs I do not manage:

[squaremo]

Thanks for putting this on our radar! In that second screenshot, I can see

manager: "pulumi-resource-kubernetes" => "pulumi-resource-kubernetes.exe"

and that makes me think of #2218. I wonder if it's the same underlying problem (and whether #2271) will fix it). @lblackstone may be able to comment on whether they are related.

The replacement not appearing in the output is another concern.

[barclayadam]

@squaremo that linked issue could be a cause, as our deployments are run from Linux primarily, but may also get deployed locally on Windows.

I commented on a different issue, pulumi/pulumi-azure-native#2059, which may actually be the same cause. It demonstrates the same symptoms here, where there is no mention of the replacement

[lblackstone]

I think this was fixed in #2445

We can take another look if this is still a problem with the v4 provider.