Changing a ConfigMap's data causes a replacement

[ghost]

Changing a ConfigMap's data causes a replacement of that resource, causing other resources that depend on it to be replaced as well (e.g. Deployment using envFrom), causing downtime.

Expected behavior

The ConfigMap should be updated in place.

Steps to reproduce

Running pulumi up twice with the following code:

const testConfigMap = new k8s.core.v1.ConfigMap(
  'test',
  {
    data: {foo: `${Date.now()}`},
  }
);

will result in a replacement of that configMap

 +-  └─ kubernetes:core/v1:ConfigMap  test                                       replace     [diff: ~data]

[lblackstone]

@pierlucg-xs Is the Deployment being replaced, or just updated?

Pulumi's k8s provider intentionally treats ConfigMap resources as immutable. I would expect this to trigger a rollout (update) in dependent Deployments rather than a replacement.

[ghost]

Yes, here's how I reproduced it with K3S:

const provider = new k8s.Provider("k3s", {
  kubeconfig,
  enableDryRun: false,
});

const configMap = new k8s.core.v1.ConfigMap(
  "configmap",
  {
    data: { foo: `${Date.now()}` },
    metadata: {
      name: "configmap",
    },
  },
  { provider: provider }
);

const deployment = new k8s.apps.v1.Deployment(
  "deployment",
  {
    apiVersion: "apps/v1",
    kind: "Deployment",
    metadata: {
      labels: {
        app: "bar",
      },
      name: "bar",
      namespace: "default",
    },
    spec: {
      replicas: 1,
      selector: {
        matchLabels: {
          app: "bar",
        },
      },
      template: {
        metadata: {
          labels: {
            app: "bar",
          },
        },
        spec: {
          containers: [
            {
              envFrom: [{ configMapRef: { name: configMap.metadata.name } }],
              image: "nginxdemos/hello",
              name: "demo-service",
              ports: [{ containerPort: 8080 }],
            },
          ],
        },
      },
    },
  },
  { provider: provider }
);

Second pulumi up:

     Type                              Name               Plan        Info
   pulumi:pulumi:Stack               k3s-configmaptest              
+-  ├─ kubernetes:apps/v1:Deployment  deployment         replace     
+-  └─ kubernetes:core/v1:ConfigMap   configmap          replace     [diff: ~data]

---

Pulumi's k8s provider intentionally treats ConfigMap resources as immutable

Any ideas why? AFAIK ConfigMaps have a PATCH api

[lblackstone]

Ah, ok. The reason you're seeing the replacement is because you are manually specifying the ConfigMap name rather than using auto-naming:

metadata: {
  name: "configmap",
},

If you remove that and let Pulumi auto-name the ConfigMap, the Deployment will update rather than replace.

---

Our k8s provider intentionally treats ConfigMap and Secret resources as immutable rather than using the PATCH API for a couple reasons:

1. Pods referencing the ConfigMap/Secret have to be restarted for changes to take effect. This is a common source of subtle errors where a subset of Pods are referencing the updated ConfigMap/Secret values, while others are still using stale data.
2. With autonamed ConfigMap/Secret resources, Pulumi will create a new instance, update dependents to refer to the new instance, and only remove the old instance once that update is successful. This makes it easier to rollback errors because the original data is still present.

I'd be interested to hear more about the use case if you are intentionally trying to reuse the same ConfigMap.

[ghost]

I was not using auto-naming because the cluster was managed with kubectl before and I imported those resources into pulumi as-is, but it's not a hard requirement and I'll definitely change it to use auto-naming.

---

I've definitely been bitten by number 1. in the past! I'd argue that by trying to alleviate a Kubernetes issue, Pulumi is acting in a non "idiomatic K8S" way.

All in all, the benefits of auto-naming are clear and will resolve my issue, thanks a lot for your quick response!

[lblackstone]

I'd argue that by trying to alleviate a Kubernetes issue, Pulumi is acting in a non "idiomatic K8S" way.

Right, sorry for the confusion here, and thanks for linking that upstream issue! I agree that this behavior could be a surprise to some k8s users.

I do think this is the behavior that most users actually want, so I'll open a work item to make this clearer in our docs.

[iridian-ks]

I know this is a closed issue so happy to start a new ticket. Unrelated to deployments, in EKS there is an aws-auth ConfigMap. I noticed when trying to add new users and roles Pulumi will replace the ConfigMap. This causes all of the NodeGroups to permanently enter a failing state and never recover (it does seems random and it doesn't happen all of the time [but most of the time]).

I understand for Deployments that a replace is beneficial, but it would be nice if the user could control the behavior if they need to instead of making assumptions on what is using the ConfigMap.

[aschleck]

This aws-auth behavior remains quite a large landmine and it's surprising behavior that the Pulumi default behavior is to risk bricking the cluster.

[ericrudder]

Let's take another look at this ...

[mikhailshilkov]

Thank you for your patience everyone. It sounds like the current behaviour of replacing ConfigMap is by design, but we need to provide a mechanism to override this behaviour and change the replacement to an in-place update. Probably, a solution should involve some sort of resource option or annotation.

I've started an internal discussion of what the design for it may be.

[lblackstone]

we need to provide a mechanism to override this behaviour and change the replacement to an in-place update

This is possible already if you use the enableConfigMapMutable provider flag. I suspect that we may want to swap the default in a future (major?) release.

Given the launch of Pulumi ESC this week, I'll also note that you could set this as a default via the environment variable. That could be a nice way to pull in standardized configurations that are applied to each stack in an organization.

[mikhailshilkov]

Awesome! Thank you @lblackstone

I'd love to hear back from folks on this ticket whether this workaround works for them.

[aschleck]

Was not familiar with that option but it certainly sounds like exactly what's needed here. Not sure when I'll have time to give it a try but will do so. Thank you all!

[EronWright]

Some background:

• aws-auth configmap does not get re-created after a cluster replacement, preventing nodes from joining the cluster pulumi-eks#107