Race between kubelet startup and mounter prepare

[mattcary]

What happened?

In GCE cluster startup, kubelet-start (which puts the cluster DNS server into the containerized mounter resolv.conf when the containerized mounter is enabled) race with prepare-mounter-rootfs (which copies the instance resolve.conf into the containerized mounter fs).

This causes domain resolution from the containerized mounter to be flakey.

What did you expect to happen?

No race to occur.

How can we reproduce it (as minimally and precisely as possible)?

Create a GCE cluster with the containerized mounter enabled. Create several nodes. The nameserver in the resolv.conf file in the mounter fs will not be the same.

Note that while it appears the race has existed for a long time, it's only started to manifest around v1.23.

Anything else we need to know?

No response

Kubernetes version

v1.23+

Cloud provider

GCE

OS version

No response

Install tools

No response

Container runtime (CRI) and version (if applicable)

No response

Related plugins (CNI, CSI, ...) and versions (if applicable)

No response

[mattcary]

/sig storage
/assign @mattcary

[mattcary]

/triage accepted