VMWare: Run open-vm-tools to report guest information to vSphere

[zmrow]

Problem

While testing for the upcoming VMWare Kubernetes variant (#1451), we noticed that Bottlerocket VMs do not report their DNS name or IP addresses to vSphere. This is because we don't currently package or run VMWare tools in our images.

When attempting to join these images to a cluster previously provisioned via CAPV, Bottlerocket nodes were able to join the cluster but could never have pods scheduled on them because the node.cloudprovider.kubernetes.io/uninitialized taint was never removed by the cloud-provider. Upon further investigation we discovered that the vSphere cloud controller manager pod ( kube-system vsphere-cloud-controller-manager) wasn't able to initialize our nodes. This is because it appears to query vCenter for VM data (that doesn't exist because we don't report it).

I0421 20:33:10.905534       1 node_controller.go:325] Initializing node 198.19.31.255 with cloud provider
E0421 20:33:10.921307       1 datacenter.go:107] Unable to find VM by DNS Name. VM DNS Name: 198.19.31.255
E0421 20:33:10.921331       1 search.go:181] Error while looking for vm=198.19.31.255(byName) in vc=<redacted> and datacenter=<redacted>: No VM found
I0421 20:33:10.936035       1 nodemanager.go:123] Discovered VM using IP address
E0421 20:33:10.940196       1 node_controller.go:370] failed to initialize node 198.19.31.255 at cloudprovider: failed to get instance ID from cloud provider: VM Guest hostname is empty

For testing #1451 , we were able to run an open-vm-tools host container with superpowered=true. This allowed the DNS name and IP's to show up in vSphere.

In testing, simply running the one shot command /usr/bin/vmware-toolbox-cmd info update network with this container as explained here didn't to the trick and info was never updated. The container needed to run continuously to satisfy the polling interval explained in the previously mentioned article.

Potential solutions

Run open-vm-tools as a host container

This is what we did during testing and it works just fine. The disadvantage is we would probably want to maintain our own container for it and this is an ongoing maintenance burden. Another disadvantage is that the container would need to be pulled at runtime and any issue with that pull results in a non-operable k8s VM.

Write our own tool to publish this information

We could write our own Rust tool to respond to the polling and also to publish this information to vSphere. No crate exists to do this (though we could look into extending vmw_backdoor), so we would need to reference existing code or... ? Doing this will not be quick and our expertise lies elsewhere, so it's probably not the best option.

Package open-vm-tools and run it as a host process via systemd

This option might serve us better. Rather than maintaining another host container we can run the tools as a daemon via systemd. Having a package means we don't need to build/publish yet another host container, don't need to worry about migrations and the necessary tooling around them.

[tjkirch]

I lean toward the host process, if this is mandatory for VMware use, but a quick look at Fedora's packaging of open-vm-tools shows a lot of dependencies we wouldn't want; hopefully they're optional for our basic use case. If not, we may want to consider writing our own...

[bcressey]

Looks like the minimum package set we'll need is:

• open-vm-tools
• glib2 for open-vm-tools
• zlib and libffi for glib2

That's not too bad, and we're in the process of adding zlib already (for #1413).