VMWare Guestinfo Interface does not take effect

[Winnie81]

Issue Report

VMWare Guestinfo Interface does not take effect

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1409.2.0
VERSION_ID=1409.2.0
BUILD_ID=2017-06-19-2321
PRETTY_NAME="Container Linux by CoreOS 1409.2.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

ESXi 6.0 and ESXi 6.5.

Expected Behavior

Configuration items in vmx file added in vm can take effect.

Actual Behavior

Configuration items in vmx file added in vm do not take effect.

Reproduction Steps

1. Install CoreOS vm from VMWare ESX, then configure user name and password.
2. Add some configuration items into vmx file when CoreOS is in powered off status, such as hostname, IP and so on.
3. Power on vm then check the related settings.

[Winnie81]

@john Savanyo jsavanyo@vmware.com

[Winnie81]

@vmware-gos-qa@vmware.com

[lucab]

Add some configuration items into vmx file when CoreOS is in powered off status, such as hostname, IP and so on.

It looks like you are trying to adjust the configuration after the first provisioning, but I'm not sure if vmtoolsd is capable of that. Instead, it is recommend that you specify the configuration to be applied on first boot as described here: https://coreos.com/os/docs/latest/booting-on-vmware.html#container-linux-configs.

[Winnie81]

@lucab, Do you mean these configurations can only be ingested at the first boot after the vm is installed? In previous versions, we can customize the vm at any time, so long as it is powered off.

[crawford]

It depends on how you are specifying the configuration. If you are using b Ignition settings will only be honored during the first boot. If you are using general vmtools commands, then they should always take effect.

Can you be more specific with your reproduction steps?

[Winnie81]

This issue could still be reproduced with CoreOS 1465.6.0 on ESX 6.5 GA with below steps:
1.Install CoreOS vm from VMWare ESX, then configure user name and password.
2. Power off vm.
3. Add some configuration items into vmx file, such as hostname, IP and so on.
4. Power on vm then check the related settings.

[crawford]

@Winnie81 I couldn't find any VMware documentation that claims that this should work. Network related information can be specified to the VM via DHCP, but I do not know if ESX will update that information internally when the vmx changes. You might need to update the vApp options via the vSphere APIs.

[Winnie81]

@crawford, according to https://coreos.com/os/docs/latest/booting-on-vmware.html#container-linux-configs, I used the fourth method "You can manually modify the VMX and reload it on the VMware Workstation, ESXi host, or in vCenter." But sometimes it works, and sometimes it doesn't work.

[crawford]

@Winnie81 Sorry for the delay, this slipped through the cracks. Can you share your Container Linux Config (or the resulting Ignition Config)? The Open VM Tools should run regardless, but I'm curious to see the config.

[Winnie81]

@crawford, they are as below:
guestinfo.hostname = “WinnieCore1520_ens192"               
guestinfo.interface.0.name = "ens192"                                                            
guestinfo.interface.0.dhcp = "no"                               
guestinfo.interface.0.role = "public"                                                                                                          
guestinfo.interface.0.ip.0.address = "10.117.16.149/23"
guestinfo.interface.0.route.0.gateway = "10.117.17.253"         
guestinfo.interface.0.route.0.destination = "0.0.0.0/0"    
guestinfo.dns.server.0 = "10.117.0.2"  

[crawford]

Those settings are only read by coreos-cloudinit. Are you setting guestinfo.coreos.config.data? If that contains an Ignition config, coreos-cloudinit won't run.

[Winnie81]

@crawford Sorry for the delay. Yes, I set guestinfo.coreos.config.data as this:
guestinfo.coreos.config.data = "xxxx"
guestinfo.coreos.config.data.encoding = "base64"
then added above configurations. But I still got "failed to fetch config: not a config (empty)". According to https://coreos.com/os/docs/latest/booting-on-vmware.html, Ignition config could be defined in Guestinfo. Please correct me for any errors. Thanks a lot!

[crawford]

This needs to be done before the first boot since Ignition only runs once. After setting these options in the VMX, can you run journalctl -b -t ignition --no-pager and /usr/share/oem/bin/vmtoolsd --cmd 'info-get coreos.config.data' (that command is from memory, so it might need to be tweaked). This will tell me if your changes in the VMX took effect and if Ignition recognized them.

[Winnie81]

@crawford
I tested it as below steps:

Install a new VM.
Modify its vmx file, and insert these configurations:
guestinfo.coreos.config.data="ewogICJpZ25pdGlvbiI6IHsgInZlcnNpb24iOiAiMi4wLjAiIH0KfQo="
guestinfo.coreos.config.data.encoding = "base64"
guestinfo.hostname = "WinnieCore1632_ens192"
guestinfo.interface.0.name = "ens192"
guestinfo.interface.0.dhcp = "no"
guestinfo.interface.0.role = "public"
guestinfo.interface.0.ip.0.address = "10.117.16.151/23"
guestinfo.interface.0.route.0.gateway = "10.117.17.253"
guestinfo.interface.0.route.0.destination = "0.0.0.0/0"
guestinfo.dns.server.0 = "10.117.0.2"
Power on vm.
Run "journalctl -b -t ignition --no-pager"
Run "/usr/share/oem/bin/vmtoolsd --cmd 'info-get guestinfo.coreos.config.data' "
At step 3, I found my injected IP address didn't take effect.
At step 4, please refer to attached 1.png and 2.png.
At step 5, I got "ewogICJpZ25pdGlvbiI6IHsgInZlcnNpb24iOiAiMi4wLjAiIH0KfQo="

Please correct me if I got anything wrong. Thanks a lot!

[arithx]

@Winnie81 can you try to set the guestinfo.coreos.config.data via vApp properties and see if Ignition detects the config that way?

If so try disabling the vApp properties and then setting them via the vmx.

[bgilbert]

@Winnie81 From your screenshots, Ignition is successfully reading your config, which means that it's preventing coreos-cloudinit from running, which means that your other guestinfo settings are not being read.

[arithx]

@bgilbert: Isn't that just the base config for the vmware oem?

[bgilbert]

@arithx The first one is. The second one corresponds to the base64 decode of ewogICJpZ25pdGlvbiI6IHsgInZlcnNpb24iOiAiMi4wLjAiIH0KfQo=.

[arithx]

Ah I missed that the user config was just a blank config.

[Winnie81]

@bgilbert Thanks. I knew Ignition has run and I'm wondering how I could pass my network settings to vm through Ignition since this page https://coreos.com/os/docs/latest/booting-on-vmware.html#container-linux-configs said Ignition config could be defined in Guestinfo.
Thanks a lot.

[crawford]

@Winnie81 You can write networkd configs using Ignition. Just take the other options you tried to use (guestinfo.interfaces and guestinfo.dns) and translate them into a .network unit.

[Winnie81]

@crawford Could I do it by modifying vmx file as mentioned above? Referring to your page of https://coreos.com/os/docs/latest/booting-on-vmware.html#container-linux-configs. I would be much appreciated if you could tell me where I was wrong.

[arithx]

@Winnie81 you can indeed modify the vmx file. Ignition only uses the value of the guestinfo.coreos.config.data field, so you'd need to construct an Ignition config that configures the networking, base64 it and include it in that field.

The other guestinfo fields are only read if an Ignition config is not provided, which explains why they're not taking effect here. We recommend configuring networking using an Ignition config instead.

More information about configuring networking in Ignition can be found here https://coreos.com/ignition/docs/latest/network-configuration.html

[snoopsean]

@Winnie81 This is super basic so it doesn't have everything you need, just a password and a ip address and gateway.

Use the below to generate an ignition file.
Once you get that ignition file, copy and paste the contents of it, and base64 encode it.
Then set guestinfo.coreos.config.data to the base64 contents, and set guestinfo.coreos.config.data.encoding to base64 through vmware vcenter.

Everything should be in the ignition file, don't add anything else.

This only works at first boot, if you are trying to do this after the coreos has already booted it wont work.

passwd:
users:
- name: core
password_hash: "$1$O9vCqE/6$YI4xYHr1/gXjs/ciXoj/C/"
networkd:
units:
- name: static.network
contents: |
[Match]
Name=ens192

    [Network]
    Address=10.117.16.149/23
    Gateway=10.117.17.253

[snoopsean]

@crawford @arithx I think the confusion is about ignition only running during first boot. While iginition is set to only run during first boot, the directions at https://coreos.com/os/docs/latest/booting-on-vmware.html seem to indicate that you can edit the vmx file, and then reboot and it will read the ignition config.

In reality, you would have to shut down the machine, edit the vmx file, reboot, somehow trigger ignition to run at next boot, then reboot

[Winnie81]

Thanks, everyone. In the past days, I worked on another task. Currently, Ignition could work in my testing. Thanks for your help.