[vmware_guest] VM won't start with connected Device/Network

[shibumi]

SUMMARY

I have a Ubuntu 18.04 Template with open-vm-tools version 10.2, perl and python installed. When I clone from this template via ansible without using the networks keyword it works like expected and I have a cloned VM with connected Ethernet Device.

But when I specify the networks keyword in the task every device that get spawned will not get connected on boot. I need to connect it manually.

ISSUE TYPE

• Bug Report

COMPONENT NAME

vmware_guest

ANSIBLE VERSION

ansible 2.5.3
  config file = /root/ansible-vxlan/ansible.cfg
  configured module search path = ['/root/ansible-vxlan/library']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.5 (default, May 11 2018, 04:00:52) [GCC 8.1.0]

CONFIGURATION

ANSIBLE_PIPELINING(/root/ansible-vxlan/ansible.cfg) = True
DEFAULT_HOST_LIST(/root/ansible-vxlan/ansible.cfg) = ['/root/ansible-vxlan/hosts']
DEFAULT_MODULE_PATH(/root/ansible-vxlan/ansible.cfg) = ['/root/ansible-vxlan/library']
DEFAULT_ROLES_PATH(/root/ansible-vxlan/ansible.cfg) = ['/root/ansible-vxlan/roles']
DEFAULT_SCP_IF_SSH(/root/ansible-vxlan/ansible.cfg) = True
RETRY_FILES_ENABLED(/root/ansible-vxlan/ansible.cfg) = False

OS / ENVIRONMENT

Host Running Ansible: Arch Linux
Target Host: VMWare ESXI Hosts with Ubuntu 18.04 Template

STEPS TO REPRODUCE

- name: clone from VM templates
  vmware_guest:
    hostname: xxxx.xxxx.xxxx
    username: "{{ vsphere_user }}"
    password: "{{ vsphere_password }}"
    datacenter: xxx.xxx
    esxi_hostname: "{{ item.esxi_hostname }}"
    validate_certs: no
    folder: /xxx
    name: "{{ item.name}}"
    state: poweredon
    template: ubuntu18.04-template
    annotation: "Created via Ansible for VXLAN Test"
    networks:
      - name: "xxx.xxx.xxx.xxx%2f27_vlan1306_vinst6"
        type: static
        ip: "{{ item.ip_address }}"
        netmask: 255.255.255.224
        gateway: xxx.xxx.xxx.126
        start_connected: True
      - name: "VTEPs"
        start_connected: True
  with_items: "{{ infrastructure }}"
  delegate_to: localhost

EXPECTED RESULTS

All added networks should get connected when the VM has booted.

ACTUAL RESULTS

The VM has no all specified network devices, but no device is in the State connected and I need to attach them manually to the VM.

ansible-playbook -vvv playbooks/vxlan.yml

TASK [create_vm : clone from VM templates] ************************************************************************************************************************************************************************************************************************************************************************************
task path: /root/ansible-vxlan/roles/create_vm/tasks/main.yml:3
Using module file /usr/lib/python3.6/site-packages/ansible/modules/cloud/vmware/vmware_guest.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
changed: [localhost -> localhost] => (item={'name': 'vvtep1', 'esxi_hostname': 'esx-test01.xxx.xxx', 'ip_address': 'xxx.xxx.xxx.99'}) => {
    "changed": true,
    "instance": {
        "annotation": "Created via Ansible for VXLAN Test",
        "current_snapshot": null,
        "customvalues": {},
        "guest_consolidation_needed": false,
        "guest_question": null,
        "guest_tools_status": "guestToolsNotRunning",
        "guest_tools_version": "2147483647",
        "hw_cores_per_socket": 1,
        "hw_datastores": [
            "iscsi-nas2"
        ],
        "hw_esxi_host": "esx-test01.xxx.xxx.xxx",
        "hw_eth0": {
            "addresstype": "assigned",
            "ipaddresses": null,
            "label": "Network adapter 1",
            "macaddress": "00:50:56:b0:8f:27",
            "macaddress_dash": "00-50-56-b0-8f-27",
            "summary": "DVSwitch: 41 0b 30 50 3b 31 29 ec-79 b2 ed 5a dc 27 78 e5"
        },
        "hw_eth1": {
            "addresstype": "assigned",
            "ipaddresses": null,
            "label": "Network adapter 2",
            "macaddress": "00:50:56:b0:c4:47",
            "macaddress_dash": "00-50-56-b0-c4-47",
            "summary": "DVSwitch: 05 f8 30 50 04 c2 6a a3-4a 73 e7 45 f0 8f 9c b7"
        },
        "hw_files": [
            "[iscsi-nas2] vvtep1/vvtep1.vmx",
            "[iscsi-nas2] vvtep1/vvtep1.nvram",
            "[iscsi-nas2] vvtep1/vvtep1.vmsd",
            "[iscsi-nas2] vvtep1/vvtep1.vmdk",
            "[iscsi-nas2] vvtep1/vvtep1_1.vmdk"
        ],
        "hw_folder": "/xxx/vm/xxx",
        "hw_guest_full_name": null,
        "hw_guest_ha_state": true,
        "hw_guest_id": null,
        "hw_interfaces": [
            "eth0",
            "eth1"
        ],
        "hw_is_template": false,
        "hw_memtotal_mb": 2048,
        "hw_name": "vvtep1",
        "hw_power_status": "poweredOn",
        "hw_processor_count": 1,
        "hw_product_uuid": "42305acc-7653-0fbe-b655-d35cf1c39b22",
        "instance_uuid": "5030b231-6e9e-4a34-5d3a-131a0f82689f",
        "ipv4": null,
        "ipv6": null,
        "module_hw": true,
        "snapshots": []
    },
    "invocation": {
        "module_args": {
            "annotation": "Created via Ansible for VXLAN Test",
            "cdrom": {},
            "cluster": null,
            "customization": {},
            "customvalues": [],
            "datacenter": "xxx",
            "disk": [],
            "esxi_hostname": "esx-test01.xxx.xxx",
            "folder": "/xxx",
            "force": false,
            "guest_id": null,
            "hardware": {},
            "hostname": "xxx.xxx.xxx.xxx",
            "is_template": false,
            "linked_clone": false,
            "name": "vvtep1",
            "name_match": "first",
            "networks": [
                {
                    "gateway": "xxx.xxx.xxx.126",
                    "ip": "xxx.xxx.xxx.99",
                    "name": "xxx.xxx.xxx.xxx%2f27_vlan1306_vinst6",
                    "netmask": "255.255.255.224",
                    "start_connected": true,
                    "type": "static"
                },
                {
                    "name": "VTEPs",
                    "start_connected": true,
                    "type": "dhcp"
                }
            ],
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "port": 443,
            "resource_pool": null,
            "snapshot_src": null,
            "state": "poweredon",
            "template": "ubuntu18.04-template",
            "username": "xxx",
            "uuid": null,
            "validate_certs": false,
            "wait_for_ip_address": false
        }
    },
    "item": {
        "esxi_hostname": "esx-test01.xxx.xxx.xxx",
        "ip_address": "xxx.xxx.xxx.99",
        "name": "vvtep1"
    }
}

What I see is that the output in verbose mode says there are no guest tools on the VM. But there are definitly guest tools running. The open-vm-tools.service is up and running.. my assumption is that the guest-modules need to long to start and the ansible module doesn't wait so long for setting the OS customization parameters.

Here you can see the running Guest Modules:

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/vmware/vmware_guest.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @Akasurde @dav1x @nerzhul @pdellaert @warthog9
click here for bot help

[Akasurde]

@shibumi Thanks for reporting this issue. Could you please paste your esxi version ?

[shibumi]

@Akasurde sure. The vsphere Server version is 6.0.0

[cigamit]

Can you check the open-vm-tools log file to see what is going on? The module isn't waiting for tools to do customization. We send the entire customization spec with the initial request, VMWare does the waiting.

[shibumi]

@cigamit Here are the logs.. sorry for the screenshots instead of text pastes. But was the easiest way to get the information without having to setup network manually.

vmware-vmvsc.log
https://paste.xinu.at/siJprp/

vmware-network.log
https://paste.xinu.at/siJprp/

I don't understand why this is a problem at all.. shouldn't set the VMWare host the NIC on 'connected'. Why does the VM need to do this via VMWareTools?

EDIT: I have tried executing this script manually /etc/vmware/tools/scripts/vmware/network poweron-vm and if this script is supposed to 'connect' the NICs to the VM.. it doesn't do this, but it also throws no error.

[shibumi]

I guess this bug report here is related to this issue: vmware/open-vm-tools#240

[shibumi]

Note: Connecting the NICs works fine with official vmware-tools. So seems like this is an issue that the open-vm-tools people need to solve.

Do we want to leave this issue open for future bug reports in this direction or do we close it? :)

[cigamit]

I would assume close it, since it will still come up via a search.

[coom]

Same issue here with Ubuntu 16.04 which is supported by VMWare vSphere 6.5.

[shibumi]

@coom please help reporting it here: vmware/open-vm-tools#240

[ZouYuhua]

@shibumi
I am trying to reproduce this issue using ansible. As I know, Ubuntu 18.04 have different images:
ubuntu-18.04-server-cloudimg-amd64.ova
ubuntu-18.04-desktop-amd64.iso
ubuntu-18.04-server-amd64.iso
ubuntu-18.04-live-server-amd64.iso

I am not sure which image do you use to create VM template ?

Now I reproduce this issue as the following steps:

1. deploy a VM with ubuntu-18.04-server-cloudimg-amd64.ova
2. check the VM and find the VM have open-vm-tools version 10.2, perl and python installed by default
3. Choose the VM and clone it to a template with name “ubuntu18.04_cloudimage_template"
4. use the ansible playbook (with or without keyword networks) to clone a VM from this template ubuntu18.04_cloudimage_template
5. check the NIC connection status via vcenter webui / ESXi webui

Anything wrong with steps ? Thanks.

Best regards

[shibumi]

@ZouYuhua nothing wrong. But as I said: This issue is clearly related to open-vm-tools.

[ZouYuhua]

@shibumi Yes. This issue is related to open-vm-tools. I can reproduce it and the developer will do analysis for it.

[PengpengSun]

@shibumi
I ping you at vmware/open-vm-tools#240, I need check toolsDeployPkg.log to see what's wrong on guest customization process.

[PengpengSun]

@coom
Same here, could you please share with me the /var/log/vmware-imc/toolsDeployPkg.log on customized Ubuntu16.04 VM.
And which types of VMTools is installed, OVT or official VMTools, and the VMTools version is?

[ansibot]

cc @ckotte
click here for bot help

[mrmeszaros]

A possible solution might be to ensure the open-vm-tools.service is started after dbus.service.
Edit /lib/systemd/system/open-vm-tools.service and insert a new line After=dbus.service under the [Unit] section.

Source: vmware/open-vm-tools#240 (comment)

[spaced]

Had the same issue with ansible 2.3.2 using centos 7 1810 image.
Problem found on the guest side:
Missing perl package crashes the scripts for the customization guest pkgs.
You may also have look in /var/log/vmware-incs/* logs after provisioning.

[ansibot]

cc @lparkes
click here for bot help

[ansibot]

cc @Tomorrow9
click here for bot help

[Tomorrow9]

It seems deploy from Windows template has the similar issue: #45834 and CentOS 7 #40564

[Tomorrow9]

@mrmeszaros，thanks for the reply and the workaround info.
So (1) if you hit the same issue on Ubuntu 18.04 with the open-vm-tools, then there is workaround as below, and install official vmware tools.
"
A possible solution might be to ensure the open-vm-tools.service is started after dbus.service.
Edit /lib/systemd/system/open-vm-tools.service and insert a new line After=dbus.service under the [Unit] section.
Source: vmware/open-vm-tools#240 (comment)"
@spaced
(2) if you hit "Perl" issue on Ubuntu or CentOS, please try to re-install perl again to get the latest and some missing modules required.

Thanks.

[ansibot]

cc @goneri
click here for bot help

[ansibot]

cc @pgbidkar
click here for bot help

[ottacom]

I tried every single trick which I found, I spent some hours a the end I fixed the problem using the Vmware tools (and not the open source as recommended) everything works for me and is faster than before.
I'm still wondering why it is recommended to use the OS version when is not possible to update the VMware tool from the GUI.

• name: "{{ vlan_1 }}"
  lablel: "{{ vlan_1 }}"
  state: present
  connected: true
  device_type: vmxnet3
  start_connected: true
  ip: '{{ vm_ipaddress }}'
  netmask: '{{ vm_netmask }}'
  gateway: '{{ vm_gateway }}'
  dns_servers: '{{ vm_dns1 }} {{ vm_dns2 }} '
  dvswitch_name: '{{dvswitch_name}}'

[cigamit]

Most people recommend the OS version because you end up updating it more often. Every week when my Servers install the latest patches, they would get the new version automatically. Versus back in the day when I used the VMWare Tools version, and I pretty much never updated them because it took a 1000 clicks in the GUI for all my servers.

Of course, in the ideal automated VMWare environment, you shouldn't be using the GUI to update VMWare Tools, you should write a playbook to do it instead also.

[ansibot]

Files identified in the description:

• test/support/integration/plugins/modules/vmware_guest.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @mattclay
click here for bot help

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/vmware/vmware_guest.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

Thank you very much for your interest in Ansible. Ansible has migrated much of the content into separate repositories to allow for more rapid, independent development. We are closing this issue/PR because this content has been moved to one or more collection repositories.

• lib/ansible/modules/cloud/vmware/vmware_guest.py -> https://galaxy.ansible.com/community/vmware

For further information, please see:
https://github.com/ansible/ansibullbot/blob/master/docs/collection_migration.md