Salt-cloud vmware provisiong fails, not connecting NIC

[Securitybits-io]

Description of Issue/Question

When deploying a VM from a template using the Vcenter provider, the VM gets created, but provisioning wont continue until the interface is brought up. Salt-cloud does not bring up the interface using open-vm-tools, so the VM will never recieve a IP Address.

What ive tried:

1. remove NIC from template, to get one with proper IP address - FAIL
2. have a default NIC, which is onnected at power on - Fail
3. like two, but have Salt create a second with correct MAC - Fail
4. 1-3 with both VMware and OpenVM tools 6x-fail
5. 1 but with a Centos 7 - Fail
6. Doing 1. but when salt-cloud is waiting on IPv4, do a manual connect on the NIC (edit settings > connnect) - Works

I have seen some issues regarding this on other Github posts here, but none specify a solution for the problem.

Theres a load of text in this issue, trying to stay ahead of the debug curve. thank you for reading through it

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)
provider for vCenter 6.7

sb-vcenter-01:
  driver: vmware
  user: 'user'
  password: 'pass'
  url: 'url'
  protocol: 'https'
  port: 443

So thats really nothing special.
The template is a Ubuntu server 18.04.3 LTS with Open-VM-Tools installed. For networking purposes i removed the network card. as i need it to have one specific network card with a specific MAC for the DHCP server to run with. Hence they are commented out. more to follow (Check the map)

vm-profile.conf
telegraf:
  provider: sb-vcenter-01
  clonefrom: VRTX_ubuntu-18.04_32GB_Template

  ## Optional arguments
  num_cpus: 1
  memory: 2GB
  #devices:
  #  network:
  #    Network adapter 1:
  #      name: vDPG-40-Securitybits.Private
  #      switch_type: distributed
  #      connected: true

  cluster: VRTX.Prod

  datastore: Storage.VRTX.Prod
  datacenter: 'Datacenter'
  host: host
  power_on: True
  extra_config:
    mem.hotadd: 'yes'
  annotation: Created by Salt-Cloud

  deploy: True
  minion:
    master: salt.securitybits.local

Here comes the map that i am using to create this specific VM

influx-mac.map
telegraf:
  - sb-telegraf-01:
      devices:
        network:
          Network Adapter 1:
            name: vDPG-40-Securitybits.Private
            switch_type: distributed
            mac: '00:50:56:b9:19:89'
#            connected: true   #Is this even supposed to do anything?

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)
salt-cloud -m influx-mac.map -l debug
Wall of text:

Proceed? [N/y] y
... proceeding
[INFO    ] Calculating dependencies for sb-telegraf-01
[DEBUG   ] Got execution order 0 for sb-telegraf-01
[DEBUG   ] No make_master found in map
[DEBUG   ] Generating minion keys for 'sb-telegraf-01'
[DEBUG   ] LazyLoaded cloud.fire_event
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/cloud/sb-telegraf-01/creating; data = {u'profile': u'telegraf', u'_stamp': '2019-03-14T22:13:21.908502', u'name': u'sb-telegraf-01', u'driver': u'vmware', u'provider': u'sb-vcenter-01:vmware', u'event': u'starting create'}
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Setting cpu to: 1
[DEBUG   ] Setting memory to: 2048 MB
[DEBUG   ] Networks adapters to create: [u'Network Adapter 1']
[DEBUG   ] The type of 'Network Adapter 1' has not been specified. Creating default type 'vmxnet3'
[DEBUG   ] clone_spec set to:
(vim.vm.CloneSpec) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   location = (vim.vm.RelocateSpec) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      service = <unset>,
      folder = <unset>,
      datastore = 'vim.Datastore:datastore-789',
      diskMoveType = <unset>,
      pool = 'vim.ResourcePool:resgroup-781',
      host = 'vim.HostSystem:host-782',
      disk = (vim.vm.RelocateSpec.DiskLocator) [],
      transform = <unset>,
      deviceChange = (vim.vm.device.VirtualDeviceSpec) [],
      profile = (vim.vm.ProfileSpec) []
   },
   template = false,
   config = (vim.vm.ConfigSpec) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      changeVersion = <unset>,
      name = <unset>,
      version = <unset>,
      createDate = <unset>,
      uuid = <unset>,
      instanceUuid = <unset>,
      npivNodeWorldWideName = (long) [],
      npivPortWorldWideName = (long) [],
      npivWorldWideNameType = <unset>,
      npivDesiredNodeWwns = <unset>,
      npivDesiredPortWwns = <unset>,
      npivTemporaryDisabled = <unset>,
      npivOnNonRdmDisks = <unset>,
      npivWorldWideNameOp = <unset>,
      locationId = <unset>,
      guestId = <unset>,
      alternateGuestName = <unset>,
      annotation = u'Created by Salt-Cloud',
      files = <unset>,
      tools = <unset>,
      flags = <unset>,
      consolePreferences = <unset>,
      powerOpInfo = <unset>,
      numCPUs = 1,
      numCoresPerSocket = <unset>,
      memoryMB = 2048,
      memoryHotAddEnabled = <unset>,
      cpuHotAddEnabled = <unset>,
      cpuHotRemoveEnabled = <unset>,
      virtualICH7MPresent = <unset>,
      virtualSMCPresent = <unset>,
      deviceChange = (vim.vm.device.VirtualDeviceSpec) [
         (vim.vm.device.VirtualDeviceSpec) {
            dynamicType = <unset>,
            dynamicProperty = (vmodl.DynamicProperty) [],
            operation = 'add',
            fileOperation = <unset>,
            device = (vim.vm.device.VirtualVmxnet3) {
               dynamicType = <unset>,
               dynamicProperty = (vmodl.DynamicProperty) [],
               key = -4080,
               deviceInfo = (vim.Description) {
                  dynamicType = <unset>,
                  dynamicProperty = (vmodl.DynamicProperty) [],
                  label = u'Network Adapter 1',
                  summary = u'vDPG-40-Securitybits.Private'
               },
               backing = (vim.vm.device.VirtualEthernetCard.DistributedVirtualPortBackingInfo) {
                  dynamicType = <unset>,
                  dynamicProperty = (vmodl.DynamicProperty) [],
                  port = (vim.dvs.PortConnection) {
                     dynamicType = <unset>,
                     dynamicProperty = (vmodl.DynamicProperty) [],
                     switchUuid = '50 39 3e bb 65 e8 1f 4f-5c fb 51 10 2c 6c ab 6d',
                     portgroupKey = 'dvportgroup-51',
                     portKey = <unset>,
                     connectionCookie = <unset>
                  }
               },
               connectable = (vim.vm.device.VirtualDevice.ConnectInfo) {
                  dynamicType = <unset>,
                  dynamicProperty = (vmodl.DynamicProperty) [],
                  migrateConnect = <unset>,
                  startConnected = true,
                  allowGuestControl = true,
                  connected = false,
                  status = <unset>
               },
               slotInfo = <unset>,
               controllerKey = <unset>,
               unitNumber = <unset>,
               addressType = u'assigned',
               macAddress = u'00:50:56:b9:19:89',
               wakeOnLanEnabled = true,
               resourceAllocation = <unset>,
               externalId = <unset>,
               uptCompatibilityEnabled = <unset>
            },
            profile = (vim.vm.ProfileSpec) [],
            backing = <unset>
         }
      ],
      cpuAllocation = <unset>,
      memoryAllocation = <unset>,
      latencySensitivity = <unset>,
      cpuAffinity = <unset>,
      memoryAffinity = <unset>,
      networkShaper = <unset>,
      cpuFeatureMask = (vim.vm.ConfigSpec.CpuIdInfoSpec) [],
      extraConfig = (vim.option.OptionValue) [
         (vim.option.OptionValue) {
            dynamicType = <unset>,
            dynamicProperty = (vmodl.DynamicProperty) [],
            key = u'mem.hotadd',
            value = u'yes'
         }
      ],
      swapPlacement = <unset>,
      bootOptions = <unset>,
      vAppConfig = <unset>,
      ftInfo = <unset>,
      repConfig = <unset>,
      vAppConfigRemoved = <unset>,
      vAssertsEnabled = <unset>,
      changeTrackingEnabled = <unset>,
      firmware = <unset>,
      maxMksConnections = <unset>,
      guestAutoLockEnabled = <unset>,
      managedBy = <unset>,
      memoryReservationLockedToMax = <unset>,
      nestedHVEnabled = <unset>,
      vPMCEnabled = <unset>,
      scheduledHardwareUpgradeInfo = <unset>,
      vmProfile = (vim.vm.ProfileSpec) [],
      messageBusTunnelEnabled = <unset>,
      crypto = <unset>,
      migrateEncryption = <unset>
   },
   customization = (vim.vm.customization.Specification) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      options = <unset>,
      identity = (vim.vm.customization.LinuxPrep) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         hostName = (vim.vm.customization.FixedName) {
            dynamicType = <unset>,
            dynamicProperty = (vmodl.DynamicProperty) [],
            name = u'sb-telegraf-01'
         },
         domain = u'local',
         timeZone = <unset>,
         hwClockUTC = <unset>
      },
      globalIPSettings = (vim.vm.customization.GlobalIPSettings) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         dnsSuffixList = (str) [],
         dnsServerList = (str) []
      },
      nicSettingMap = (vim.vm.customization.AdapterMapping) [
         (vim.vm.customization.AdapterMapping) {
            dynamicType = <unset>,
            dynamicProperty = (vmodl.DynamicProperty) [],
            macAddress = <unset>,
            adapter = (vim.vm.customization.IPSettings) {
               dynamicType = <unset>,
               dynamicProperty = (vmodl.DynamicProperty) [],
               ip = (vim.vm.customization.DhcpIpGenerator) {
                  dynamicType = <unset>,
                  dynamicProperty = (vmodl.DynamicProperty) []
               },
               subnetMask = <unset>,
               gateway = (str) [],
               ipV6Spec = <unset>,
               dnsServerList = (str) [],
               dnsDomain = <unset>,
               primaryWINS = <unset>,
               secondaryWINS = <unset>,
               netBIOS = <unset>
            }
         }
      ],
      encryptionKey = (byte) []
   },
   powerOn = true,
   snapshot = <unset>,
   memory = <unset>
}
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
<<<<<<<<<<<<<<<<<<<<<<<<<<<KEY GENERATION>>>>>>>>>>>>>>>>>>>>>>>>>>>
u'protocol': u'https', u'minion': {u'master': u'salt.securitybits.local'}, u'cluster': u'VRTX.Prod', u'datastore': u'Storage.VRTX.Prod', u'port': 443, u'num_cpus': 1, u'power_on': True, u'memory': u'2GB', u'extra_config': {u'mem.hotadd': u'yes'}, u'profile': u'telegraf', u'deploy': True, u'provider': u'sb-vcenter-01:vmware', u'driver': u'vmware', u'host': u'eldir-01.securitybits.local', u'user': u'securitybits.local\\christoffer', u'inline_script': None, u'annotation': u'Created by Salt-Cloud', u'datacenter': u'Huskvarna Datacenter', u'name': u'sb-telegraf-01', u'clonefrom': u'VRTX_ubuntu-18.04_32GB_Template', u'level': 0, u'url': u'sb-vcenter-01.securitybits.local', u'devices': OrderedDict([(u'network', OrderedDict([(u'Network Adapter 1', OrderedDict([(u'name', u'vDPG-40-Securitybits.Private'), (u'switch_type', u'distributed'), (u'mac', u'00:50:56:b9:19:89')]))]))]), u'pub_key': u'-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAoMYB0tHHx7nODxrJlWXs\nO+HqZNaaHDi/wbCgWltsRPYGJpyuI4bMCH+t25T347mbytSKyy94lkw4HIGlFsBo\n6QduNGa7auvG6OXmnK1fJmq41Bbj8EtvqN3zjpPFUum2vDUkNMs1iryHSV2Bt+W7\nrplVs0ithLSFFgsk3jR78MnOVd5bArGIcTzkmPbwGZAScAtxRSJG3U7WvBAxrubS\n+Rd51JOR25ho2JySLzsJLHm8iY5folctBuXQ2KoGxKcgJLkLWN5jFbjVQACicFm5\nzZY+on5v7klADJfCommvafHgvq7AclnNLQyMd83oijZVqEcchaaY2I6WAnTgkcQv\nLVODVmcFWwLtITlB27qbvm35sFfO9AdkwcmDGAJq954s5sLvYULj8CPvwI+cr+tX\n8EP27tqya6Z9CmCFiCi87HilWXDCMM20XvQye1DABXBVCwvy1ISeP242Enf77nm9\nDpfUw4RI068iK8rOidoX/XPRl+AUG7bh2TzpdX5Y//aqhW2uieHHr282PWSTclSs\nv7mihNisnJz+F4RIpHxtsQ34gNVNwYbnzhHoqzP/2K6iJyec9asG877RU6EToPi8\nPhjAPqulCsrWlc3g/FLGXB7WXKzuVwTs9isPgAAdJE4/kbu9cb4meWHJjTAqkjdS\n1sjiJmR4hKZP4o/ohXRumQMCAwEAAQ==\n-----END PUBLIC KEY-----', u'os': u'bootstrap-salt'}
[DEBUG   ] Closing IPCMessageClient instance
[INFO    ] Creating sb-telegraf-01 from vm(VRTX_ubuntu-18.04_32GB_Template)
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [0 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [5 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [10 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [15 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [20 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [25 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [30 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [35 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [40 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [45 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [50 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [55 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for clone task to finish [60 s]
[INFO    ] [ sb-telegraf-01 ] Successfully completed clone task in 62 seconds
[INFO    ] [ sb-telegraf-01 ] Waiting for VMware tools to be running [0 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for VMware tools to be running [5 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for VMware tools to be running [10 s]
[INFO    ] [ sb-telegraf-01 ] Waiting for VMware tools to be running [15 s]
[INFO    ] [ sb-telegraf-01 ] Successfully got VMware tools running on the guest in 18 seconds
[INFO    ] [ sb-telegraf-01 ] Waiting to retrieve IPv4 information [0 s]
[INFO    ] [ sb-telegraf-01 ] Waiting to retrieve IPv4 information [5 s]
[INFO    ] [ sb-telegraf-01 ] Waiting to retrieve IPv4 information [10 s]
[INFO    ] [ sb-telegraf-01 ] Waiting to retrieve IPv4 information [15 s]
[INFO    ] [ sb-telegraf-01 ] Waiting to retrieve IPv4 information [20 s]
[INFO    ] [ sb-telegraf-01 ] Waiting to retrieve IPv4 information [25 s]

And there it sits for about a half eternity before giving up.
This has been tried with both Open-VM-Tools and VMware-tools from the vCenter server itself. Both givesthe same result.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

root@salt:~# salt --versions-report
Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.4
      gitpython: 2.1.11
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Oct  2 2018, 22:12:08)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 17.1.0
           RAET: Not Installed
          smmap: 2.0.4
        timelib: Not Installed
        Tornado: 5.0.2
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.10 cosmic
         locale: UTF-8
        machine: x86_64
        release: 4.18.0-15-generic
         system: Linux
        version: Ubuntu 18.10 cosmic

[Securitybits-io]

Looking through the VMWare.py Connector i can se that there are alot of checks to idetntify whether the NIC is connected or not, but doing any action on the status. maybe a feature request that have a function in vmware.py which will connect a NIC if it is not connected?
https://puu.sh/D0fYR/704f6392da.png

[Securitybits-io]

Sooooo i did alot of digging around!
https://kb.vmware.com/s/article/54986
what you have to do when running Ubuntu 18.04.2(3) is that they removed ifup and started using netplan and cloud-init. and the customization does occur long beofre dbus service starts, hence it cannot control VM hardware.

What worked for me

1. apt purge cloud-init -y #Dont want to see that one again!
2. comment out "/tmp 1777 root root -" from /usr/lib/tmpfiles.d/tmp.conf
3. apt install open-vm-tools
4. add "After=dbus.service" under [unit] in file /lib/systemd/system/open-vm-tools.service

[priyank-R]

@Securitybits-io Is this resolution specific for Ubuntu VMs or does this apply to Centos 7 as well ? Any ideas