no IP address assigned to new cloned 18.04 template

[j4y4r]

Hello,

i have an issue when deploying ubuntu18.04 Preview on an esxi version 6.0.

open-vm-tools version: 2:10.2.0-3ubuntu2

After creating a new VM from a Template with pyvmomi, the machine does not get an IP address assigned, also in the VM properties->Network the options "Connect" and "Connect at power on" are not checked.

The file /etc/network/interfaces still exists on the system, but there was nothing written into the file.

I also tried to create a new Vm from a Template via the vSphere Client with the same result.

Ubuntu changed their default Networking to netplan with 17.10 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2017-June/001215.html.

If you need any further infomation, logfiles, etc. please let me know.

[hangrao]

Thanks for the info.
Netplan is not supported in customization yet, it will be addressed in future vSphere releases.

Please see the KB for details:
https://ikb.vmware.com/s/article/52646

[j4y4r]

Thanks for your answer and the link, i guess we have to wait for vmware then.

[saschalucas]

even if you remove netplan and install legacy ifup, guest customisation does not work (see /var/log/vmware-imc/toolsDeployPkg.log at https://gist.github.com/saschalucas/5595235e6ff41bc4a4ea2bac845bd2dc).

somehow it trys cloud-init but gave up later (don't know why).

[pengzhencao]

We do have several internal PRs tracking customization failure for ubuntu18.04.
error code 2 may be permission issue.
Could you also show me the "vmware*.log" inside the VM's folder on ESXi server?
(usually /vmfs/volumes/DATASTORE_NAME/VM_NAME/vmware*.log")
Could you also show me the open-vm-tools build number? Run "vmtoolsd -v" inside the VM could get this info.

Regarding the cloud-init customization option, there are two things must be done before using cloud-init based guest customization,

1. Cloud-init package is installed properly(You have done this, this is what we see the "cloud-init 18.1" in log)
2. "DISABLE_VMWARE_CUSTOMIZATION: false" set in "/etc/cloud/cloud.cfg" (you did not do this, this is why it fallback to traditional customization)
   ref: https://bazaar.launchpad.net/~baoli/cloud-init/cloud-init/revision/1173

[shibumi]

Hello everybody,
Is there a way to get this work? I really need this feature for my deployment. Would be nice to have a workaround. Maybe official vmware-tools?

EDIT: Ok seems like the official vmware-tools work with ubuntu18.04 (at least connecting the NIC work.. I am doing the rest via ansible)

[oliverkurth]

@shibumi , did you provide the additional inormation requested by Peter, see #240 (comment) ?

[pengzhencao]

@shibumi , please try the following workaround to use customization with cloud-init:

1. #apt get install cloud-init ( I think you've done this)
2. Add ‘disable_vmware_customization: false’ to /etc/cloud/cloud.cfg
3. Do the following to workaround tmp dir issue:
   1). Edit tmp.conf under /usr/lib/tmpfiles.d/
   2). comment out line 11 by adding '#' in front of this line.
   ex: #D /tmp 1777 root root -

Then you could try to run customization again.

[saschalucas]

Hi,

@pengzhencao here is the requested log (somehow anonymised):
vmware.log

the open-vm-tools is from stock ubuntu package:

vmtoolsd -v VMware Tools daemon, version 10.2.0.1608 (build-7253323)

I'll give cloud-init a try.

Thanks, Sascha.

[shibumi]

@oliverkurth @pengzhencao sorry maybe I expressed myself wrongly.. I am not interested in cloud-init configuration. In my case ansible does this via pyvmomi (the ESXI API) and the esxi server handles this via the running guest agent. What I want/need is a working open-vm-tools for Ubuntu 18.04

The official vm-tools are working. There the NICs get connected via the guest agent.. on ubuntu 18.04 and open-vm-tools this doesn't work.

[AndrewSav]

@shibumi wrong thread then?

[shibumi]

@AndrewSav no idea why you think this is the wrong thread. My case is exactly the one that the thread opener has. No idea why everybody started talking about cloud-init support for vmware suddenly in this thread.

The main problem is the same. open-vm-tools doesn't seem to connect NICs on Ubuntu 18.04. (Doesn't matter which tool you run above open-vm-tools. If pyvmomi, ansible-vmware or cloud-init. They all use open-vm-tools to get the NICs automatically connected. And this doesn't seem to work in ubuntu 18.04)

[hangrao]

@shibumi Which network controller are you using in your VM? As I know, the driver of the nics doesn't depend on OpenVMTools, all drivers (for e1000e, vmxnet3. ) are all included in the Linux kernel. (e1000 is not supported)

You could try different nic when you create the VM and see how it behave. In my understanding even there's no OVT installed the nics should work.

[shibumi]

@hangrao I use vmxnet3. The NIC works but they doesn't come up automatically and get connected via OVT. When I manually click on connect in the GUI of vmware center everything works fine.

[jord-bh]

I am hitting the same error as @shibumi through ansible, deploying a centos image works and brings up the NIC correctly, but starting an ubuntu image doesn't. I've tried to force call the object and setting connectable.connected = True to no effect. Doing it manually in GUI makes it connected.

Unsure if related or not, but I notice they have different VMWare Tools version installed (ubuntu: 10304, centos: 10282)

Also tried the workaround regarding cloud-init approach to no avail.

[hangrao]

@jord-bh @shibumi We tried to install ubuntu 18.04 from iso, and tried deploy ovf using ovftool or vSphere web client, either way is working fine. The Nic can go up successfully and the VM can get ip address.

Could you capture a screen shot with the error and add the reproducing steps there? I'd like to try that further.

[shibumi]

@hangrao See also this Bug Issue for Ansible: ansible/ansible#41133
Which version did of ovf did you use? I use the one in the Ubuntu 18.04 Repository.

And what we talk about is getting the nic up automatically via Vmware API (that uses OVF) not via manually clicking on things.

[hangrao]

@shibumi Thanks for the clarification, will have a try using steps you described in ansible/ansible#41133.

[jwalkins2017]

I am seeing the same issue when using salt-cloud to clone a vm. The clone's network adapter starts disconnected

[jord-bh]

@hangrao any luck reproducing this yet?

[hangrao]

@jord-bh We're able to reproduce this issue, and have already raised internal ticket to track it. Will update this thread if there's any progress.

[jstoja]

@pengzhencao Modifying the template to use cloud-init + setting to false the disable_vmware_customization worked for me. Waiting for any update to make it work straight away though!

[pengzhencao]

@jstoja Glad to know it could work.
That option is not totally controlled by VMware. Cloud-init developers disabled vmware customization by default, as they do not want to check for customization every OS boot due to performance concern.
refer:
https://git.launchpad.net/cloud-init/commit/?id=05afe04edbe4c28f2170194d226821c1e755ee2d

[shibumi]

@jstoja For me this even doesn't work when I have no cloudinit installed. So it's definitly something with Ubuntu 18.04

[jstoja]

@shibumi @pengzhencao I'll rebuild everything from scratch and try again to double check. If I get something working, I'll share it here.

[shibumi]

@jstoja did you install deprecated network-ifup scripts on that host?

[jstoja]

Humm I tested a dozen times with the exact same template each time and it worked twice without even doing something special... I didn't find something interesting in the logs... I tried with the legacy network-ifup and it doesn't help.

[shibumi]

@jstoja which esxi version? which template? which distribution and distribution version? was the template modified? if so, what has been modified? No idea why it works for you. Fact is for a lot of people it doesn't work :)

[jstoja]

Like I said in my last message, I relaunched it a dozen of times and it worked only twice... No idea why yet.
I'm trying with:
ESXI 6.0
Ubuntu 18.04 (https://github.com/heizo/packer-ubuntu-18.04 exported to https://gist.github.com/jstoja/544aac68ab2e8f142ad94ab9985aac5d)
I modified the template just by replacing the Network to one that is present into vSphere, but also the virtualbox-2.2 to vmx-07 value.

I've tried manually and it didn't work either, so I'm trying to find why did it work very rarely... The thing is that it's not only about the NIC only, nothing is customized, not even the hostname, so it's pretty easy to check.

[PengpengSun]

@bitmand Yes, it's the same issue. The root cause is dbus.service is not running before guest customization happens, this is not always happening, depends on VM's cpu/memory numbers.
There is a workaround on this issue:
Add dependence to open-vm-tools.service by inserting a new line "After=dbus.service" under [Unit] of /lib/systemd/system/open-vm-tools.service file.

[bitmand]

@PengpengSun I added the line to open-vm-tools.service to make sure dbus service is started to the template.

Then created 5 VMs and all worked as expected - thank you very much for workaround, highly appreciated :)

[shibumi]

@PengpengSun I will try it tomorrow, when I am at work again.

[gaui]

@PengpengSun THANK YOU !!!!!

[gruffled]

We are also experiencing this exact issue on Ubuntu 16.04.

[PengpengSun]

@idoru78 The dbus issue should only reproduce on Ubuntu18.04, but I need know VMTools version on your Ubuntu 16.04 VM, and could you please share the /var/log/vmware-imc/toolsDeployPkg.log file.

[gruffled]

Sure.

Edit: Oops first one was from 18.04...

The log appears differently but the symptom is the same in that the NIC starts in a disconnected state.
log16.04.txt

We have worked around it for now my keep the open-vm-tools package at v10.0.7 and not upgrading to 10.2.

[PengpengSun]

From the log, it says the VM is Ubuntu18.04 but 16.04, please try the workaround in my previous comment at Aug 15.

DEBUG: Command: 'cat /etc/lsb-release'
DEBUG: Result: DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

[getSurreal]

VMware KB that was just posted as a workaround solved it for me

[maxim-hansen]

Hi!

Could we get a clarification on whether or not netplan is supported during customization using open-vm-tools?

My understanding is that Ubuntu 18.04, including the configuration of netplan during the guest os customization is supported on vSphere 6.7 and 6.5U2.

We've tried this now with both cloud-init and the "traditional GOSC" workflows, but with no luck on getting the network configured on 18.04.

• cloud-init workflow
  This one we couldn't get working at all, since nothing happened after it copied over the files to /var/run/vmware-imc. It just logged the that "Deployment delegated to Cloud-init. Returning success.", and then proceeds to exit the customization.
  Looking through the customization code, I don't really understand how the cloud-init workflow is supposed to even work, since there is nothing that triggers cloud-init init, or any other command, once the files have been copied over.

• traditional GOSC workflow
  This one works, but not when it comes to configuring the network interface, since it just customizes the /etc/network/interfaces file during the network config stage of the workflow, which of course doesn't work on 18.04 unless we re-install ifupdown.

Environment Info:
vCenter: 6.5.0 Build 5973321
ESXi: 6.5.0 Build 8294253
Guest OS: Ubuntu Server 18.04.1
open-vm-tools: 2:10.3.0-0ubuntu1~18.04.2

Deployment logs:
toolsDeployPkg.log

[PengpengSun]

@maximhan Your understanding is correct, netplan supporting is available on 6.5U2, while the fix is on vCenter side, you need a 6.5U2 vCenter, your ESXi build is good.
Your current vCenter is 6.5U1, I double-checked toolsDeployPkg.log, the netplan fix is not there.
Please upgrade your vCenter to 6.5U2+ and have a try.

Thanks,
Pengpeng

[AndrewSav]

Works fine here on vCenter 6.7

[BrightVictoria]

Hi guys,
I have a similar error... no IP address assigned to new cloned 18.04 template, but host name and dns suffix assign without problems.
The error: Customization command failed: Could not set property: Access $ after the string DEBUG: Removing lock file /var/lock/vmware/gosc in the /var/log/vmware-imc/toolsDeployPkg.log file

ESXi 6.7.0
Ubuntu 18.04
open-vm-tools 2:10.3.0-0ubuntu1

[hangrao]

Hi guys,
I have a similar error... no IP address assigned to new cloned 18.04 template, but host name and dns suffix assign without problems.
The error: Customization command failed: Could not set property: Access $ after the string DEBUG: Removing lock file /var/lock/vmware/gosc in the /var/log/vmware-imc/toolsDeployPkg.log file

ESXi 6.7.0
Ubuntu 18.04
open-vm-tools 2:10.3.0-0ubuntu1

Hi BrightVictoria,

Could you check if you have already follow the steps in this KB:
https://kb.vmware.com/s/article/54986

Best regards,
Hang Rao

[mrlesmithjr]

Just adding my .02 here for reference. I am using vCenter 6.7 to start with. However, my scenario might be a bit different so let me explain.

The below involves Ubuntu 18.04 server w/out cloud-init.

I am building OVA images using Packer using the vsphere-iso builder. As part of the process I am having Packer remove the network interfaces as we may need to define different types of network adapter during provisioning. This OVA is uploaded to vCenter using Ansible as a new VM, remaining powered off. Then this VM is converted to a template via Ansible as well. When provisioning a VM the network adapter type is defined in which case the VM is spun up appropriately with the defined adapter type. The OS sees the adapter but the customization is failing during the netplan configuration.

Update
After following the recommendation above which lead to this KB https://kb.vmware.com/s/article/56409, I still experienced the same issue. However, switching from a vmxnet2 adapter type to a vmxnet3, the issue is not experienced. Is this a known thing with vmxnet2 adapters by chance?

Update
After looking at the VMware compatibility guide, it appears that vmxnet2 is not supported on Ubuntu 18.04 but was on 16.04.

[hangrao]

Update
After looking at the VMware compatibility guide, it appears that vmxnet2 is not supported on Ubuntu 18.04 but was on 16.04.

Right, vmxnet2 has been superseded by vmxnet3, please use vmxnet3 or other supported nics.

[spaced]

had a similar issue using centos 7 1810 image.
Problem found on the guest side:
Missing perl package crashes the scripts for the customization guest pkgs.
You may also have look in /var/log/vmware-incs/* logs after provisioning.

[PengpengSun]

Hi @spaced ,

perl package is necessary for guest customization to work, please include 'perl' at OS installation or install 'perl' package before do guest customization.

[rthimons85]

On ubuntu 18.04 cloud images, I found that /etc/netplan/50-cloud-init.yaml was conflicting with the vmware customizations.
Disabling it by deleting 50-cloud-init.yaml and creating a file named /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg and putting in

network: 
  config: disabled

fixed the issue, and my customizations completed properly.

[PengpengSun]

Thanks @rthimons85, Ubuntu 18.04 cloud image should have cloud-init installed according to the /etc/netplan/50-cloud-init.yaml file exits.
For Ubuntu 18.04 Live server image, it has cloud-init installed too. There is a KB article on disabling cloud-init if using perl based VMware customization.

[AndrewSav]

This is now closed as invalid over at LaunchPad Ubuntu.

[mylesagray]

Posted a blog on how to use cloud-init for full setup on vSphere: https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/

This took me days to figure out, but it hinged on a few things (covered in the Troubleshooting section of the blog) - but essentially, do NOT change the open-vm-tools system unit file if you want to use cloud-init.

Also, it seemed to matter that in the OVA spec, a hostname was not set for the initial template VM, and so it used the default.

Clearing /etc/machine-id is also essential for DHCP leases to not conflict as netplan uses this instead of MAC address for uniqueness.

[dasmokedog]

If you are running VIO (VMware Integrated Openstack) and smashing your head against the wall trying to get cloud-init to identify the datasource, like me, then you will need to ensure vmtools (or open-vm-tools) runs and registers the IP with SpoofGuard. Until this happens all traffic will be blocked. The quickest way is to arm cloud-init to run on 2nd boot.

[vmwarelab]

@dasmokedog how do you configure cloud-init to run on 2nd boot instead of first boot?

[dasmokedog]

@dasmokedog how do you configure cloud-init to run on 2nd boot instead of first boot?

@vmwarelab Initially I approached it by adding /etc/cloud/cloud-init.disabled at the end of the Packer run before it creates the image. I also included code in rc.local that would simply write and check for a file. If the file exists then the server has been rebooted and it would delete the /etc/cloud/cloud-init.disabled file and execute cloud-init clean to do a clean run. This had mixed results as rc.local is executed at the end of each run level. So I scrapped that and decided the to create a simple service. The service does a few things it will prep the server then run cloud-init clean, write a file to the /var/lib/cloud dir indicating it was ran but not validated. When the server comes back up it checks for that file then if it finds it the script will run validation checks to ensure that cloud-init was successful (checks the host name is properly set to the hash of its IP, curls metadata and queries AD for a domain account) if any fail it will re-execute cloud-init clean. Once the server passes validation it writes a file to the same location indicating a successful run.

I admit it is a tad heavy handed but it has been the only way I have been able to get 100% success rate for ubuntu 18.04 on vIO. The problems imo come down to timing of when cloud-init runs and when the system is actually in a proper state for it to be ran, and a service seemed the logical route.

Attached is a text file that is the contents of the script and the unit file for the service. Its nothing special except I no longer get called because a users new instance failed to provision ;-)

cloud-init-fix.txt

[vmwarelab]

thanks for sharing @dasmokedog i was hoping for something easier.. its never easy when it comes to cloud-init and i am hoping i never have to use it on vSphere. its just ugly, not consistent results . meaning not always successful.

In my case i m testing it with vRA 8 and the only way currently working is if i use DHCP without any customization where the machine doesn't reboot. if i try to use a static IP assignment , the VM's NIC would be in a disconnected state when the VM is powered on and until customization is completed and the machine rebooted, the NIC then switch to a connected state . of course in that time Cloud-Init already tried running when there was no network and also gets interrupted by the customization reboot

still for having it work with DHCP the template had to be configured, like for example in my case :

1. I used this kb https://kb.vmware.com/s/article/56409
2. run sudo dpkg-reconfigure cloud-init and only select OVF only.
3. run sudo cloud-init clean --logs before shutting down the VM and turning it into a template.

In my case its a hit or mess still, for example if my deployment has two VMs being provisioned from the same image i prepared but with each having different user data . one would work and executes the command and one ends failing with DataSource not found. and sometimes they both work.

i tried some of the suggestions in the troubleshooting section in this blog by using cloud-init as the customization engine instead . https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/

Everything went okay except the DataSource ends being seed=VMwareTools and not OVF since vRA map the user-data as an iso connected to the VM CD-ROM device, but again cloud-init ends not executing anything but customization works.

[sqqqrly]

I am able to deploy ubuntu 18.04 to vSphere using terraform v0.13.5 and using cloud-init to configure the network.

Adding "After=dbus.service" to /lib/systemd/system/open-vm-tools.service fixed it.

I have an ansible role for installing open-vm-tools (from packer) and added this after installing the package:

      18 # Fix network interface not having a carrier.
      19 - name: Fix missing network error
      20   lineinfile:
      21     path: /lib/systemd/system/open-vm-tools.service
      22     insertafter: "After=apparmor.service"
      23     line: "After=dbus.service"

[PengpengSun]

Hi @sqqqrly

I am able to deploy ubuntu 18.04 to vSphere using terraform v0.13.5 and using cloud-init to configure the network.

Were you using cloud-init as customization engine which is described in this KB https://kb.vmware.com/s/article/54986 ?

I'm asking this since adding 'After=dbus.service' to open-vm-tools.service and customizing VM by cloud-init creates a dependency cycle for systemd:

• dbus has DefaultDependencies which makes it after sysinit.target
• Cloud-init-local needs to be early to make changes required before certain targets start e.g. Before=sysinit.target
• But open-vm-tools is before cloud-init-local

[sqqqrly]

@PengpengSun My Terraform deploy is similar to this example: https://github.com/grantorchard/terraform-vsphere-cloudinit-simple

I install open-vm-tools (modified service file) and cloud-init using packer to build an image. Later, Terraform then deploys the image and cloud-init runs using the extra_config data found in main.tf.

[PengpengSun]

@sqqqrly Great, so you are using https://github.com/vmware/cloud-init-vmware-guestinfo to deploy data to cloud-init, with that please ignore my previous comment on dependency cycle, they are different process.