Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

no IP address assigned to new cloned 18.04 template #240

Open
j4y4r opened this issue Mar 12, 2018 · 58 comments

Comments

Projects
None yet
@j4y4r
Copy link

commented Mar 12, 2018

Hello,

i have an issue when deploying ubuntu18.04 Preview on an esxi version 6.0.

open-vm-tools version: 2:10.2.0-3ubuntu2

After creating a new VM from a Template with pyvmomi, the machine does not get an IP address assigned, also in the VM properties->Network the options "Connect" and "Connect at power on" are not checked.

The file /etc/network/interfaces still exists on the system, but there was nothing written into the file.

I also tried to create a new Vm from a Template via the vSphere Client with the same result.

Ubuntu changed their default Networking to netplan with 17.10 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2017-June/001215.html.

If you need any further infomation, logfiles, etc. please let me know.

@hangrao

This comment has been minimized.

Copy link

commented Mar 13, 2018

Thanks for the info.
Netplan is not supported in customization yet, it will be addressed in future vSphere releases.

Please see the KB for details:
https://ikb.vmware.com/s/article/52646

@j4y4r

This comment has been minimized.

Copy link
Author

commented Mar 13, 2018

Thanks for your answer and the link, i guess we have to wait for vmware then.

@saschalucas

This comment has been minimized.

Copy link

commented May 8, 2018

even if you remove netplan and install legacy ifup, guest customisation does not work (see /var/log/vmware-imc/toolsDeployPkg.log at https://gist.github.com/saschalucas/5595235e6ff41bc4a4ea2bac845bd2dc).

somehow it trys cloud-init but gave up later (don't know why).

@pengzhencao

This comment has been minimized.

Copy link

commented May 21, 2018

We do have several internal PRs tracking customization failure for ubuntu18.04.
error code 2 may be permission issue.
Could you also show me the "vmware*.log" inside the VM's folder on ESXi server?
(usually /vmfs/volumes/DATASTORE_NAME/VM_NAME/vmware*.log")
Could you also show me the open-vm-tools build number? Run "vmtoolsd -v" inside the VM could get this info.

Regarding the cloud-init customization option, there are two things must be done before using cloud-init based guest customization,

  1. Cloud-init package is installed properly(You have done this, this is what we see the "cloud-init 18.1" in log)
  2. "DISABLE_VMWARE_CUSTOMIZATION: false" set in "/etc/cloud/cloud.cfg" (you did not do this, this is why it fallback to traditional customization)
    ref: https://bazaar.launchpad.net/~baoli/cloud-init/cloud-init/revision/1173
@shibumi

This comment has been minimized.

Copy link

commented Jun 7, 2018

Hello everybody,
Is there a way to get this work? I really need this feature for my deployment. Would be nice to have a workaround. Maybe official vmware-tools?

EDIT: Ok seems like the official vmware-tools work with ubuntu18.04 (at least connecting the NIC work.. I am doing the rest via ansible)

@oliverkurth

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2018

@shibumi , did you provide the additional inormation requested by Peter, see #240 (comment) ?

@pengzhencao

This comment has been minimized.

Copy link

commented Jun 8, 2018

@shibumi , please try the following workaround to use customization with cloud-init:

  1. #apt get install cloud-init ( I think you've done this)
  2. Add ‘disable_vmware_customization: false’ to /etc/cloud/cloud.cfg
  3. Do the following to workaround tmp dir issue:
    1). Edit tmp.conf under /usr/lib/tmpfiles.d/
    2). comment out line 11 by adding '#' in front of this line.
    ex: #D /tmp 1777 root root -

Then you could try to run customization again.

@saschalucas

This comment has been minimized.

Copy link

commented Jun 8, 2018

Hi,

@pengzhencao here is the requested log (somehow anonymised):
vmware.log

the open-vm-tools is from stock ubuntu package:

vmtoolsd -v VMware Tools daemon, version 10.2.0.1608 (build-7253323)

I'll give cloud-init a try.

Thanks, Sascha.

@shibumi

This comment has been minimized.

Copy link

commented Jun 8, 2018

@oliverkurth @pengzhencao sorry maybe I expressed myself wrongly.. I am not interested in cloud-init configuration. In my case ansible does this via pyvmomi (the ESXI API) and the esxi server handles this via the running guest agent. What I want/need is a working open-vm-tools for Ubuntu 18.04

The official vm-tools are working. There the NICs get connected via the guest agent.. on ubuntu 18.04 and open-vm-tools this doesn't work.

@AndrewSav

This comment has been minimized.

Copy link

commented Jun 8, 2018

@shibumi wrong thread then?

@shibumi

This comment has been minimized.

Copy link

commented Jun 10, 2018

@AndrewSav no idea why you think this is the wrong thread. My case is exactly the one that the thread opener has. No idea why everybody started talking about cloud-init support for vmware suddenly in this thread.

The main problem is the same. open-vm-tools doesn't seem to connect NICs on Ubuntu 18.04. (Doesn't matter which tool you run above open-vm-tools. If pyvmomi, ansible-vmware or cloud-init. They all use open-vm-tools to get the NICs automatically connected. And this doesn't seem to work in ubuntu 18.04)

@hangrao

This comment has been minimized.

Copy link

commented Jun 20, 2018

@shibumi Which network controller are you using in your VM? As I know, the driver of the nics doesn't depend on OpenVMTools, all drivers (for e1000e, vmxnet3. ) are all included in the Linux kernel. (e1000 is not supported)

You could try different nic when you create the VM and see how it behave. In my understanding even there's no OVT installed the nics should work.

@shibumi

This comment has been minimized.

Copy link

commented Jun 20, 2018

@hangrao I use vmxnet3. The NIC works but they doesn't come up automatically and get connected via OVT. When I manually click on connect in the GUI of vmware center everything works fine.

@jord-bh

This comment has been minimized.

Copy link

commented Jun 21, 2018

I am hitting the same error as @shibumi through ansible, deploying a centos image works and brings up the NIC correctly, but starting an ubuntu image doesn't. I've tried to force call the object and setting connectable.connected = True to no effect. Doing it manually in GUI makes it connected.

Unsure if related or not, but I notice they have different VMWare Tools version installed (ubuntu: 10304, centos: 10282)

Also tried the workaround regarding cloud-init approach to no avail.

@hangrao

This comment has been minimized.

Copy link

commented Jun 22, 2018

@jord-bh @shibumi We tried to install ubuntu 18.04 from iso, and tried deploy ovf using ovftool or vSphere web client, either way is working fine. The Nic can go up successfully and the VM can get ip address.

Could you capture a screen shot with the error and add the reproducing steps there? I'd like to try that further.

@shibumi

This comment has been minimized.

Copy link

commented Jun 22, 2018

@hangrao See also this Bug Issue for Ansible: ansible/ansible#41133
Which version did of ovf did you use? I use the one in the Ubuntu 18.04 Repository.

And what we talk about is getting the nic up automatically via Vmware API (that uses OVF) not via manually clicking on things.

@hangrao

This comment has been minimized.

Copy link

commented Jun 22, 2018

@shibumi Thanks for the clarification, will have a try using steps you described in ansible/ansible#41133.

@jwalkins2017

This comment has been minimized.

Copy link

commented Jul 13, 2018

I am seeing the same issue when using salt-cloud to clone a vm. The clone's network adapter starts disconnected

@jord-bh

This comment has been minimized.

Copy link

commented Aug 3, 2018

@hangrao any luck reproducing this yet?

@hangrao

This comment has been minimized.

Copy link

commented Aug 6, 2018

@jord-bh We're able to reproduce this issue, and have already raised internal ticket to track it. Will update this thread if there's any progress.

@jstoja

This comment has been minimized.

Copy link

commented Aug 8, 2018

@pengzhencao Modifying the template to use cloud-init + setting to false the disable_vmware_customization worked for me. Waiting for any update to make it work straight away though!

@pengzhencao

This comment has been minimized.

Copy link

commented Aug 9, 2018

@jstoja Glad to know it could work.
That option is not totally controlled by VMware. Cloud-init developers disabled vmware customization by default, as they do not want to check for customization every OS boot due to performance concern.
refer:
https://git.launchpad.net/cloud-init/commit/?id=05afe04edbe4c28f2170194d226821c1e755ee2d

@shibumi

This comment has been minimized.

Copy link

commented Aug 9, 2018

@jstoja For me this even doesn't work when I have no cloudinit installed. So it's definitly something with Ubuntu 18.04

@jstoja

This comment has been minimized.

Copy link

commented Aug 9, 2018

@shibumi @pengzhencao I'll rebuild everything from scratch and try again to double check. If I get something working, I'll share it here.

@shibumi

This comment has been minimized.

Copy link

commented Aug 9, 2018

@jstoja did you install deprecated network-ifup scripts on that host?

@jstoja

This comment has been minimized.

Copy link

commented Aug 9, 2018

Humm I tested a dozen times with the exact same template each time and it worked twice without even doing something special... I didn't find something interesting in the logs... I tried with the legacy network-ifup and it doesn't help.

@shibumi

This comment has been minimized.

Copy link

commented Aug 9, 2018

@jstoja which esxi version? which template? which distribution and distribution version? was the template modified? if so, what has been modified? No idea why it works for you. Fact is for a lot of people it doesn't work :)

@jstoja

This comment has been minimized.

Copy link

commented Aug 9, 2018

Like I said in my last message, I relaunched it a dozen of times and it worked only twice... No idea why yet.
I'm trying with:
ESXI 6.0
Ubuntu 18.04 (https://github.com/heizo/packer-ubuntu-18.04 exported to https://gist.github.com/jstoja/544aac68ab2e8f142ad94ab9985aac5d)
I modified the template just by replacing the Network to one that is present into vSphere, but also the virtualbox-2.2 to vmx-07 value.

I've tried manually and it didn't work either, so I'm trying to find why did it work very rarely... The thing is that it's not only about the NIC only, nothing is customized, not even the hostname, so it's pretty easy to check.

@shibumi

This comment has been minimized.

Copy link

commented Aug 9, 2018

@jstoja
Ok here we have the difference.
I use ESXI 6.5 and I have installed ubuntu 18.04 Server Edition manually from ISO and installed Open-vm-tools in it and generated a template out of it.

So I guess it's working in your case either because of your esxi version or because of something special in your via hashicorp packer generated image..

@jstoja

This comment has been minimized.

Copy link

commented Aug 9, 2018

@jstoja

This comment has been minimized.

Copy link

commented Aug 9, 2018

Linked to #269 #267 #261

@PengpengSun

This comment has been minimized.

Copy link

commented Aug 14, 2018

@shibumi
Could you please share with me the /var/log/vmware-imc/toolsDeployPkg.log on the VM has this issue.

@PengpengSun

This comment has been minimized.

Copy link

commented Aug 14, 2018

@jstoja
No, ESXi 6.0 isn't supported for 18.04.
Ubuntu18.04 adopts Netplan on network, it supports from ESXi6.5 Update2.

@bitmand

This comment has been minimized.

Copy link

commented Aug 15, 2018

  • ESXi 6.5.0 (build 8307201)
  • Ubuntu 18.04
  • open-vm-tools 2:10.2.0-3ubuntu3
  • Bootstrapping a VM from a 18.04 template, using Ansible's vmware_guest module.

Sometimes it works - as in, the NIC is being connected in vmware, the network comes up and everything works as expected.

Sometimes it doesn't work - as in, the API request to vmware just hangs. If I look in vcenter, the NIC is not connected. If I wait it out, it just fails. If I manually connect the NIC in vcenter, during the API request, the bootstrap process continues and works as expected.

Wild guess, it's currently about 50/50 if it works or not.

@PengpengSun

This comment has been minimized.

Copy link

commented Aug 15, 2018

@bitmand This is very similar with an issue we found recently. Could you please share with me the /var/log/vmware-imc/toolsDeployPkg.log file when it doesn't work.

@bitmand

This comment has been minimized.

Copy link

commented Aug 15, 2018

@PengpengSun sure - attached :)

toolsDeployPkg.log

@PengpengSun

This comment has been minimized.

Copy link

commented Aug 15, 2018

@bitmand Yes, it's the same issue. The root cause is dbus.service is not running before guest customization happens, this is not always happening, depends on VM's cpu/memory numbers.
There is a workaround on this issue:
Add dependence to open-vm-tools.service by inserting a new line "After=dbus.service" under [Unit] of /lib/systemd/system/open-vm-tools.service file.

@bitmand

This comment has been minimized.

Copy link

commented Aug 15, 2018

@PengpengSun I added the line to open-vm-tools.service to make sure dbus service is started to the template.

Then created 5 VMs and all worked as expected - thank you very much for workaround, highly appreciated :)

@shibumi

This comment has been minimized.

Copy link

commented Aug 15, 2018

@PengpengSun I will try it tomorrow, when I am at work again.

@gaui

This comment has been minimized.

Copy link

commented Sep 6, 2018

@PengpengSun THANK YOU !!!!!

@idoru78

This comment has been minimized.

Copy link

commented Sep 14, 2018

We are also experiencing this exact issue on Ubuntu 16.04.

@PengpengSun

This comment has been minimized.

Copy link

commented Sep 14, 2018

@idoru78 The dbus issue should only reproduce on Ubuntu18.04, but I need know VMTools version on your Ubuntu 16.04 VM, and could you please share the /var/log/vmware-imc/toolsDeployPkg.log file.

@idoru78

This comment has been minimized.

Copy link

commented Sep 14, 2018

Sure.

Edit: Oops first one was from 18.04...

The log appears differently but the symptom is the same in that the NIC starts in a disconnected state.
log16.04.txt

We have worked around it for now my keep the open-vm-tools package at v10.0.7 and not upgrading to 10.2.

@PengpengSun

This comment has been minimized.

Copy link

commented Sep 14, 2018

From the log, it says the VM is Ubuntu18.04 but 16.04, please try the workaround in my previous comment at Aug 15.

DEBUG: Command: 'cat /etc/lsb-release'
DEBUG: Result: DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

@getSurreal

This comment has been minimized.

Copy link

commented Sep 15, 2018

VMware KB that was just posted as a workaround solved it for me

@maximhan

This comment has been minimized.

Copy link

commented Oct 2, 2018

Hi!

Could we get a clarification on whether or not netplan is supported during customization using open-vm-tools?

My understanding is that Ubuntu 18.04, including the configuration of netplan during the guest os customization is supported on vSphere 6.7 and 6.5U2.

We've tried this now with both cloud-init and the "traditional GOSC" workflows, but with no luck on getting the network configured on 18.04.

  • cloud-init workflow
    This one we couldn't get working at all, since nothing happened after it copied over the files to /var/run/vmware-imc. It just logged the that "Deployment delegated to Cloud-init. Returning success.", and then proceeds to exit the customization.
    Looking through the customization code, I don't really understand how the cloud-init workflow is supposed to even work, since there is nothing that triggers cloud-init init, or any other command, once the files have been copied over.

  • traditional GOSC workflow
    This one works, but not when it comes to configuring the network interface, since it just customizes the /etc/network/interfaces file during the network config stage of the workflow, which of course doesn't work on 18.04 unless we re-install ifupdown.

Environment Info:
vCenter: 6.5.0 Build 5973321
ESXi: 6.5.0 Build 8294253
Guest OS: Ubuntu Server 18.04.1
open-vm-tools: 2:10.3.0-0ubuntu1~18.04.2

Deployment logs:
toolsDeployPkg.log

@PengpengSun

This comment has been minimized.

Copy link

commented Oct 3, 2018

@maximhan Your understanding is correct, netplan supporting is available on 6.5U2, while the fix is on vCenter side, you need a 6.5U2 vCenter, your ESXi build is good.
Your current vCenter is 6.5U1, I double-checked toolsDeployPkg.log, the netplan fix is not there.
Please upgrade your vCenter to 6.5U2+ and have a try.

Thanks,
Pengpeng

@AndrewSav

This comment has been minimized.

Copy link

commented Oct 3, 2018

Works fine here on vCenter 6.7

@BrightVictoria

This comment has been minimized.

Copy link

commented Nov 15, 2018

Hi guys,
I have a similar error... no IP address assigned to new cloned 18.04 template, but host name and dns suffix assign without problems.
The error: Customization command failed: Could not set property: Access $ after the string DEBUG: Removing lock file /var/lock/vmware/gosc in the /var/log/vmware-imc/toolsDeployPkg.log file

ESXi 6.7.0
Ubuntu 18.04
open-vm-tools 2:10.3.0-0ubuntu1

@hangrao

This comment has been minimized.

Copy link

commented Nov 19, 2018

Hi guys,
I have a similar error... no IP address assigned to new cloned 18.04 template, but host name and dns suffix assign without problems.
The error: Customization command failed: Could not set property: Access $ after the string DEBUG: Removing lock file /var/lock/vmware/gosc in the /var/log/vmware-imc/toolsDeployPkg.log file

ESXi 6.7.0
Ubuntu 18.04
open-vm-tools 2:10.3.0-0ubuntu1

Hi BrightVictoria,

Could you check if you have already follow the steps in this KB:
https://kb.vmware.com/s/article/54986

Best regards,
Hang Rao

@mrlesmithjr

This comment has been minimized.

Copy link

commented Dec 5, 2018

Just adding my .02 here for reference. I am using vCenter 6.7 to start with. However, my scenario might be a bit different so let me explain.

The below involves Ubuntu 18.04 server w/out cloud-init.

I am building OVA images using Packer using the vsphere-iso builder. As part of the process I am having Packer remove the network interfaces as we may need to define different types of network adapter during provisioning. This OVA is uploaded to vCenter using Ansible as a new VM, remaining powered off. Then this VM is converted to a template via Ansible as well. When provisioning a VM the network adapter type is defined in which case the VM is spun up appropriately with the defined adapter type. The OS sees the adapter but the customization is failing during the netplan configuration.

image

image

image

image

image

Update
After following the recommendation above which lead to this KB https://kb.vmware.com/s/article/56409, I still experienced the same issue. However, switching from a vmxnet2 adapter type to a vmxnet3, the issue is not experienced. Is this a known thing with vmxnet2 adapters by chance?

Update
After looking at the VMware compatibility guide, it appears that vmxnet2 is not supported on Ubuntu 18.04 but was on 16.04.

@hangrao

This comment has been minimized.

Copy link

commented Jan 29, 2019

Update
After looking at the VMware compatibility guide, it appears that vmxnet2 is not supported on Ubuntu 18.04 but was on 16.04.

Right, vmxnet2 has been superseded by vmxnet3, please use vmxnet3 or other supported nics.

@spaced

This comment has been minimized.

Copy link

commented Feb 15, 2019

had a similar issue using centos 7 1810 image.
Problem found on the guest side:
Missing perl package crashes the scripts for the customization guest pkgs.
You may also have look in /var/log/vmware-incs/* logs after provisioning.

@PengpengSun

This comment has been minimized.

Copy link

commented Feb 22, 2019

Hi @spaced ,

perl package is necessary for guest customization to work, please include 'perl' at OS installation or install 'perl' package before do guest customization.

@rthimons85

This comment has been minimized.

Copy link

commented Mar 19, 2019

On ubuntu 18.04 cloud images, I found that /etc/netplan/50-cloud-init.yaml was conflicting with the vmware customizations.
Disabling it by deleting 50-cloud-init.yaml and creating a file named /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg and putting in

network: 
  config: disabled

fixed the issue, and my customizations completed properly.

@PengpengSun

This comment has been minimized.

Copy link

commented Mar 21, 2019

Thanks @rthimons85, Ubuntu 18.04 cloud image should have cloud-init installed according to the /etc/netplan/50-cloud-init.yaml file exits.
For Ubuntu 18.04 Live server image, it has cloud-init installed too. There is a KB article on disabling cloud-init if using perl based VMware customization.

@AndrewSav

This comment has been minimized.

Copy link

commented May 9, 2019

This is now closed as invalid over at LaunchPad Ubuntu.

@mylesagray

This comment has been minimized.

Copy link

commented Jun 9, 2019

Posted a blog on how to use cloud-init for full setup on vSphere: https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/

This took me days to figure out, but it hinged on a few things (covered in the Troubleshooting section of the blog) - but essentially, do NOT change the open-vm-tools system unit file if you want to use cloud-init.

Also, it seemed to matter that in the OVA spec, a hostname was not set for the initial template VM, and so it used the default.

Clearing /etc/machine-id is also essential for DHCP leases to not conflict as netplan uses this instead of MAC address for uniqueness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.