Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

no IP address assigned to new cloned 18.04 template #240

Open
j4y4r opened this issue Mar 12, 2018 · 66 comments
Open

no IP address assigned to new cloned 18.04 template #240

j4y4r opened this issue Mar 12, 2018 · 66 comments

Comments

@j4y4r
Copy link

@j4y4r j4y4r commented Mar 12, 2018

Hello,

i have an issue when deploying ubuntu18.04 Preview on an esxi version 6.0.

open-vm-tools version: 2:10.2.0-3ubuntu2

After creating a new VM from a Template with pyvmomi, the machine does not get an IP address assigned, also in the VM properties->Network the options "Connect" and "Connect at power on" are not checked.

The file /etc/network/interfaces still exists on the system, but there was nothing written into the file.

I also tried to create a new Vm from a Template via the vSphere Client with the same result.

Ubuntu changed their default Networking to netplan with 17.10 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2017-June/001215.html.

If you need any further infomation, logfiles, etc. please let me know.

@hangrao
Copy link

@hangrao hangrao commented Mar 13, 2018

Thanks for the info.
Netplan is not supported in customization yet, it will be addressed in future vSphere releases.

Please see the KB for details:
https://ikb.vmware.com/s/article/52646

@j4y4r
Copy link
Author

@j4y4r j4y4r commented Mar 13, 2018

Thanks for your answer and the link, i guess we have to wait for vmware then.

@saschalucas
Copy link

@saschalucas saschalucas commented May 8, 2018

even if you remove netplan and install legacy ifup, guest customisation does not work (see /var/log/vmware-imc/toolsDeployPkg.log at https://gist.github.com/saschalucas/5595235e6ff41bc4a4ea2bac845bd2dc).

somehow it trys cloud-init but gave up later (don't know why).

@pengzhencao
Copy link

@pengzhencao pengzhencao commented May 21, 2018

We do have several internal PRs tracking customization failure for ubuntu18.04.
error code 2 may be permission issue.
Could you also show me the "vmware*.log" inside the VM's folder on ESXi server?
(usually /vmfs/volumes/DATASTORE_NAME/VM_NAME/vmware*.log")
Could you also show me the open-vm-tools build number? Run "vmtoolsd -v" inside the VM could get this info.

Regarding the cloud-init customization option, there are two things must be done before using cloud-init based guest customization,

  1. Cloud-init package is installed properly(You have done this, this is what we see the "cloud-init 18.1" in log)
  2. "DISABLE_VMWARE_CUSTOMIZATION: false" set in "/etc/cloud/cloud.cfg" (you did not do this, this is why it fallback to traditional customization)
    ref: https://bazaar.launchpad.net/~baoli/cloud-init/cloud-init/revision/1173

@shibumi
Copy link

@shibumi shibumi commented Jun 7, 2018

Hello everybody,
Is there a way to get this work? I really need this feature for my deployment. Would be nice to have a workaround. Maybe official vmware-tools?

EDIT: Ok seems like the official vmware-tools work with ubuntu18.04 (at least connecting the NIC work.. I am doing the rest via ansible)

@oliverkurth
Copy link
Contributor

@oliverkurth oliverkurth commented Jun 7, 2018

@shibumi , did you provide the additional inormation requested by Peter, see #240 (comment) ?

@pengzhencao
Copy link

@pengzhencao pengzhencao commented Jun 8, 2018

@shibumi , please try the following workaround to use customization with cloud-init:

  1. #apt get install cloud-init ( I think you've done this)
  2. Add ‘disable_vmware_customization: false’ to /etc/cloud/cloud.cfg
  3. Do the following to workaround tmp dir issue:
    1). Edit tmp.conf under /usr/lib/tmpfiles.d/
    2). comment out line 11 by adding '#' in front of this line.
    ex: #D /tmp 1777 root root -

Then you could try to run customization again.

@saschalucas
Copy link

@saschalucas saschalucas commented Jun 8, 2018

Hi,

@pengzhencao here is the requested log (somehow anonymised):
vmware.log

the open-vm-tools is from stock ubuntu package:

vmtoolsd -v VMware Tools daemon, version 10.2.0.1608 (build-7253323)

I'll give cloud-init a try.

Thanks, Sascha.

@shibumi
Copy link

@shibumi shibumi commented Jun 8, 2018

@oliverkurth @pengzhencao sorry maybe I expressed myself wrongly.. I am not interested in cloud-init configuration. In my case ansible does this via pyvmomi (the ESXI API) and the esxi server handles this via the running guest agent. What I want/need is a working open-vm-tools for Ubuntu 18.04

The official vm-tools are working. There the NICs get connected via the guest agent.. on ubuntu 18.04 and open-vm-tools this doesn't work.

@AndrewSav
Copy link

@AndrewSav AndrewSav commented Jun 8, 2018

@shibumi wrong thread then?

@shibumi
Copy link

@shibumi shibumi commented Jun 10, 2018

@AndrewSav no idea why you think this is the wrong thread. My case is exactly the one that the thread opener has. No idea why everybody started talking about cloud-init support for vmware suddenly in this thread.

The main problem is the same. open-vm-tools doesn't seem to connect NICs on Ubuntu 18.04. (Doesn't matter which tool you run above open-vm-tools. If pyvmomi, ansible-vmware or cloud-init. They all use open-vm-tools to get the NICs automatically connected. And this doesn't seem to work in ubuntu 18.04)

@hangrao
Copy link

@hangrao hangrao commented Jun 20, 2018

@shibumi Which network controller are you using in your VM? As I know, the driver of the nics doesn't depend on OpenVMTools, all drivers (for e1000e, vmxnet3. ) are all included in the Linux kernel. (e1000 is not supported)

You could try different nic when you create the VM and see how it behave. In my understanding even there's no OVT installed the nics should work.

@shibumi
Copy link

@shibumi shibumi commented Jun 20, 2018

@hangrao I use vmxnet3. The NIC works but they doesn't come up automatically and get connected via OVT. When I manually click on connect in the GUI of vmware center everything works fine.

@jord-bh
Copy link

@jord-bh jord-bh commented Jun 21, 2018

I am hitting the same error as @shibumi through ansible, deploying a centos image works and brings up the NIC correctly, but starting an ubuntu image doesn't. I've tried to force call the object and setting connectable.connected = True to no effect. Doing it manually in GUI makes it connected.

Unsure if related or not, but I notice they have different VMWare Tools version installed (ubuntu: 10304, centos: 10282)

Also tried the workaround regarding cloud-init approach to no avail.

@hangrao
Copy link

@hangrao hangrao commented Jun 22, 2018

@jord-bh @shibumi We tried to install ubuntu 18.04 from iso, and tried deploy ovf using ovftool or vSphere web client, either way is working fine. The Nic can go up successfully and the VM can get ip address.

Could you capture a screen shot with the error and add the reproducing steps there? I'd like to try that further.

@shibumi
Copy link

@shibumi shibumi commented Jun 22, 2018

@hangrao See also this Bug Issue for Ansible: ansible/ansible#41133
Which version did of ovf did you use? I use the one in the Ubuntu 18.04 Repository.

And what we talk about is getting the nic up automatically via Vmware API (that uses OVF) not via manually clicking on things.

@hangrao
Copy link

@hangrao hangrao commented Jun 22, 2018

@shibumi Thanks for the clarification, will have a try using steps you described in ansible/ansible#41133.

@jwalkins2017
Copy link

@jwalkins2017 jwalkins2017 commented Jul 13, 2018

I am seeing the same issue when using salt-cloud to clone a vm. The clone's network adapter starts disconnected

@jord-bh
Copy link

@jord-bh jord-bh commented Aug 3, 2018

@hangrao any luck reproducing this yet?

@hangrao
Copy link

@hangrao hangrao commented Aug 6, 2018

@jord-bh We're able to reproduce this issue, and have already raised internal ticket to track it. Will update this thread if there's any progress.

@jstoja
Copy link

@jstoja jstoja commented Aug 8, 2018

@pengzhencao Modifying the template to use cloud-init + setting to false the disable_vmware_customization worked for me. Waiting for any update to make it work straight away though!

@pengzhencao
Copy link

@pengzhencao pengzhencao commented Aug 9, 2018

@jstoja Glad to know it could work.
That option is not totally controlled by VMware. Cloud-init developers disabled vmware customization by default, as they do not want to check for customization every OS boot due to performance concern.
refer:
https://git.launchpad.net/cloud-init/commit/?id=05afe04edbe4c28f2170194d226821c1e755ee2d

@shibumi
Copy link

@shibumi shibumi commented Aug 9, 2018

@jstoja For me this even doesn't work when I have no cloudinit installed. So it's definitly something with Ubuntu 18.04

@jstoja
Copy link

@jstoja jstoja commented Aug 9, 2018

@shibumi @pengzhencao I'll rebuild everything from scratch and try again to double check. If I get something working, I'll share it here.

@shibumi
Copy link

@shibumi shibumi commented Aug 9, 2018

@jstoja did you install deprecated network-ifup scripts on that host?

@jstoja
Copy link

@jstoja jstoja commented Aug 9, 2018

Humm I tested a dozen times with the exact same template each time and it worked twice without even doing something special... I didn't find something interesting in the logs... I tried with the legacy network-ifup and it doesn't help.

@shibumi
Copy link

@shibumi shibumi commented Aug 9, 2018

@jstoja which esxi version? which template? which distribution and distribution version? was the template modified? if so, what has been modified? No idea why it works for you. Fact is for a lot of people it doesn't work :)

@jstoja
Copy link

@jstoja jstoja commented Aug 9, 2018

Like I said in my last message, I relaunched it a dozen of times and it worked only twice... No idea why yet.
I'm trying with:
ESXI 6.0
Ubuntu 18.04 (https://github.com/heizo/packer-ubuntu-18.04 exported to https://gist.github.com/jstoja/544aac68ab2e8f142ad94ab9985aac5d)
I modified the template just by replacing the Network to one that is present into vSphere, but also the virtualbox-2.2 to vmx-07 value.

I've tried manually and it didn't work either, so I'm trying to find why did it work very rarely... The thing is that it's not only about the NIC only, nothing is customized, not even the hostname, so it's pretty easy to check.

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Aug 15, 2018

@bitmand Yes, it's the same issue. The root cause is dbus.service is not running before guest customization happens, this is not always happening, depends on VM's cpu/memory numbers.
There is a workaround on this issue:
Add dependence to open-vm-tools.service by inserting a new line "After=dbus.service" under [Unit] of /lib/systemd/system/open-vm-tools.service file.

@bitmand
Copy link

@bitmand bitmand commented Aug 15, 2018

@PengpengSun I added the line to open-vm-tools.service to make sure dbus service is started to the template.

Then created 5 VMs and all worked as expected - thank you very much for workaround, highly appreciated :)

@shibumi
Copy link

@shibumi shibumi commented Aug 15, 2018

@PengpengSun I will try it tomorrow, when I am at work again.

@gaui
Copy link

@gaui gaui commented Sep 6, 2018

@PengpengSun THANK YOU !!!!!

@idoru78
Copy link

@idoru78 idoru78 commented Sep 14, 2018

We are also experiencing this exact issue on Ubuntu 16.04.

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Sep 14, 2018

@idoru78 The dbus issue should only reproduce on Ubuntu18.04, but I need know VMTools version on your Ubuntu 16.04 VM, and could you please share the /var/log/vmware-imc/toolsDeployPkg.log file.

@idoru78
Copy link

@idoru78 idoru78 commented Sep 14, 2018

Sure.

Edit: Oops first one was from 18.04...

The log appears differently but the symptom is the same in that the NIC starts in a disconnected state.
log16.04.txt

We have worked around it for now my keep the open-vm-tools package at v10.0.7 and not upgrading to 10.2.

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Sep 14, 2018

From the log, it says the VM is Ubuntu18.04 but 16.04, please try the workaround in my previous comment at Aug 15.

DEBUG: Command: 'cat /etc/lsb-release'
DEBUG: Result: DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

@getSurreal
Copy link

@getSurreal getSurreal commented Sep 15, 2018

VMware KB that was just posted as a workaround solved it for me

@maximhan
Copy link

@maximhan maximhan commented Oct 2, 2018

Hi!

Could we get a clarification on whether or not netplan is supported during customization using open-vm-tools?

My understanding is that Ubuntu 18.04, including the configuration of netplan during the guest os customization is supported on vSphere 6.7 and 6.5U2.

We've tried this now with both cloud-init and the "traditional GOSC" workflows, but with no luck on getting the network configured on 18.04.

  • cloud-init workflow
    This one we couldn't get working at all, since nothing happened after it copied over the files to /var/run/vmware-imc. It just logged the that "Deployment delegated to Cloud-init. Returning success.", and then proceeds to exit the customization.
    Looking through the customization code, I don't really understand how the cloud-init workflow is supposed to even work, since there is nothing that triggers cloud-init init, or any other command, once the files have been copied over.

  • traditional GOSC workflow
    This one works, but not when it comes to configuring the network interface, since it just customizes the /etc/network/interfaces file during the network config stage of the workflow, which of course doesn't work on 18.04 unless we re-install ifupdown.

Environment Info:
vCenter: 6.5.0 Build 5973321
ESXi: 6.5.0 Build 8294253
Guest OS: Ubuntu Server 18.04.1
open-vm-tools: 2:10.3.0-0ubuntu1~18.04.2

Deployment logs:
toolsDeployPkg.log

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Oct 3, 2018

@maximhan Your understanding is correct, netplan supporting is available on 6.5U2, while the fix is on vCenter side, you need a 6.5U2 vCenter, your ESXi build is good.
Your current vCenter is 6.5U1, I double-checked toolsDeployPkg.log, the netplan fix is not there.
Please upgrade your vCenter to 6.5U2+ and have a try.

Thanks,
Pengpeng

@AndrewSav
Copy link

@AndrewSav AndrewSav commented Oct 3, 2018

Works fine here on vCenter 6.7

@BrightVictoria
Copy link

@BrightVictoria BrightVictoria commented Nov 15, 2018

Hi guys,
I have a similar error... no IP address assigned to new cloned 18.04 template, but host name and dns suffix assign without problems.
The error: Customization command failed: Could not set property: Access $ after the string DEBUG: Removing lock file /var/lock/vmware/gosc in the /var/log/vmware-imc/toolsDeployPkg.log file

ESXi 6.7.0
Ubuntu 18.04
open-vm-tools 2:10.3.0-0ubuntu1

@hangrao
Copy link

@hangrao hangrao commented Nov 19, 2018

Hi guys,
I have a similar error... no IP address assigned to new cloned 18.04 template, but host name and dns suffix assign without problems.
The error: Customization command failed: Could not set property: Access $ after the string DEBUG: Removing lock file /var/lock/vmware/gosc in the /var/log/vmware-imc/toolsDeployPkg.log file

ESXi 6.7.0
Ubuntu 18.04
open-vm-tools 2:10.3.0-0ubuntu1

Hi BrightVictoria,

Could you check if you have already follow the steps in this KB:
https://kb.vmware.com/s/article/54986

Best regards,
Hang Rao

@mrlesmithjr
Copy link

@mrlesmithjr mrlesmithjr commented Dec 5, 2018

Just adding my .02 here for reference. I am using vCenter 6.7 to start with. However, my scenario might be a bit different so let me explain.

The below involves Ubuntu 18.04 server w/out cloud-init.

I am building OVA images using Packer using the vsphere-iso builder. As part of the process I am having Packer remove the network interfaces as we may need to define different types of network adapter during provisioning. This OVA is uploaded to vCenter using Ansible as a new VM, remaining powered off. Then this VM is converted to a template via Ansible as well. When provisioning a VM the network adapter type is defined in which case the VM is spun up appropriately with the defined adapter type. The OS sees the adapter but the customization is failing during the netplan configuration.

image

image

image

image

image

Update
After following the recommendation above which lead to this KB https://kb.vmware.com/s/article/56409, I still experienced the same issue. However, switching from a vmxnet2 adapter type to a vmxnet3, the issue is not experienced. Is this a known thing with vmxnet2 adapters by chance?

Update
After looking at the VMware compatibility guide, it appears that vmxnet2 is not supported on Ubuntu 18.04 but was on 16.04.

@hangrao
Copy link

@hangrao hangrao commented Jan 29, 2019

Update
After looking at the VMware compatibility guide, it appears that vmxnet2 is not supported on Ubuntu 18.04 but was on 16.04.

Right, vmxnet2 has been superseded by vmxnet3, please use vmxnet3 or other supported nics.

@spaced
Copy link

@spaced spaced commented Feb 15, 2019

had a similar issue using centos 7 1810 image.
Problem found on the guest side:
Missing perl package crashes the scripts for the customization guest pkgs.
You may also have look in /var/log/vmware-incs/* logs after provisioning.

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Feb 22, 2019

Hi @spaced ,

perl package is necessary for guest customization to work, please include 'perl' at OS installation or install 'perl' package before do guest customization.

@rthimons85
Copy link

@rthimons85 rthimons85 commented Mar 19, 2019

On ubuntu 18.04 cloud images, I found that /etc/netplan/50-cloud-init.yaml was conflicting with the vmware customizations.
Disabling it by deleting 50-cloud-init.yaml and creating a file named /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg and putting in

network: 
  config: disabled

fixed the issue, and my customizations completed properly.

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Mar 21, 2019

Thanks @rthimons85, Ubuntu 18.04 cloud image should have cloud-init installed according to the /etc/netplan/50-cloud-init.yaml file exits.
For Ubuntu 18.04 Live server image, it has cloud-init installed too. There is a KB article on disabling cloud-init if using perl based VMware customization.

@AndrewSav
Copy link

@AndrewSav AndrewSav commented May 9, 2019

This is now closed as invalid over at LaunchPad Ubuntu.

@mylesagray
Copy link

@mylesagray mylesagray commented Jun 9, 2019

Posted a blog on how to use cloud-init for full setup on vSphere: https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/

This took me days to figure out, but it hinged on a few things (covered in the Troubleshooting section of the blog) - but essentially, do NOT change the open-vm-tools system unit file if you want to use cloud-init.

Also, it seemed to matter that in the OVA spec, a hostname was not set for the initial template VM, and so it used the default.

Clearing /etc/machine-id is also essential for DHCP leases to not conflict as netplan uses this instead of MAC address for uniqueness.

@dasmokedog
Copy link

@dasmokedog dasmokedog commented Jun 24, 2019

If you are running VIO (VMware Integrated Openstack) and smashing your head against the wall trying to get cloud-init to identify the datasource, like me, then you will need to ensure vmtools (or open-vm-tools) runs and registers the IP with SpoofGuard. Until this happens all traffic will be blocked. The quickest way is to arm cloud-init to run on 2nd boot.

@vmwarelab
Copy link

@vmwarelab vmwarelab commented Feb 1, 2020

@dasmokedog how do you configure cloud-init to run on 2nd boot instead of first boot?

@dasmokedog
Copy link

@dasmokedog dasmokedog commented Feb 1, 2020

@dasmokedog how do you configure cloud-init to run on 2nd boot instead of first boot?

@vmwarelab Initially I approached it by adding /etc/cloud/cloud-init.disabled at the end of the Packer run before it creates the image. I also included code in rc.local that would simply write and check for a file. If the file exists then the server has been rebooted and it would delete the /etc/cloud/cloud-init.disabled file and execute cloud-init clean to do a clean run. This had mixed results as rc.local is executed at the end of each run level. So I scrapped that and decided the to create a simple service. The service does a few things it will prep the server then run cloud-init clean, write a file to the /var/lib/cloud dir indicating it was ran but not validated. When the server comes back up it checks for that file then if it finds it the script will run validation checks to ensure that cloud-init was successful (checks the host name is properly set to the hash of its IP, curls metadata and queries AD for a domain account) if any fail it will re-execute cloud-init clean. Once the server passes validation it writes a file to the same location indicating a successful run.

I admit it is a tad heavy handed but it has been the only way I have been able to get 100% success rate for ubuntu 18.04 on vIO. The problems imo come down to timing of when cloud-init runs and when the system is actually in a proper state for it to be ran, and a service seemed the logical route.

Attached is a text file that is the contents of the script and the unit file for the service. Its nothing special except I no longer get called because a users new instance failed to provision ;-)

cloud-init-fix.txt

@vmwarelab
Copy link

@vmwarelab vmwarelab commented Feb 2, 2020

thanks for sharing @dasmokedog i was hoping for something easier.. its never easy when it comes to cloud-init and i am hoping i never have to use it on vSphere. its just ugly, not consistent results . meaning not always successful.

In my case i m testing it with vRA 8 and the only way currently working is if i use DHCP without any customization where the machine doesn't reboot. if i try to use a static IP assignment , the VM's NIC would be in a disconnected state when the VM is powered on and until customization is completed and the machine rebooted, the NIC then switch to a connected state . of course in that time Cloud-Init already tried running when there was no network and also gets interrupted by the customization reboot

still for having it work with DHCP the template had to be configured, like for example in my case :

  1. I used this kb https://kb.vmware.com/s/article/56409
  2. run sudo dpkg-reconfigure cloud-init and only select OVF only.
  3. run sudo cloud-init clean --logs before shutting down the VM and turning it into a template.

In my case its a hit or mess still, for example if my deployment has two VMs being provisioned from the same image i prepared but with each having different user data . one would work and executes the command and one ends failing with DataSource not found. and sometimes they both work.

i tried some of the suggestions in the troubleshooting section in this blog by using cloud-init as the customization engine instead . https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/

Everything went okay except the DataSource ends being seed=VMwareTools and not OVF since vRA map the user-data as an iso connected to the VM CD-ROM device, but again cloud-init ends not executing anything but customization works.

@sqqqrly
Copy link

@sqqqrly sqqqrly commented Dec 1, 2020

I am able to deploy ubuntu 18.04 to vSphere using terraform v0.13.5 and using cloud-init to configure the network.

Adding "After=dbus.service" to /lib/systemd/system/open-vm-tools.service fixed it.

I have an ansible role for installing open-vm-tools (from packer) and added this after installing the package:

      18 # Fix network interface not having a carrier.
      19 - name: Fix missing network error
      20   lineinfile:
      21     path: /lib/systemd/system/open-vm-tools.service
      22     insertafter: "After=apparmor.service"
      23     line: "After=dbus.service"

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Dec 2, 2020

Hi @sqqqrly

I am able to deploy ubuntu 18.04 to vSphere using terraform v0.13.5 and using cloud-init to configure the network.

Were you using cloud-init as customization engine which is described in this KB https://kb.vmware.com/s/article/54986 ?

I'm asking this since adding 'After=dbus.service' to open-vm-tools.service and customizing VM by cloud-init creates a dependency cycle for systemd:

  • dbus has DefaultDependencies which makes it after sysinit.target
  • Cloud-init-local needs to be early to make changes required before certain targets start e.g. Before=sysinit.target
  • But open-vm-tools is before cloud-init-local

@sqqqrly
Copy link

@sqqqrly sqqqrly commented Dec 2, 2020

@PengpengSun My Terraform deploy is similar to this example: https://github.com/grantorchard/terraform-vsphere-cloudinit-simple

I install open-vm-tools (modified service file) and cloud-init using packer to build an image. Later, Terraform then deploys the image and cloud-init runs using the extra_config data found in main.tf.

@PengpengSun
Copy link

@PengpengSun PengpengSun commented Dec 3, 2020

@sqqqrly Great, so you are using https://github.com/vmware/cloud-init-vmware-guestinfo to deploy data to cloud-init, with that please ignore my previous comment on dependency cycle, they are different process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet