VMware Workstation IP Detection Issue

[ladar]

I believe I ran into a problem I've also had with vagrant. For some reason, with VMware workstation, at least on GUI desktops, there is an issue with getting the virtual networking up and running. The easiest fix seems to be opening the GUI, but I'm at a loss as to what the root cause is. I'm on CentOS 7.5.1804 using VMware Workstation 12.5.9.

The related issue in vagrant:

hashicorp/vagrant#9902

And the relevant debug logs from packer. I'm submitting a sample from early this morning, when builds were failing with error documented in issue #6689, and the corresponding snippet from the same section once the builds started working. Notice that in both, the dhcp.leases file is consulted, only in the failing builds, it doesn't yield the IP address.

2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 Special code '<enter>' found, replacing with: 0xFF0D
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 Located networkmapper configuration file using Workstation: /etc/vmware/netmap.conf
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 GuestIP discovered device matching nat: vmnet8
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 Lookup up IP information...
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 GuestAddress found MAC address in VMX: 00:0c:29:54:ef:f9
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 Trying DHCP leases path: /etc/vmware/vmnet8/dhcpd/dhcpd.leases
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 IP lookup failed: None of the found device(s) [vmnet8] has a DHCP lease for MAC 00:0c:29:54:ef:f9
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 [DEBUG] Unable to get address during connection step: IP lookup failed: None of the found device(s) [vmnet8] has a DHCP lease for MAC 00:0c:29:54:ef:f9
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 [INFO] Waiting for SSH, up to timeout: 1h0m0s
2018/09/11 07:32:55 ui: ==> generic-ubuntu1804-vmware: Waiting for SSH to become available...0m
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 Located networkmapper configuration file using Workstation: /etc/vmware/netmap.conf
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 GuestIP discovered device matching nat: vmnet8
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 Lookup up IP information...
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 GuestAddress found MAC address in VMX: 00:0c:29:54:ef:f9
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 Trying DHCP leases path: /etc/vmware/vmnet8/dhcpd/dhcpd.leases
2018/09/11 07:32:55 packer: 2018/09/11 07:32:55 IP lookup failed: None of the found device(s) [vmnet8] has a DHCP lease for MAC 00:0c:29:54:ef:f9

While the above was happening, I connected to the guest using VNC, and if I recall correctly, I wasn't able to access the HTTP server via the 172.16.207.1 address, but I could access it if I typed in the IP address for the host network card, aka 192.168.1.211 in this case.

Once things started working, the logs changed:

2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Special code '<enter>' found, replacing with: 0xFF0D
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Located networkmapper configuration file using Workstation: /etc/vmware/netmap.conf
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 GuestIP discovered device matching nat: vmnet8
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Lookup up IP information...
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 GuestAddress found MAC address in VMX: 00:0c:29:54:ef:f9
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Trying DHCP leases path: /etc/vmware/vmnet8/dhcpd/dhcpd.leases
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Detected IP: 172.16.207.128
2018/09/11 09:25:06 ui: ==> generic-ubuntu1804-vmware: Using ssh communicator to connect: 172.16.207.1280m
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 [INFO] Waiting for SSH, up to timeout: 1h0m0s
2018/09/11 09:25:06 ui: ==> generic-ubuntu1804-vmware: Waiting for SSH to become available...0m
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Located networkmapper configuration file using Workstation: /etc/vmware/netmap.conf
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 GuestIP discovered device matching nat: vmnet8
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Lookup up IP information...
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 GuestAddress found MAC address in VMX: 00:0c:29:54:ef:f9
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Trying DHCP leases path: /etc/vmware/vmnet8/dhcpd/dhcpd.leases
2018/09/11 09:25:06 packer: 2018/09/11 09:25:06 Detected IP: 172.16.207.128

It's worth noting that I don't seem to have this problem on my GUI-less robots (at least I haven't encountered this issue yet).

I'll hold onto the entire logs, in case anybody wants to me look for something specific, and I will update this ticket if I uncover any clues.

[SwampDragons]

Thanks. This one feels like it's going to be a tough one to crack.

[arizvisa]

Hey @ladar, sorry for getting to this so late. I was just going through looking for vsphere issues and spotted this one. But looking at the debug log that you posted in vagrant's tracker, as you mentioned the execution of vmrun.exe failed due to the VMware tools not running in the VM. I know it's so long ago, but were you able to verify that the service was actually running properly once it boots up and you shell into the VM despite there being no gui?

You should be able to see the state of the tools by checking the status of the vmtoolsd service, or if you need to do something lower-level, you can see the modules by grepping lsmod or checkin' with grep ^vm /proc/modules. The guestinfo plugin for vmtoolsd is responsible for updating VMware's state with the IP address (I think every 30 seconds or some other interval), so if the vmxnet drivers aren't loaded no address is able to be fetched.

btw, thanks for attempting to protect people's privacy way back when.

==> default: Waiting for the VM to receive an address...
 INFO vmware_driver: Reading an accessible IP for machine...
 INFO vmware_driver: Trying vmrun getGuestIPAddress...
 INFO subprocess: Starting process: ["/usr/bin/vmrun", "getGuestIPAddress", "/home/ladar/Desktop/vagrant/debian9-vmware/.vagrant/machines/default/vmware_workstation/5c9eef6a-0e0e-42b7-a3f0-1256f0dbb47e/generic-debian9-vmware.vmx"]
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stdout: Error: The VMware Tools are not running in the virtual machine: /home/ladar/Desktop/vagrant/debian9-vmware/.vagrant/machines/default/vmware_workstation/5c9eef6a-0e0e-42b7-a3f0-1256f0dbb47e/generic-debian9-vmware.vmx
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 255
 INFO vmware_driver: vmrun getGuestIPAddress failed: VMRunError

[arizvisa]

Actually, I just saw an issue somewhere else that you can test to see if you can exfiltrate an IP-address with vmtoolsd: vmtoolsd --cmd "info-set guestinfo.ip $IP". This might be another way to test that vmrun and vmtoolsd are communicating properly without a GUI.

[ladar]

@arizvisa I haven't seen this issue resurface lately... so perhaps the more recent packer builds, and or OS updates have fixed it... I'm going to close the ticket, at least until I run into the issue again.

[arizvisa]

Perfect. Thx.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.