vsphere: virtual_machine creation hangs on waiting for VM #7479

partamonov · 2016-07-04T15:15:10Z

terraform-0.6.16
Terraform version: 0.7.0 rc2 46a0709

Example resource:

resource "vsphere_virtual_machine" "web" {
  name   = "terraform-web"
  folder = "${vsphere_folder.frontend.path}"
  vcpu   = 1
  memory = 512
  cluster = "Cluster"
  domain = "TESTDOMAIN"
  datacenter = "DC"
  skip_customization = true

  network_interface {
    label = "some"
    ipv4_address = "10.X.X.X"
    ipv4_prefix_length = "27"
    ipv4_gateway = "10.X.X.1"
  }

  disk {
    template = "Templates/template-vyos.1.1.7-basic"
    type = "eager_zeroed"
    datastore = "datastore"
  }
}

And it is waiting for ever:

vsphere_virtual_machine.web: Still creating... (1m0s elapsed)
2016/07/04 17:00:41 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
2016/07/04 17:00:46 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
vsphere_virtual_machine.web: Still creating... (1m10s elapsed)
2016/07/04 17:00:47 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:00:47 [DEBUG] new vm: {VirtualMachine vm-770502} @ /VDC1/vm/Projects/terraform-test/terraform-web
2016/07/04 17:00:51 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
2016/07/04 17:00:53 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:00:53 [DEBUG] add cdroms: []
2016/07/04 17:00:53 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:00:53 [DEBUG] VM customization skipped
2016/07/04 17:00:53 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:00:53 [INFO] Created virtual machine: Projects/terraform-test/terraform-web
2016/07/04 17:00:53 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:00:53 [DEBUG] virtual machine resource data: &schema.ResourceData{schema:map[string]*schema.Schema{"vcpu":(*schema.Schema)(0x1c721a40), "memory":(*schema.Schema)(0x1c721ab0), "cluster":(*schema.Schema)(0x1c721c00), "linked_clone":(*schema.Schema)(0x1c721ce0), "cdrom":(*schema.Schema)(0x1c782230), "domain":(*schema.Schema)(0x1c721dc0), "dns_suffixes":(*schema.Schema)(0x1c721ea0), "enable_disk_uuid":(*schema.Schema)(0x1c782000), "custom_configuration_parameters":(*schema.Schema)(0x1c782070), "network_interface":(*schema.Schema)(0x1c782150), "windows_opt_config":(*schema.Schema)(0x1c7820e0), "memory_reservation":(*schema.Schema)(0x1c721b20), "datacenter":(*schema.Schema)(0x1c721b90), "resource_pool":(*schema.Schema)(0x1c721c70), "time_zone":(*schema.Schema)(0x1c721e30), "dns_servers":(*schema.Schema)(0x1c721f10), "name":(*schema.Schema)(0x1c7218f0), "folder":(*schema.Schema)(0x1c7219d0), "gateway":(*schema.Schema)(0x1c721d50), "skip_customization":(*schema.Schema)(0x1c721f80), "disk":(*schema.Schema)(0x1c7821c0)}, config:(*terraform.ResourceConfig)(nil), state:(*terraform.InstanceState)(nil), diff:(*terraform.InstanceDiff)(0x1c750d38), meta:map[string]string(nil), multiReader:(*schema.MultiLevelFieldReader)(0x1c6e83a0), setWriter:(*schema.MapFieldWriter)(0x1c6e8350), newState:(*terraform.InstanceState)(0x1c987140), partial:false, partialMap:map[string]struct {}(nil), once:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x1}, isNew:true}
2016/07/04 17:00:56 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
vsphere_virtual_machine.web: Still creating... (1m20s elapsed)
2016/07/04 17:01:01 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
2016/07/04 17:01:06 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
vsphere_virtual_machine.web: Still creating... (1m30s elapsed)
2016/07/04 17:01:11 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
2016/07/04 17:01:16 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
vsphere_virtual_machine.web: Still creating... (1m40s elapsed)
2016/07/04 17:01:21 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
2016/07/04 17:01:26 [DEBUG] vertex provider.vsphere (close), waiting for: vsphere_virtual_machine.web
vsphere_virtual_machine.web: Still creating... (1m50s elapsed)

At least I saw like 25 minutes waiting

When I abort run and try to re-run, I'm getting:

* vsphere_virtual_machine.web: The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.
2016/07/04 17:11:54 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* vsphere_virtual_machine.web: The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.
2016/07/04 17:11:54 [ERROR] root: eval: *terraform.EvalOpFilter, err: 1 error(s) occurred:

* vsphere_virtual_machine.web: The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.
2016/07/04 17:11:54 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* vsphere_virtual_machine.web: The guest operating system did not respond to a hot-remove request for device ethernet0 in a timely manner.
2016/07/04 17:11:54 [TRACE] [walkApply] Exiting eval tree: vsphere_virtual_machine.web
2016/07/04 17:11:54 [DEBUG] vertex provider.vsphere (close), got dep: vsphere_virtual_machine.web

Can you point me if this is terraform issue or something with template, which I'm using.

And this 2016/07/04 17:22:15 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:22:15 [DEBUG] add cdroms: [] 2016/07/04 17:22:15 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:22:15 [DEBUG] VM customization skipped 2016/07/04 17:22:15 [DEBUG] plugin: terraform: vsphere-provider (internal) 2016/07/04 17:22:15 [INFO] Created virtual machine: Projects/terraform-test/terraform-web is repeating from time to time

The text was updated successfully, but these errors were encountered:

natefanaro · 2016-07-12T21:47:34Z

Experiencing the same issue with Terraform v0.6.16 and ESXi 6.0.0. The machine is created and is bootable.

resource "vsphere_virtual_machine" "test" {
  name   = "terraform-test"
  vcpu   = 1
  memory = 4096
  network_interface {
    label = "VM Network"
  }
  datacenter = "ha-datacenter"
  skip_customization = true
  disk {
      datastore = "local"
      size = "20"
      type = "thin"
  }
  cdrom {
      datastore = "rackstation"
      path = "os_disk_images/ubuntu-15.10-server-amd64.iso"
  }
}

terraform output

$ terraform apply
vsphere_virtual_machine.test: Creating...
  cdrom.#:                                "" => "1"
  cdrom.0.datastore:                      "" => "rackstation"
  cdrom.0.path:                           "" => "os_disk_images/ubuntu-15.10-server-amd64.iso"
  datacenter:                             "" => "ha-datacenter"
  disk.#:                                 "" => "1"
  disk.0.bootable:                        "" => "false"
  disk.0.datastore:                       "" => "local"
  disk.0.size:                            "" => "20"
  disk.0.type:                            "" => "thin"
  domain:                                 "" => "vsphere.local"
  linked_clone:                           "" => "false"
  memory:                                 "" => "4096"
  memory_reservation:                     "" => "0"
  name:                                   "" => "terraform-test"
  network_interface.#:                    "" => "1"
  network_interface.0.ip_address:         "" => "<computed>"
  network_interface.0.ipv4_address:       "" => "<computed>"
  network_interface.0.ipv4_prefix_length: "" => "<computed>"
  network_interface.0.ipv6_address:       "" => "<computed>"
  network_interface.0.ipv6_prefix_length: "" => "<computed>"
  network_interface.0.label:              "" => "VM Network"
  network_interface.0.subnet_mask:        "" => "<computed>"
  skip_customization:                     "" => "true"
  time_zone:                              "" => "Etc/UTC"
  vcpu:                                   "" => "1"
vsphere_virtual_machine.test: Still creating... (10s elapsed)
vsphere_virtual_machine.test: Still creating... (20s elapsed)
vsphere_virtual_machine.test: Still creating... (30s elapsed)
vsphere_virtual_machine.test: Still creating... (40s elapsed)
vsphere_virtual_machine.test: Still creating... (50s elapsed)
vsphere_virtual_machine.test: Still creating... (1m0s elapsed)
vsphere_virtual_machine.test: Still creating... (1m10s elapsed)
vsphere_virtual_machine.test: Still creating... (1m20s elapsed)
vsphere_virtual_machine.test: Still creating... (1m30s elapsed)
vsphere_virtual_machine.test: Still creating... (1m40s elapsed)
vsphere_virtual_machine.test: Still creating... (1m50s elapsed)
vsphere_virtual_machine.test: Still creating... (2m0s elapsed)
vsphere_virtual_machine.test: Still creating... (2m10s elapsed)
vsphere_virtual_machine.test: Still creating... (2m20s elapsed)
vsphere_virtual_machine.test: Still creating... (2m30s elapsed)

nick-rumyantsev · 2016-07-13T20:10:38Z

How does terraform verify the VM state? Does it rely on vmwaretools inside the guest? If so, do you have them installed?

partamonov · 2016-07-18T18:16:39Z

terraform 0.7.0_rc3 is working for me. I will investigate why in coming days.

partamonov · 2016-08-01T15:31:52Z

I can say, that for me it was related to 2 things:

check that VM creation is really completed, sometimes applying storage recommendations take a lot of time
check that VM really got IP address, terraform was waiting for VM creation if, I created 2 VMs in same network without customizations from 1 template which already has some IP inside and once there were conflict one VM never replied to terraform about completeness.

rothgar · 2016-08-04T22:59:51Z

I have the same issue with 0.7.0

I let the loop run for ~40 minutes. Even though the VM is in vSphere. I wouldn't think the VM would be required to have network for it to be created. In my infrastructure I need to take the MAC address from the created machine to assign an IP lease for the system. But I can't get the MAC address until the machine has been created.

Is there a way to skip the IP check or a way to get details of the VM when the machine has not been verified to be up?

I'm using skip_customization but still am not able to verify the machine has been created.

wmangano · 2016-09-07T20:18:20Z

is there a fix for this issue?

im assigning a static ip address to a ubuntu vm (terraform .0.7.2/vsphere 6.0)

the "still creating" loop stops after 10 minutes. can you please implement a timeout....cause the vm is already running after 4-5 minutes.

the state detection does not work for whatever reason.

lasko · 2016-10-19T18:04:02Z

+1 to this.

Just spins and spins.

vsphere_virtual_machine.laskotest1: Still creating... (1m40s elapsed)
..
vsphere_virtual_machine.laskotest1: Still creating... (10m29s elapsed)
...

UPDATE: My problem is solved if I removed the second network_interface definition, and also only have a "label" definition. Otherwise I get the infinite loop.

Once my VM has been cloned, I then have to run govc vm.network.add to add the second vmxnet3 interface.

This is obviously not ideal and I don't know enough about Go to dive in and fix this. I do know enough about go to realize that is basically stalling on the fact that the interfaces don't have IP's and the network I'm connected to isn't going to be providing an IP via DHCP.

The IP's are assigned by hand -- effectively the bug as I see it is that the current implementation requires that a Network interface exist, and also that the network interface have an IP otherwise it just sits spinning on the WaitForNetIP method forever.

In a perfect world I presume that we should expect that the interface we are creating should have an IP, however, thats not always going to be the case -- Why hasn't a sensible timeout been set to at a minimum stop trying to wait after x seconds and continue on.

fulloflilies · 2016-12-01T20:45:59Z

+1 with v0.7.13
I only have "label" for network_interface.

network_interface {
label = "VM Network"
}

vsphere_virtual_machine.newvm: Still creating... (2m50s elapsed)
.....
vsphere_virtual_machine.newvm: Still creating... (35m20s elapsed)

tahirstamboli · 2016-12-20T05:56:32Z

I am facing the same issue. Please help me to resolve this issue. Or please specify alternate solution for this scenario.

Terraform file:

{
"resource" : {
"vsphere_virtual_machine" : {
"demo1_clone" : {
"disk" : {
"template" : "temp",
"datastore" : "datastore1"
},
"name" : "demo1_clone",
"network_interface" : {
"label" : "VM Network"
},
"cluster" : "192.168.1.216",
"skip_customization" : true,
"memory" : "1000",
"vcpu" : "1"
}
}
}
}

rickardrosen · 2017-01-03T11:02:21Z

Trying to create hosts with two NICs where the first is the "management interface" that will receive a DHCP lease and the second is an interface used only as a bridge adapter, and does not receive an IP address.
The interfaces are marked "connected" in vsphere, but Terraform waits indefinitely while trying to create this configuration.
If I remove the second interface it works as expected.

Is actual IP connectivity to all interfaces a requirement for the vsphere provider to succeed?
If so, perhaps a solution would be to mark interfaces with a "management" or "provisioner" bool, and disregard IP connectivity to other interfaces?

maxramqvist · 2017-01-03T15:33:45Z

Seems to be that every NIC with a mac-address needs an IP-adress:
WaitForNetIP-function in govmomi https://github.com/vmware/govmomi/blob/e702e188b645cf5f5b5fc00538eb2149b98ade0f/object/virtual_machine.go

rickardrosen · 2017-01-04T09:29:20Z

I guess finer control of this is required from govmomi.

Updating the WaitForNetIp function from:

		for _, ips := range macs {
			if len(ips) == 0 {
				return false
			}
		}

		return true

to:

		for _, ips := range macs {
			if len(ips) > 0 {
				return true
			}
		}

		return false

will continue if there is more than 0 ips assigned.

This at least allows terraform to complete in my "Multi-homed & bridge" scenario.

rickardrosen · 2017-01-31T14:00:16Z

The WaitForNetIP function has been updated in govmomi.
Now terraform needs an update to use this updated functionality.

moshevayner · 2017-03-28T19:54:28Z

Was this change already pushed? I have just updated to v0.9.2 and I still have this issue..
DEBUG log is showing that it's getting stuck waiting for interfaces to appear:

2017/03/28 15:46:28 [DEBUG] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "vsphere_virtual_machine.vm"
2017/03/28 15:46:28 [DEBUG] plugin: terraform: vsphere-provider (internal) 2017/03/28 15:46:28 [DEBUG] Waiting for interfaces to appear
2017/03/28 15:46:33 [DEBUG] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "vsphere_virtual_machine.vm"
2017/03/28 15:46:38 [DEBUG] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "vsphere_virtual_machine.vm"

Is there any workaround for this issue?

pryorda · 2017-04-14T19:28:05Z

+1 I need a work around for this as well. Taking 10 minutes to even get to provisioning the vm.

moshevayner · 2017-04-17T19:20:29Z

Is there anyone looking into this issue? Any ETA?

ziurjam · 2017-04-19T17:04:46Z

Any updates?

bpoland · 2017-04-20T14:21:31Z

I am seeing this too, when running "terraform refresh" it gets stuck on "waiting for interfaces to appear"

It only seems to happen on VMs that have been running for a while, and rebooting the VM seems to get it to go through for some reason...

pryorda · 2017-04-21T16:56:39Z

bpoland. I found rebooting the vm works as well.. Just dealt with this yesterday too..

bpoland · 2017-04-25T13:48:07Z

I think this may be related to a reboot of virtualcenter. It seems like if vCenter is rebooted then all VMs deployed before the reboot encounter this issue.

inequity · 2017-08-01T22:25:34Z

Sometimes I've found that a VM will just remain in a powered off state. If I manually start the machine, the terraform apply completes. It also seems that if an automatic migration is triggered on a new VM midway through the provisioning process, the process will fail.
ESXi 6.0.0 2715440
Terraform v0.9.11

poshpaws · 2018-11-22T21:14:09Z

I too may have this issue ,

Terraform v0.11.10

provider.vsphere v1.9.0
console output and base.tf are here : https://pastebin.com/VR1e11mt

rushi47 · 2019-04-03T10:17:38Z

Was facing the same issue but got resolved once I have added gateway and netmask.
And it seems that you have specified gateway inside network_interface try writing it outside the block.

Below is the code for reference:

resource "vsphere_virtual_machine" "vm" {
  name             = "TF Managed Server"
  folder           = "TestR&D"
  resource_pool_id = "${data.vsphere_compute_cluster.cluster.resource_pool_id}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"
  num_cpus = 1
  memory   = 1024
  guest_id = "rhel7_64Guest"
  network_interface = {
    network_id = "${data.vsphere_network.network.id}"
  }
  disk {
    label            = "disk0"
    size             = "50"
  }
  disk {
    label            = "disk1"
    size             = "50"
    unit_number      = 1
  }
  clone {
    template_uuid = "${data.vsphere_virtual_machine.template.id}"
    customize {
      linux_options {
        host_name = "terraform-test"
        domain    = "test.internal"
      }
      network_interface {
        ipv4_address = "10.111.1.11"
      }
      ipv4_gateway = "10.111.0.1"
    }
  }
}

ghost · 2019-08-13T02:15:54Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

stack72 added bug provider/vsphere labels Jul 5, 2016

rickardrosen mentioned this issue Jan 5, 2017

waitForNetIp to take arguments allowing to wait for specific nic(s) vmware/govmomi#653

Closed

dougm mentioned this issue Jan 24, 2017

Add device option to WaitForNetIP vmware/govmomi#660

Merged

bpoland mentioned this issue Apr 25, 2017

Crash when acting upon powered off VM #13803

Closed

hashibot mentioned this issue Jun 13, 2017

vsphere: virtual_machine creation hangs on waiting for VM hashicorp/terraform-provider-vsphere#31

Closed

hashibot closed this as completed Jun 13, 2017

hashicorp locked and limited conversation to collaborators Aug 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vsphere: virtual_machine creation hangs on waiting for VM #7479

vsphere: virtual_machine creation hangs on waiting for VM #7479

partamonov commented Jul 4, 2016 •

edited

natefanaro commented Jul 12, 2016

nick-rumyantsev commented Jul 13, 2016

partamonov commented Jul 18, 2016

partamonov commented Aug 1, 2016

rothgar commented Aug 4, 2016

wmangano commented Sep 7, 2016

lasko commented Oct 19, 2016 •

edited

fulloflilies commented Dec 1, 2016 •

edited

tahirstamboli commented Dec 20, 2016

rickardrosen commented Jan 3, 2017

maxramqvist commented Jan 3, 2017

rickardrosen commented Jan 4, 2017 •

edited

rickardrosen commented Jan 31, 2017

moshevayner commented Mar 28, 2017 •

edited

pryorda commented Apr 14, 2017

moshevayner commented Apr 17, 2017 •

edited

ziurjam commented Apr 19, 2017

bpoland commented Apr 20, 2017

pryorda commented Apr 21, 2017

bpoland commented Apr 25, 2017

inequity commented Aug 1, 2017

poshpaws commented Nov 22, 2018

rushi47 commented Apr 3, 2019 •

edited

ghost commented Aug 13, 2019

vsphere: virtual_machine creation hangs on waiting for VM #7479

vsphere: virtual_machine creation hangs on waiting for VM #7479

Comments

partamonov commented Jul 4, 2016 • edited

natefanaro commented Jul 12, 2016

nick-rumyantsev commented Jul 13, 2016

partamonov commented Jul 18, 2016

partamonov commented Aug 1, 2016

rothgar commented Aug 4, 2016

wmangano commented Sep 7, 2016

lasko commented Oct 19, 2016 • edited

fulloflilies commented Dec 1, 2016 • edited

tahirstamboli commented Dec 20, 2016

rickardrosen commented Jan 3, 2017

maxramqvist commented Jan 3, 2017

rickardrosen commented Jan 4, 2017 • edited

rickardrosen commented Jan 31, 2017

moshevayner commented Mar 28, 2017 • edited

pryorda commented Apr 14, 2017

moshevayner commented Apr 17, 2017 • edited

ziurjam commented Apr 19, 2017

bpoland commented Apr 20, 2017

pryorda commented Apr 21, 2017

bpoland commented Apr 25, 2017

inequity commented Aug 1, 2017

poshpaws commented Nov 22, 2018

rushi47 commented Apr 3, 2019 • edited

ghost commented Aug 13, 2019

partamonov commented Jul 4, 2016 •

edited

lasko commented Oct 19, 2016 •

edited

fulloflilies commented Dec 1, 2016 •

edited

rickardrosen commented Jan 4, 2017 •

edited

moshevayner commented Mar 28, 2017 •

edited

moshevayner commented Apr 17, 2017 •

edited

rushi47 commented Apr 3, 2019 •

edited