Support VAppConfig for vsphere_virtual_machine resource

[rowanjacobs]

Hi,

We'd like to automate deploying a template that has some VAppConfig values, including IP, DNS server, gateway, etc. set, but it looks like the vsphere_virtual_machine resource currently doesn't support configuring those values.

We'd like to PR this support into the provider. We were thinking the config in the resource might look like this:

resource "vsphere_virtual_machine" "some-virtual-machine" {
  name     = "some-virtual-machine"
  hostname = "some-host"
  folder   = "some-folder"
  vcpu     = 2
  memory   = 4096

  app_config {
    ip = "10.10.10.10"
    dns = "8.8.8.8"
    admin_password = "some-password"
  }

...
}

We wanted to get some feedback on this before we start working on the PR; is this UI acceptable? any concerns about this that we might not be aware of?

cc @christianang

[vancluever]

Hey @rowanjacobs - thanks for checking in before starting work on this!

I have an incoming PR for the new vsphere_virtual_machine resource. Part of this work is a large re-organization of higher-level VM lifecycle config stuff like cloning and customization to dedicated blocks like you have have laid out above, so I think the config structure is on the right track.

However, I'm curious to the value of these would have when there is no current support for vApps within Terraform. This sounds like it might be setting some vApp options but not really doing anything outside of that. Just quickly looking at the docs here, it looks like editing vApp config options just simply assists with when you want to export the VM, which is a process that might be better suited for something like Packer or some other tool that is designed to build applications based on virtual machines, versus their deployment.

If you could reply with more of your use case, or if I'm missing something, we can see if this would be a good fit for the VM resource or not.

Thanks!

[rowanjacobs]

Our main use case is to deploy an existing template with configurable IP address, gateway, DNS, and NTP options.

Having spiked out some changes to the provider to add values specific to our vSphere cluster for those vAppConfig options, we were able to get the template working after terraform apply without any additional configuration. I admit I don't understand that much about vApps but that seems to me like it's doing more than just assisting export.

[vancluever]

Hey @rowanjacobs, when you say "template" - do you mean a traditional template, OVF file, or VM from a vApp?

If it's a regular template, we do support customization currently - it's mainly coupled to the entire resource. It's being re-factored in #244 and moved to its own dedicated section - you can see the PR for more details.

For OVFs, we do plan on supporting this eventually. The implementation has been envisioned as a two-part import of the VMDKs and VM configuration, and then population of various options in extra_config. I wonder if vApp configuration settings have some overlap there?

Will wait to hear from you on those questions 🙂

[rowanjacobs]

We deploy from an OVF file. Do you have a roadmap for OVF support, or an idea of what the UI for VM configuration or extra_config will look like?

[vancluever]

@rowanjacobs - interesting - so do you upload the OVF first and then clone it?

You can view #244 for links to the docs. extra_config is documented there, and is pretty much a key/value interface to the VM's advanced configuration options.

The exact options necessary will vary depending on the OVF you are deploying, I would imagine, although there might be common ones for vApps.

As far as the formal OFV support, there is no ETA, but we will be adding it sometime earlier in the new year, more than likely. The interface will more than likely look like a data source -> resource mapping, so that the VM disk names can be extracted from the virtual machine and what not to be uploaded, but until we dive into it, we won't know for sure. The data source could possibly also provide some sugar around generating the extra_config options so that they don't necessarily need to be entered completely "manually" into the VM resource, if that makes sense (depending on if that actually adds value or not).

A quick mockup of both the data source and resource example:

data "vsphere_ovf_file" "ovf" {
  source = "./virutal-appliance.ovf"
 
  properties {
    "Address Family"  = "ipv4"
    "IP Address"      = "172.16.0.100"
    "IP Prefix"       = "24"
    "Default Gateway" = "172.16.0.1"
  }
}

resource "vsphere_virtual_machine" "vm" {
  name = "deployed-appliance"

  num_cpus = "${data.vsphere_ovf_file.ovf.num_cpu}"
  memory   = "${data.vsphere_ovf_file.ovf.memory}"
  guest_id = "${data.vsphere_ovf_file.ovf.memory.guest_id}"
 
  ovf          = "${data.vsphere_ovf_file.ovf.source}"
  extra_config = "${data.vsphere_ovf_file.ovf.extra_config}"
} 

This is obviously pretty abridged, and the fields would be subject to change, but this is what I'm thinking. The main challenge I could see is that we would need to know the properties ahead of time. These are all things that will sort themselves out over the next couple of months though, and the VM resource is now in a pretty good place to be able to support this functionality in a pretty pluggable way.

Hope that gives you a bit of an idea of what's to come!

[vancluever]

Hey all, just thought I'd put some info here on some stuff I've been playing around with trying to get a CoreOS OVF template to work over here.

It looks like coreos-cloudinit in this scenario primarily attempts to use the OVF environment here to map the guestinfo confiugration namespace. This maps to the guestinfo.ovfEnv attribute. The problem is there seems to be a ton of conflicting information on how to properly configure this variable, or if it can be even configured altogether. Unfortunately, a lot of the info is not vSphere-centric, alluding to things that are not reasonably possible in vSphere, like manually editing the .vmx file, etc.

The main truth here that I can see from my own testing is that guestinfo.ovfEnv is a read-only variable that is not visible to either API or the vSphere client, and only serves as a transport to get the vApp configuration to the guest in an open format (OVF environment) via VMware tools. Using vmtoolsd --cmd "info-get guestinfo.ovfEnv" will return the correctly populated OVF environment, even though the value of this setting is not visible or settable anywhere else.

The conclusion here is that we will definitely need to support vApp config, at least in a way that allows the appropriate properties to be set. What I would almost like to see is the following:

resource "vsphere_virtual_machine" "vm" {
  name = "deployed-appliance"
   ...

  vapp {
    properties {
      "key" = "value"
    }
  }
} 

This will allow us to group vapp configuration, at whatever level we want to support (I don't see why we couldn't support the whole spec though), so that it can properly be set. Default the spec to support communication via the OVF environment transport (aka VMware tools).

Absence of the vapp block should remove vApp configuration on virtual machines that have it during clone or update. I'm still looking into this, but what I'm hoping is that this would allow systems that have things like cloud-init, or Ignition and what not to not look to the OVF environment first for this information and instead look to raw guestinfo from the VM config directly if people want this option, but I still have to test this manually.

[pall-valmundsson]

Hey @vancluever, just chiming in.

As it's unclear how successful you've been with your research I just wanted to let you know that I have been successful in deploying the CoreOS OVA with Ignition using just guestinfo.coreos.config.data and guestinfo.coreos.config.data.encoding and the ct_config data provider from CoreOS. I'm uploading the OVA using ovftool.

I haven't had to deploy a plain OVA with TF that doesn't have the glue that CoreOS provides with their cloud-init/ignition vSphere provider (which is pretty unique in the vSphere/OVA landscape), but it seems like that's possible using either ovftool, using --X:injectOvfEnv, or govc with a basic json options file.

Here's a blog that goes into using govc to deploy an OVA that depends on ovfEnv https://www.virtuallyghetto.com/2016/04/slick-way-of-deploying-ovfova-directly-to-esxi-vcenter-server-using-govc-cli.html

It looks like the flow is reading the base info of the OVA, merging provided options and injecting a marshalled XML into guestinfo.ovfEnv.

If this gets implemented it would be nice to document how to dump the available options and correctly configure the provider since all OVAs that I've come in contact with seem to name their options differently and, as you stated, there doesn't seem to be any good source of information on how to do this. I had to combine a lot of unconnected resources to get the Ignition deployment to work.

Regarding the interface, one might want to both inject OVF metadata as well as other extra_config. Would that be possible in your latest mockup?

Maybe a data provider for extra_config/guestinfo that contains the OVF marshalling logic which always injects that as guestinfo.ovfEnv?

Hope this is not redundant info :)

ps. I haven't tried this but users needing to deploy an OVA right now could probably manually create an XML template and inject that as guestinfo.ovfEnv. This might be handy for those going down that path: https://www.virtuallyghetto.com/2014/06/an-alternate-way-to-inject-ovf-properties-when-deploying-virtual-appliances-directly-onto-esxi.html

[vancluever]

Hey @pall-valmundsson,

It looks like the flow is reading the base info of the OVA, merging provided options and injecting a marshalled XML into guestinfo.ovfEnv.

This is the part that I actually tried. I wrote up a quick vsphere_ovf_environment data source (would have been awesome if it worked) that leveraged the ovf package in govmomi to marshal an XML and tried adding that as guestinfo.ovfEnv. No dice - vSphere just wiped the value every time it was injected.

That's when I realized what was going on with the vApp options and how that was being used as a transport for the guestinfo.ovfEnv. Altering the appropriate settings the vApp options section allowed them to show up in the OVF environment in the guest.

Regarding the interface, one might want to both inject OVF metadata as well as other extra_config. Would that be possible in your latest mockup?

Maybe a data provider for extra_config/guestinfo that contains the OVF marshalling logic which always injects that as guestinfo.ovfEnv?

Yeah, having the ability to do both is super important. vApp and extraConfig can co-exist in vSphere, so there's no reason they would not be able to in Terraform. It's just a matter of if your extra_config options would be ignored if cloud-init was prioritizing OVF environment over raw guestinfo as what seems to be going on here.

The issue is if the marshalling data source is actually necessary for the the issues that I ran into. Even if it works in some vSphere implementations, if it doesn't work in all of them then I don't necessarily want it in confusing things, especially if throwing your options in the vApp config properties works 100% of the time.

I want to take this on post-1.0.0. Importing OVF/OVA is something that I want to tackle specifically, but honestly, exposing vApp I think gets us a lot farther in supporting this path than we are currently and is really not much work at all, so I think that part might happen first. I think it would be a big win if someone could even just clone an existing template that was imported from OVF that has vApp options set on it and use it like they had just deployed the OVF - without going through all the hassle of importing, etc.

PS: On the CoreOS side, I had success by importing the OVF, then removing the vApp options, which allowed me to use the raw guestinfo.* values in extra_config. I think this causes cloud-init/Ignition to skip OVF environment, which seems to take priority.

[vancluever]

CC @rowanjacobs on the above ^^^ as you mentioned you wanted to work on a PR. 🙂 if this is something you were planning on taking on, let us know! Also, if you do plan on working on it can you make sure to base your work off #244.

[rowanjacobs]

Thanks! I'll check in with the team but I think it's likely we will be working on a PR. If we do, we'll make sure to use that branch.

[rowanjacobs]

We checked out the provider version with the latest changes to the resource. Without having made any code changes, acceptance tests don't pass because we get 501 Not Implemented when trying to query OS family for the VM guest ID.

Is there a particular version of vSphere that the new resource is targeting?

Here's the debug logs:

2017/11/30 12:25:40 [DEBUG] Looking for OS family for guest ID "other3xLinux64Guest"
2017/11/30 12:25:40 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 501 Not Implemented\r\nDate: Thu, 30 Nov 2017 20:26:43 GMT\r\nConnection: close\r\nContent-Type: text; charset=plain\r\nContent-Length: 0\r\n\r\n"; err=<nil>
2017/11/30 12:25:40 [ERROR] root: eval: *terraform.EvalDiff, err: cannot find OS family for guest ID "other3xLinux64Guest": ServerFaultCode: Unable to resolve WSDL method name QueryConfigOptionEx for namespace name urn:vim25

while parsing SOAP body
at line 2, column 77

while parsing SOAP envelope
at line 2, column 0

while parsing HTTP request before method was determined
at line 1, column 0

[vancluever]

@rowanjacobs I just checked the API docs and QueryConfigOptionEx is only available on vSphere 6.0 and higher. I'm checking to see what our options are as this is used in a couple of places in the new VM resource.

[vancluever]

@rowanjacobs looks like we were able to cleanly remove QueryConfigOptionEx from our environment browser helpers. The tests are running right now and this should be merged in about an hour, but things are looking good so far.

[genevieve]

Amazing! Thank you

[rowanjacobs]

We're seeing some more failures, mysterious to us, which might also be related to vSphere version.

2017/11/30 15:12:14 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* vsphere_virtual_machine.vm: error cloning virtual machine: ServerFaultCode:
Unexpected element tag "sharing" seen

while parsing serialized DataObject of type vim.vm.device.VirtualDisk.FlatVer2BackingInfo
at line 2, column 425

while parsing property "diskBackingInfo" of static type VirtualDeviceBackingInfo

while parsing serialized DataObject of type vim.vm.RelocateSpec.DiskLocator
at line 2, column 343

while parsing property "disk" of static type ArrayOfVirtualMachineRelocateSpecDiskLocator

while parsing serialized DataObject of type vim.vm.RelocateSpec
at line 2, column 233

while parsing property "location" of static type VirtualMachineRelocateSpec

while parsing serialized DataObject of type vim.vm.CloneSpec
at line 2, column 227

while parsing call information for method CloneVM_Task
at line 2, column 83

while parsing SOAP body
at line 2, column 77

while parsing SOAP envelope
at line 2, column 0

while parsing HTTP request for method clone
on object of type vim.VirtualMachine
at line 1, column 0

It's unclear to us where this "sharing" tag comes from, as we can't find the "sharing" literal in the codebase, and none of the code paths that set values on a FlatVer2BackingInfo object seem to be getting hit during the test.

[vancluever]

Hey @rowanjacobs, can you paste your config?

[vancluever]

Actually scratch that. I see that disk sharing is indeed a 6.0 feature. Seeing what we can do about dropping that as well.

[rowanjacobs]

Thanks!

[vancluever]

@rowanjacobs can you try against #258?

[rowanjacobs]

The "efiSecureBootEnabled" tag seems to also be a problem.

(By the way, we've just been running TestAccResourceVSphereVirtualMachine with everything but boot up, shutdown, and clone from template commented out.)

[amarruedo]

First of all, sorry if this is not the place for making questions, but I'm having a really difficult time trying to deploy a CoreOS vm machine on vSphere.

I imported a CoreOS OVA file, then created a virtual machine template, and finally, I'm trying to create the virtual machine using that template.

@vancluever when you point this:

PS: On the CoreOS side, I had success by importing the OVF, then removing the vApp options, which allowed me to use the raw guestinfo.* values in extra_config. I think this causes cloud-init/Ignition to skip OVF environment, which seems to take priority.

How do you set up the extra_config property? I'm doing this

...

#Define ignition data
data "ignition_networkd_unit" "vmnetwork" {
  name  = "00-ens192.network"

  content = <<EOF
  [Match]
  Name=ens192
  [Network]
  DNS=8.8.8.8
  Address=192.168.105.27/24
  Gateway=192.168.105.1
EOF
}

data "ignition_config" "node" {
  networkd = [
    "${data.ignition_networkd_unit.vmnetwork.id}"
  ]
}

# Define the VM resource
resource "vsphere_virtual_machine" "vm" {
 name   = "terraform-test"
 folder = "${vsphere_folder.TestPath.path}"
 resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
 datastore_id     = "${data.vsphere_datastore.datastore.id}"

 num_cpus = 2
 memory   = 1024
 guest_id = "other26xLinux64Guest"

 network_interface {
   network_id = "${data.vsphere_network.network.id}"
 }

 disk {
    name             = "terraform-test.vmdk"
    size             = "${data.vsphere_virtual_machine.template.disks.0.size}"
    eagerly_scrub    = "${data.vsphere_virtual_machine.template.disks.0.eagerly_scrub}"
    thin_provisioned = "${data.vsphere_virtual_machine.template.disks.0.thin_provisioned}"
  }

 clone {
    template_uuid = "${data.vsphere_virtual_machine.template.id}"
  }

 extra_config {
    guestinfo.coreos.config.data.encoding = "base64"
    guestinfo.coreos.config.data          = "${base64encode(data.ignition_config.node.rendered)}"
  }
...

and the only thing a I get is a virtual machine with no ip. Is this the right way to set that extra_config property?

On the other hand, my cluster is not enabled with DRS, so I have no vApp option tab in my machines. Should I enable it?

[pasikarkkainen]

extra_config {
guestinfo.foo = "bar"
}

Inside the guest I can see the value of key "foo" with command: vmtoolsd --cmd "info-get guestinfo.foo"

So that syntax definitely seems to work for me, using terraform v0.11.1 and vsphere provider v1.1.1.

[amarruedo]

Based on this comment I am starting to think that my problem is related to that vApp option disabling part. The problem is that the IT guys of my company have told me that we don't own the appropriate license to enable DRS, so I cannot interact with that vApp option tab (that tab is missing in my virtual machines).

The question now is, do we have to buy a new license in order to deploy CoreOS virtual machines in vSphere using Terraform?

I guess long term solution would be to wait for OVF support on the vSphere provider.

[rowanjacobs]

Since PR #303 was merged I'm going to go ahead and close this issue.