[RFC] builder/vmware: Support ESXi with the vmware-vmx builder

[klaamane]

This PR enables the vmware-vmx builder to do remote (esxi) builds in the same way that vmware-iso does. Some notes:

• We have been using this feature now for doing incremental iso->vmx->vmx builds in an ESXi environment.
• Basically this PR moves multiple files from the vmware/iso to the shared vmware/common package. The (re)moved files are: artifact.go, artifact_test.go, driver.go, driver_esx5.go, driver_esx5_test.go, output_dir.go, remote_driver.go, remote_driver_mock.go, remote_driver_mock_test.go, step_register.go, step_register_test.go and step_upload_vmx.go
• In addition this PR does some refactoring in order to use the common code with both builders.

Unfortunately this PR is much bigger than I originally anticipated. What would it take to get this PR merged some day?

[spmiller]

I've tested this PR and have had mostly good results. A couple of comments:

• source_path is treated as a being on the remote server. I was expecting it to be on the local filesystem. This is something that could be resolved with some documentation.
• Since it is scripting it needs to quote filename variables as they may have spaces in them. I have fixed all the occurrences I found -- details below
• It leaves the VM on the remote server after it's finished (EDIT: on the disk, not powered on). This could be resolved by migrating the StepExport from the iso builder to common as well.

I've pushed my quoting changes here, but I don't know how to get them into this PR.

[klaamane]

Thanks @spmiller for testing this out! I added a comment to the documentation about the source_path. Also I merged your filename fix.

I will still investigate the last issue you brought up. It should definitely not leave the built sever on. Having the option to export might be a good idea even though that would be a totally new feature for the vmware-vmx builder.

[jimmythedog]

Before this PR is accepted, I'd like to add a few comments, if that is ok, as we would like to use a big part of this PR if it makes it into packer

I, like @spmiller, was also surprised that the source_path pointed to files on the remote server
It got me thinking as to how those files would be put on the server?
Would it be expected that a previous build be left on the server?
Would I have to perform some pre-build work to remotely copy the appropriate files?
If there is a requirement to use remote source files, could we introduce a mutually exclusive remote_source_path property? Or have some sort of source_path_is_remote property?

I also agree with @spmiller that, ideally, the built VM should be exported - preferably exported to output_dir (however, there is an outstanding bug for output_dir when building remotely - see #4447)

Please don't take this as criticism, I'm really excited by this work, and I think it is so close, but it doesn't quite fit our needs as it stands, due to the input/output paths being remote
If I can help in away, please holler (though I'm only just attempting to learn go!)

Keep up the good work!

[marema31]

@jimmythedog, if I manage to achieve the vsphere builder I've begun (see #4526 and #1050 for more information), you will be able to start from a clone operation of an already existing VM. An ovftool import before the packer build may fit your need.

[jimmythedog]

Hi @marema31,
Would this work against ESXi hosts, as we have not gone to the expense of a full on vCentre installation in the build environment (ESXi standalone hosts are now a free download)
Once this PR is complete, I think an ovf builder, built on top of this work, should be a trivial exercise - that is why I'm so interested.
Would be interesting to see if anyone else would use an ova/ovf builder

[marema31]

Hi @jimmythedog
Sorry, but I think it won't work without a licensed version due to VMware restrictions on free ESXi (API are only read-only in free mode). In my lab it's working, but it may be due to the fact that I've installed the ESXi recently.

I'm not sure of this assumption, all references I found on this policies are two or three years old. It could be interresting to give a try.

[klaamane]

Hi @jimmythedog, I'm trying to understand your use ase better. If you have a local source virtual machine why don't you just use the vmware-vmx builder locally? Our use case is that all the VMs are on an ESXi host and we don't want to copy them back and forth just for doing packer builds.

It would not be too hard to support a local vm as a source (and clone it by copying it over the network) if there really is a need for it.

[spmiller]

@klaamane I didn't communicate clearly before. I meant that the builder left the modified VM image on the disk of the remote server, not powered on. Sorry for the confusion!

I can't speak for @jimmythedog, but our use case is that we already had a (free) ESXi server running and we didn't want the expense of installing a local VMWare software package on all of our a build agents. Thus we have a local VM that we want to upload, modify, and download.

Expecting the existing VM to be stored remotely seems reasonable -- if I need to, I can scp it on there myself -- and we can get it back from the remote server with an ovftool post processor step.

[jimmythedog]

Yep, our use case sounds similar to that of @spmiller
We also do not have vmware installed on local clients, and have a free ESXi server on the network that is used for the builds.

Again same as @spmiller, we would have to perform some additional pre-processing to get the files onto the remote server, and some additional post-processing to export the VM

However, where we vary is that we would not be able to leave VMs stored on the server, as we would not have the disk space to store the various branches/versions - we store them on a nexus server

[tjohnston01]

How realistic is it to have the best of both worlds?

• Local or remote source (OVF/OVA/VMX)
• Build on ESXi using SSH or VSphere using the API

[ideologysec]

@jimmythedog https://www.thehumblelab.com/automate-vsphere-template-builds-packer/

Essentially, you have to enable GuestIPHack on the ESXi hypervisor, and configure the firewall to allow connections. 👎

[jimmythedog]

We don't view that as a big problem, as the ESXi server is behind our firewall and, to be honest, using the patched builder that behaves like other ones do (referencing local files) is a big benefit to us. In other words, we can have a free ESXi server, without the expense and complication of the vCentre infrastructure, and our scripts are more or less the same for different builders 👍

…

On 16 Feb 2018 5:44 p.m., "Eric" ***@***.***> wrote: @jimmythedog <https://github.com/jimmythedog> https://www.thehumblelab.com/automate-vsphere-template-builds-packer/ Essentially, you have to enable GuestIPHack on the ESXi hypervisor, and configure the firewall to allow connections. 👎 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4591 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC8mE4xloJi6aj8aJiEpSlz7z9uMVmWpks5tVb5ugaJpZM4MLQSG> .

[ideologysec]

And I've got almost completely the opposite use case - I don't control the hypervisor, it is a big licensed vSphere cluster, I can't make the GuestIPHack changes (nor do I want to), I need all of the files to be stored on the remote datastores, I only build on and for this cluster, and I'd much prefer the -iso as opposed to the -vmx build chain, at least for now.

I'm testing the 2.0 beta, and it seems to work fairly well, thankfully. (No edge cases yet).

[spmiller]

It's good to see the two different use cases articulated so clearly here. Hopefully this goes some way to determining the need to support both scenarios.

FWIW, we have the same use case as @jimmythedog, and we've been using his patch in our production setup for nearly a year now.

[tjohnston01]

+1 for @jimmythedog 's use case. I have an internal-only ESXi server (which I already use to build with the released packer options i.e. vmware-iso). I'm mainly interested in building incrementally on this same setup, although moving to a VSphere API-based approach later would be nice too.

Question for @jimmythedog @spmiller - which version of ESXi are you running? I tried the plugin and ran into some weird issues, but it might have been my fault..need to revisit it soon.

@mwhooker what's the path forward like for getting this added to the main packer release? I guess we need to find some common ground between the different use cases and then rebase that code to the latest..?

[ideologysec]

I believe that the JetBrains vsphere plugin (https://github.com/jetbrains-infra/packer-builder-vsphere) is approaching stable and if I recall correctly (from a comment on an issue somewhere) the plan was to merge that in as an officially supported builder once it’s “done.”

_{Sent with GitHawk}

[spmiller]

@tjohnston01 esxcli system version get reports we're using version 6.0.0.

[collin-bachi-sp]

Any updates on this?

[ideologysec]

Try this: https://github.com/jetbrains-infra/packer-builder-vsphere

…

On Oct 24, 2018, at 12:18, D. Collin Bachi ***@***.***> wrote: Any updates on this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[SwampDragons]

I'm torn on what to do with this PR. It predates my maintainership of Packer, which makes it a little weird for me to manage.

First, I will say that I'm sorry that we on the Packer team have dropped the ball on reviewing this.

Normally, in a situation where I have a stale PR that hasn't been worked on for a year and a half, I just close and say "reopen if you get a chance to resolve conflicts"

But deleting it seems wrong because based on the above conversation, there are people who successfully use this branch for their builds. Originally I had hoped I'd be able to close this when the jetbrains team decided to open a PR for their vsphere builder, but no one has done that yet. I've been holding out hope that someone would either take over this PR or that someone would submit a new one, but neither has happened.

I think for this to get over the finish line we either need:

1. someone to take ownership of of this PR
2. someone to submit a clean PR with these changes
   or
3. someone to submit a PR for the jetbrains plugin.

My preference here is 3) because it's been maintained this entire time and I suspect is more production hardened.

Now, I can close this PR to try to force one of the above options, or I can leave it open in the hopes that someone who has a vested interest in seeing a vsphere builder merged will carry it forward. I think I'm going to leave it, but I'll close if we get to the new year and no one has decided to take it on.

Again, I'm really sorry for how long it took to get this non-answer from the Packer team. I've made reviewing incoming PRs a huge priority, so I feel pretty comfortable saying this won't happen again to a fresh PR without conflicts.

[marema31]

Hello @SwampDragons,

If you are willing to clarify the vSphere packer plugin never-ending story, you can refer to #4526 that contain the most part of the history on the fate of all the attempt. For my part, I've already close my proposal several months back since it was less achieved than the jetbrain proposal (usage of command-line tool for API interaction in the POC I had provided for review, direct usage of the SDK by them, compatibility with vSphere 6.0 and below for me, with 6.5 and above for them).

I'm upset that this hypervisor is not a first-class citizen in Packer, but it seems to be not a priority at this time. Working now for a european cloud provider offering vSphere based product, I can propose you help on testing the selected solution if you need to.

[SwampDragons]

@marema31 I remember closing that PR some time back when you said you'd be happy with the jetbrains one; and then I waited for a jetbrains PR that never came in.

I'm sorry you're upset; you have a right to be. If I were waiting on a vsphere builder that never arrived, I'd be upset too.

I would love help testing the selected solution, whether it's this PR that someone cleans up or the jetbrains code that someone turns into a PR.

And if someone does open a new VSphere PR and you can help me test, I'll schedule it for the next possible release and make getting it in a priority.

[collin-bachi-sp]

Maybe I'm misunderstanding, but I don't see how the jetbrains plugin allows building from a VMX base, or that it replicates the features available in the vmware-vmx builder, the way this PR does. @ideologysec

[SwampDragons]

@collin-bachi-sp looks like that's what their vsphere-clone does, no?

[collin-bachi-sp]

I'm still digging into it, but at first glance it seems to be missing "vmx_data" and "boot_wait" options from vmware-vmx. It also requires VSphere, while vmware-iso running in remote mode (and presumably vmware-vmx in this PR branch) can work with a standalone ESXi server.

[marema31]

@collin-bachi-sp, effectively there is limitation from vmWare on the usage of API (license only), in #4526 (more precisely #issuecomment-280164510), I've resumed what was my understanding of it.

The JetBrain plugin (and the one I had proposed) use the vSphere API that can avoid the direct modification of ESXi configuration, does not need ssh access to ESXi and allow usage of dvs by example. For people that use cloud managed vSphere (as the product I'm working on day job), it is the only solution to use packer. For my test I was able to deploy an nested ESXi and use it with API but only in the trial period.

The 'remote' mode of the vmware plugin consider that the ESXi is just another version of the desktop product, it works well if you are able to control every aspect of the ESXi configuration, this PR will effectively work in this case.

As I said previously, for my opinion, the two kind of plugin correspond to different needs and one may not superseed the other.

@SwampDragons , may my english is too imprecise, I wanted to help this projet (and I always do) I used every day at that time, I proposed a proof of concept based on discussion I had with mainteners and imcomplete because of the fact I was not a developper and does know the golang, it occurs just before the 1.0 release and the team was not able to review and no other helping hands arise. Jetbrain team reviewed my PR, saying that they were developping a better solution and it was decided to wait for their PR. It is not a problem for me, their solutions seems more advanced than the one I proposed (even that some feature I personnaly use was not implemented the last time I tried to use it, but they may be here now). I prefer that the best implementation "wins", but I'd prefer if this implementation could become a "official" one.

[SwampDragons]

@marema31 I understand, thanks.

I slept on this, and the reality is that it's on the Packer team for letting it go this long and therefore it's my job to fix it. I'm going to clear my queue and make this PR's particular implementation (updating our vmware-vmx builder to allow chained builds) a priority; I probably won't get it done in time for the 1.3.2 release but hopefully I will for 1.3.3.

There are a lot of merge conflicts in this branch. I'll see if it's possible to save much of the original code because I want to preserve credit, but I'm a little worried that I'll end up having to rewrite most of it.

As for the vsphere builder/jetbrains approach: you're right that it's a different use case entirely. I'd love to see a PR for it from the jetbrains team, since their builder seems well-maintained and fairly full-featured. After I'm done with this PR, I'll reach out to the jetbrains folks and ask if they mind me pulling their builder into the Packer core.

[SwampDragons]

To follow the rebase process for this branch, see #6927. I decided it was easier to work from a clean PR than force push to this one. I'll leave this PR open until the other is merged.

[SwampDragons]

Builds of a cleaned up/rebased version of this branch are available at #6927; I'd really appreciate it if anyone who has been using this branch for their builds could test out the build on that PR and notify me if anything behaves differently than they expect. I'm hoping to merge it early next week.

[SwampDragons]

The rebased version of this PR has been merged. Thank you for your patience on this one.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.