Problem uploading iso/floppy to ESX host

[pmanno]

Overview of the Issue

Whenever I try to build more than one VM at at time, even on two different build servers, I get an error:
vmware-iso: Error uploading file: Process exited with status 1

Reproduction Steps

Attempt to build more than 1 VM at a time.

Packer version

1.6.0

Simplified Packer Buildfile

gist

Operating system and Environment details

Build Server packer is run on: Ubuntu 18.04.4 LTS

Log Fragments and crash.log files

gist

[SwampDragons]

Hi, thanks for reaching out. I'm trying to reproduce this but I'm not able to. Your example packer template only defines one builder. Do you run that template twice from the same machine? Or does your real template have two builders in it? Are they uploading the same floppy files/iso files, or are the files being uploaded by the parallel builds different?

[SwampDragons]

We did resolve a checksum error in uploading to esx here: #9584. That patch released with v1.6.1. Can you try that version and let me know if this is still a problem?

[pmanno]

Tested 1.6.1. Still getting that same error.

[pmanno]

Hi, thanks for reaching out. I'm trying to reproduce this but I'm not able to. Your example packer template only defines one builder. Do you run that template twice from the same machine? Or does your real template have two builders in it? Are they uploading the same floppy files/iso files, or are the files being uploaded by the parallel builds different?

Hello,
I have a jenkins server with two build servers that have packer installed. I have a throttle rule setup in Jenkins to only run one packer build per worker node but allow two total concurrent builds (one from each node). When I queue one build, then queue another, the first build succeeds, but the second one (started just a few milliseconds after the first) fails with this error. In each case, the identical template is being executed. Typically this is the iso url and checksum

    return "http://cdimage.ubuntu.com/ubuntu/releases/bionic/release/ubuntu-18.04.4-server-amd64.iso"
    return "e2ecdace33c939527cbc9e8d23576381c493b071107207d2040af72595f8990b"

And both instances of the packer run have that set.

For floppy files, again, both instances have the same files.

Log here: https://gist.github.com/pmanno/0c3e4a795690bd5149b3132f8555d3e9

[pmanno]

So this is only an issue when setting the datastore (or probably packer_cache) dir to be a location on shared storage. If I use discrete datastores on each host, I can build in parallel without error. So I'm guessing the first process starts uploading to the cache, and the second one tries to do the same but the file is locked by the first process.

[SwampDragons]

I just took a closer look at this and I believe you're right -- the iso upload is failing because the independent builders are trying to upload at the same time. I could look at implementing a file lock, but that'll probably cause issues in a situation where the builder crashes while trying to upload. The best solution is probably to just run a single builder once to make sure the iso has been uploaded to the cache (making sure that https://www.packer.io/docs/builders/vmware-iso#cleanup_remote_cache is false) and then next time you run your milliseconds-apart-builders the iso is already there so the builders won't race each other trying to upload it.

[pmanno]

I'm not sure even that would work because I think they would run into an error when multiple VMs try to mount the same iso to build from?

[SwampDragons]

If that's the case you're going to have to just run the builds sequentially or from different datastores; is there a reason you're running identical builds at the same time? There may be a better pipeline for you.

[ghost]

This issue has been automatically migrated to hashicorp/packer-plugin-vmware#3 because it looks like an issue with that plugin. If you believe this is not an issue with the plugin, please reply to hashicorp/packer-plugin-vmware#3.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.