Packer on Jenkins with vSphere/Ovftool CPU Pegged - IOCTL Error in strace

[rvalente]

Environment Details

Ubuntu 14.04.3
Jenkins 1.636
Packer 0.8.6
VMware Workstation 11.1.2
Ansible 1.9.4
Ovftool 4.1.0

Build Flow

vmware-iso > script provisioner > ansible provisioner > vsphere post-processor

Reproduce

1. Jenkins Job Runs PACKER_LOG=1 packer build template.json upon upstream code change.
2. Build Completes Successfully. Hangs on "Uploading output-vmware.vmx"
3. Never finishes, and the ovftool process has the CPU pegged with an "Inappropriate ioctl for device" error.
4. strace -p $PID of jenkins build and its streaming the ioctl error.

NOTE: This also occurs when run manually on my Mac so this isn't a Jenkins specific error to the best of my knowledge. The error on the Mac side is IOCTL Err#19

I copied the ovftool command from the debug log, entered it manually, and it works/uploads just fine.

Gist

https://gist.github.com/rvalente/5801d5296d476d66e6d6

[cbednarski]

Thanks for the report! I'm curious if this is somehow related to #2595. Can you try this build using packer built from master? Also if you can share a minimally-reproducible packer template + scripts that will help me fix this. Thanks!

[rvalente]

@cbednarski Absolutely, I will put a repo up on my github account and link it back here so you can clone and kick it off. Should have that done soon. In the meantime I will kick off a packer build from master. Another thing I am trying now is using disk mode 4 and vmx9 (I was on vmx10).

[rvalente]

Issue #2595 doesn't look to be identical, but my host would run out of memory eventually. There might be a correlation there. Side note: I had a FreeBSD image build and upload clean this morning...

[cbednarski]

FreeBSD images are typically pretty small compared to ubuntu / cent so I'm curious if it squeaks by for that reason. There's definitely at least one bug here but if both are related to ovftool the fix(es) are likely in the same place.

[markpeek]

I'd be curious if it reproduced after removing the Stdout buffering here.

[rvalente]

Here is my upstream cleaned repo: https://github.com/rvalente/packer-trusty64

[rvalente]

On the Jenkins build server here is the size of the flat VMDK: -rw------- 1 jenkins 20G Nov 9 17:26 disk-flat.vmdk

[rvalente]

Ok, I am now running Packer version: 0.8.7 dev 0660c703ad7cc93a81a1038be2812d5d61cb459f and the post-processor failed with the following debug output. I had the same symptoms of running out of swap on the node.

2015/11/09 21:49:01 ui: �[1;32m==> vmware-iso: Running post-processor: vsphere�[0m
2015/11/09 21:49:01 ui: �[0;32m    vmware-iso (vsphere): Uploading output-vmware-iso/packer-vmware-iso.vmx to vSphere�[0m
2015/11/09 21:49:01 packer: 2015/11/09 21:49:01 Starting ovftool with parameters: --noSSLVerify=true --acceptAllEulas --name=trusty64-vmx-2.0.20 --datastore=DS_Cluster --diskMode=thick --network=Template_Net --vmFolder=Templates output-vmware-iso/packer-vmware-iso.vmx vi://packer%40vpshere.local:PASSWORD@vcenter.domain.tld/DATACENTER/host/CLUSTER/Resources/Bronze
2015/11/09 23:12:33 ui: �[1;32m==> vmware-iso: Running post-processor: vsphere�[0m
2015/11/09 23:12:33 /home/username/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/09 23:12:33 ui: �[0;32m    vmware-iso (vsphere): Uploading output-vmware-iso/packer-vmware-iso.vmx to vSphere�[0m
2015/11/09 23:12:33 packer: 2015/11/09 23:12:33 Starting ovftool with parameters: --noSSLVerify=true --acceptAllEulas --name=trusty64-vmx-2.0.20 --datastore=DS_Cluster --diskMode=thick --network=Template_Net --vmFolder=Templates output-vmware-iso/packer-vmware-iso.vmx vi://packer%40vpshere.local:PASSWORD@vcenter2.domain.tld/DATACENTER/host/CLUSTER/Resources/Bronze
2015/11/10 00:35:54 [ERR] yamux: keepalive failed: session shutdown
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/10 00:35:54 Deleting original artifact for build 'vmware-iso'
2015/11/10 00:35:54 ui error: �[1;31mBuild 'vmware-iso' errored: 2 error(s) occurred:

* Post-processor failed: unexpected EOF
* Post-processor failed: unexpected EOF�[0m
2015/11/10 00:35:54 Builds completed. Waiting on interrupt barrier...
2015/11/10 00:35:54 machine readable: error-count []string{"1"}
2015/11/10 00:35:54 ui error: 
==> Some builds didn't complete successfully and had errors:
2015/11/10 00:35:54 machine readable: vmware-iso,error []string{"2 error(s) occurred:\n\n* Post-processor failed: unexpected EOF\n* Post-processor failed: unexpected EOF"}
2015/11/10 00:35:54 ui error: --> vmware-iso: 2 error(s) occurred:

* Post-processor failed: unexpected EOF
* Post-processor failed: unexpected EOF
2015/11/10 00:35:54 ui: 
==> Builds finished but no artifacts were created.
2015/11/10 00:35:54 waiting for all plugin processes to complete...
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/10 00:35:54 [ERR] Error decoding response stream 8: EOF
2015/11/10 00:35:54 [ERR] Error decoding response stream 7: EOF
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited
2015/11/10 00:35:54 /home/user/go/src/github.com/mitchellh/packer/bin/packer: plugin process exited

[cbednarski]

@rvalente Thanks for testing with master. That rules out the possibility that this is a side effect of a race condition.

[rvalente]

@cbednarski You're welcome! Anything I can do to help out. I am going to set the disk type back to 0 but leave skip_compaction set to true. If there is anything else you want me to test I am happy to do so. I am no Go expert yet by any means though...

[rvalente]

I just ran a FreeBSD build that has two post-processors. I ship the build to multiple vCenters, hence the multiple post-processors. Got the following error:

Progress: 99%
Transfer Completed                    
    vmware-iso (vsphere): Completed successfully
    vmware-iso (vsphere):
Build 'vmware-iso' errored: 1 error(s) occurred:

* Post-processor failed: unexpected EOF

It successfully uploaded to one VC, but not the other. Not sure if this helps or not.

[~]% packer --version
0.8.7

[rvalente]

Ok, so this is clearly an issue with ovftool I got them to both succeed reliably with vm.swappiness = 1 that said, I don't think that was the problem. ovftool never alerted that the password was expired on the VC where it was failing. I logged in as the packer account and it presented me with a reset password prompt.

Too bad it doesn't actually throw something more useful so it can be exposed in packer. That would have saved me quite a bit of time.

😢

[mwhooker]

a lot has changed since this was open, and I suspect it's been fixed. Please open a new issue if you continue to encounter this