Packer crash while uploading to vsphere #2595
Labels
Comments
|
Gist with details: https://gist.github.com/m1ndful/02d34bc9950d62789b9a |
cbednarski
added
bug
builder/vmware-esxi
post-processor/vsphere
and removed
builder/vmware-esxi
labels
|
Thanks for the report! Your crash log says |
|
12GB memory and 2 CPUs. Machine is VM inside vSphere cloud. |
|
Adding another log from #2656 |
|
The machine that is doing the build is. |
|
It is funny that I am getting this error given that the machine is a physical machine with 16gb of ram but the log says I am using 64gb. |
Yeah I agree. It could be using swap or virtual memory, but I'm guessing there's some kind of memory leak or a loop that doesn't terminate correctly. On an internals note, I'm not exactly sure how the go runtime allocates heap -- perhaps in fixed-size blocks or by doubling current usage, etc. However, it looks like it is failing on a malloc call to allocate 64gb, but may not have actually used that much yet. |
|
I looked at the logs and I don't see anything particularly exciting. The build crashes from malloc fail just as it tries to run the ovftool. You can see this in the second go routine which fails malloc while trying to make the exec call. This is a fairly boring part of code since most of the work here happens in another program. I noticed some go routines are hanging around with msgpack calls. These look suspicious. I have not yet confirmed by my guesses at this point:
The pointer offsets look to be very similar in both stack traces but I don't see a significant spread in any of the go routines, so I'm curious whether the offending go routine has exited already but the memory has not been garbage collected for some reason (that would suggest a leak). |
|
Another Debug using Fusion 8 Pro which allows for direct push to vSphere . But still not working as designed. |
|
+1 on getting this issue resolved |
|
I would agree. Thanks |
|
Any updates on this issue ? I'm getting same out of memory error when using the vsphere post-processor. |
|
Any update on this issue? |
|
I have not dug into this or repro'd yet, but we have changed the msgpack implementation and fixed a lot of race conditions here. I'm curious if this might be incidentally fixed now. We will be fixing this for sure by 0.9. Sorry to keep you waiting! |
|
Looking at this we'll need to bring up a full vsphere to repro. But looking at the code the output of the ovftool run does get buffered into memory. It's not clear how that could account for GB of data to run the process out of memory but it does mesh with the call stacks. @m1ndful you might want to try running that ovftool command (with the exact command line) and see what is output. Something like this: |
|
Have you verified that the account you're using with packer/ovftool does indeed have all the permissions it needs as well as it can login cleanly? @m1ndful What version of vSphere vCenter do you have running? Make sure you are using |
|
I just ran into this myself the other day, getting ready to try using the ovftool as mentioned by @markpeek |
|
Ok, so digging into it it looks like --network is getting added to the ovftool call, and at least in my instance I wasn't defining that in the template file, so once I added a valid vm_network it worked fine. @m1ndful, I looked at your gist and it looks like you have the same situation I did, so try adding that value. The documentation says that vm_network is optional, but for whatever reason it's getting added to the ovftool execution with no value and I suspect vSphere doesn't like that. If you run the ovftool parameters from the Packer debug log without vm_network defined, the ovftool gives an error with output stating a valid network is required and what the options are for your target. Maybe Packer could capture that response and provide that useful information instead? Looking at the debug output from @ubailb1 I notice that he's trying to push a Mac OS VM to vSphere, are you running ESXi on that Mac Mini? |
|
There is definitely an issue with the vsphere post-processor and eating up memory if the ovftool does something it doesn't expect. For example, I tried running the same build as before (with vm_network defined) on my Mac, the post-processor started eating memory and would have continued if I didn't kill the process. Taking the ovftool parameters from the debug logs and running it manually prompted me to accept the SSL fingerprint. Once I did that and re-ran the build it uploaded to vsphere just fine. |
|
I think this got fixed as part of #3321 Please open a new issue if you continue to encounter this |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Crashed during uploading to vSphere.
==> vmware-iso: Running post-processor: vsphere vmware-iso (vsphere): Uploading output-vmware-iso/packer-vmware-iso.vmx to vSphere panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x4ae2c0]The text was updated successfully, but these errors were encountered: