proxmox: Error updating template: 500 got no worker upid - start worker failed

[hc-github-team-packer]

This issue was originally opened by @dnv in hashicorp/packer#11149 and has been migrated to this repository. The original issue description is below.

---

Overview of the Issue

Roughly 2/3 of the time I am building my templates, the build errors out in the final stages, succesfully creating the actual template regardless, but with its "Packer ephemeral build VM" description and name intact. At first it "felt" like I got rid of the issue by adding the final sync command to the shell builder, but this was apparently transient/imaginary, as after 3-4 succesfull builds with no errors in a row I started experiencing the problem again.

I have now run the build several times in a row changing nothing in-between and sometimes I do get the error and sometimes not. Having done some additional testing I can now see that this has something to do with creating new VMs/templates and deleting them. The error does not seem to ever occur in the following scenario: 6 VMs are running, I run Packer to create a new template, it suceeeds, I delete the template and create it anew. But somewhere in the logic, there is a problem that manifests itself if you create a template, create a new VM from said template, keep it running, delete something with a lower vm_id and try creating a new template with Packer.

Packer version

Packer version: 1.7.3 [go1.16.5 linux amd64]

Simplified Packer Buildfile

https://gist.github.com/dnv/7e6778870f070b125cd6fade7f98aa33

Operating system and Environment details

Running Packer on RHEL 8.4 VM
Proxmox Virtual Environment 6.4-8

Log Fragments and crash.log files

The relevant part of the log is:

2021/07/15 18:14:20 [INFO] (telemetry) ending shell
==> proxmox: Stopping VM
==> proxmox: Converting VM to template
2021/07/15 18:14:22 packer-builder-proxmox plugin: template_id: 108
==> proxmox: Error updating template: 500 got no worker upid - start worker failed
==> proxmox: Provisioning step had errors: Running the cleanup provisioner, if present...
==> proxmox: Stopping VM
==> proxmox: Deleting VM
2021/07/15 18:14:22 [INFO] (telemetry) ending proxmox
==> Wait completed after 5 minutes 47 seconds
2021/07/15 18:14:22 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2021/07/15 18:14:22 machine readable: proxmox,error []string{"Error updating template: 500 got no worker upid - start worker failed"}
==> Builds finished but no artifacts were created.
Build 'proxmox' errored after 5 minutes 47 seconds: Error updating template: 500 got no worker upid - start worker failed

==> Wait completed after 5 minutes 47 seconds

==> Some builds didn't complete successfully and had errors:
--> proxmox: Error updating template: 500 got no worker upid - start worker failed

==> Builds finished but no artifacts were created.

[besmirzanaj]

reporting same here

==> proxmox-iso: Provisioning with shell script: /tmp/packer-shell348222852
    proxmox-iso: vps3.cloudalbania.com -- Complete
==> proxmox-iso: Stopping VM
==> proxmox-iso: Converting VM to template
==> proxmox-iso: Error converting VM to template: 500 got no worker upid - start worker failed
==> proxmox-iso: Provisioning step had errors: Running the cleanup provisioner, if present...
==> proxmox-iso: Stopping VM
==> proxmox-iso: Deleting VM
Build 'proxmox-iso' errored after 12 minutes 46 seconds: Error converting VM to template: 500 got no worker upid - start worker failed

==> Wait completed after 12 minutes 46 seconds

==> Some builds didn't complete successfully and had errors:
--> proxmox-iso: Error converting VM to template: 500 got no worker upid - start worker failed

==> Builds finished but no artifacts were created.
```

[c-romeo]

Same issue here, from 6 VMs completed only 5.
Terraform configuration is pretty much unchanged since a dozen of runs.

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.�[0m

�[0m�[1mInitializing provider plugins...�[0m
- Reusing previous version of hashicorp/tls from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of telmate/proxmox from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Using previously-installed hashicorp/aws v3.65.0
- Using previously-installed telmate/proxmox v2.9.0
- Using previously-installed hashicorp/local v2.1.0
- Using previously-installed hashicorp/tls v3.1.0

�[0m�[1m�[32mTerraform has been successfully initialized!�[0m�[32m�[0m
.....
�[0m�[1mproxmox_vm_qemu.kube-worker["kube-worker2"]: Creation complete after 3m3s [id=pve1/qemu/205]�[0m
�[31m╷�[0m�[0m
�[31m│�[0m �[0m�[1m�[31mError: �[0m�[0m�[1m500 got no worker upid - start worker failed�[0m
�[31m│�[0m �[0m
�[31m│�[0m �[0m�[0m  with proxmox_vm_qemu.kube-worker["kube-worker3"],
�[31m│�[0m �[0m  on workers.tf line 16, in resource "proxmox_vm_qemu" "kube-worker":
�[31m│�[0m �[0m  16: resource "proxmox_vm_qemu" "kube-worker" �[4m{�[0m�[0m
�[31m│�[0m �[0m
�[31m╵�[0m�[0m

Nothing special at line 16

I've just restarted the Jenkins job and it finished the vm creation without issue.

+ terraform apply '-input=false' tfplan
�[0m�[1mproxmox_vm_qemu.kube-worker["kube-worker3"]: Destroying... [id=pve1/qemu/206]�[0m�[0m
�[0m�[1mproxmox_vm_qemu.kube-worker["kube-worker3"]: Destruction complete after 2s�[0m
�[0m�[1mproxmox_vm_qemu.kube-worker["kube-worker3"]: Creating...�[0m�[0m
....
�[0m�[1mproxmox_vm_qemu.kube-worker["kube-worker3"]: Creation complete after 1m47s [id=pve1/qemu/206]�[0m
Releasing state lock. This may take a few moments...
�[0m�[1m�[32m
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

pve-manager/7.0-11/63d82f4e (running kernel: 5.11.22-4-pve)