Skip to content
This repository has been archived by the owner on Mar 8, 2023. It is now read-only.

vsphere-iso: unexpected EOF #87

Open
lifeisfoo opened this issue Apr 21, 2018 · 24 comments
Open

vsphere-iso: unexpected EOF #87

lifeisfoo opened this issue Apr 21, 2018 · 24 comments
Labels
problem

Comments

@lifeisfoo
Copy link

I'm trying to creare a VM from iso using vsphere-iso 2.0-beta4 with vCenter Server 6.0.0, but I receive this error:

$ ./packer -version
1.2.2

$ ./packer build -debug packer-iso-template.json
Debug mode enabled. Builds will not be parallelized.
vsphere-iso output will be in this color.

==> vsphere-iso: Pausing after run of step 'StepConnect'. Press enter to continue. 
==> vsphere-iso: Creating VM...
==> vsphere-iso: Pausing before cleanup of step 'StepConnect'. Press enter to continue. 
Build 'vsphere-iso' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso: unexpected EOF

==> Builds finished but no artifacts were created.

I'm using a configuration adapted from the ubuntu example:

{
  "builders": [
    {
      "type": "vsphere-iso",

      "vcenter_server": "vcenter.example.com",
      "username": "user@example.com",
      "password": "xxx,

      "guest_os_type": "ubuntu64Guest",
      "disk_controller_type": "pvscsi",
      "disk_size": 10,
      "iso_paths": "[Equallogic 1] ubuntu-16.04.4-server-amd64.iso",

      "datacenter": "datacenter 1",
      "cluster": "Cluster3",
      "resource_pool": "resource-pool",
      "datastore": "Compellent pr1",
      "folder": "Company/pr1",
      
      "vm_name": "example-ubuntu",
      "boot_wait": "5s",

      "CPUs": 1,
      "CPU_limit": 1000,
      "network": "pub",
      "network_card": "vmxnet3",
      "RAM": 1024,

      "ssh_username": "jetbrains",
      "ssh_password": "jetbrains",

      "floppy_files": [
        "{{template_dir}}/preseed.cfg"
      ],
      "boot_command": [
        "<enter><wait><f6><wait><esc><wait>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs>",
        "/install/vmlinuz",
        " initrd=/install/initrd.gz",
        " priority=critical",
        " locale=it_IT",
        " file=/media/preseed.cfg",
        "<enter>"
      ]
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": [ "echo hello" ]
    }
  ]
}

The preseed.cfg file is in the same directory of the template, and it's content is unchanged. I also tested without floppy_files and boot_command but I receive the same errror.

It's not a connection problem because I can create VMs using vsphere-clone 2.0-beta4.

I can't see any recent task or error in vSphere UI.
@mkuzmin
Copy link
Contributor

mkuzmin commented May 3, 2018

The error happens at VM creation moment.
I'm pretty sure vCenter shows an event with details for that.

I'd suggest two tests to localize the root cause:

  1. Create VM at different host (maybe without clusters)
  2. Use vsphere-clone builder on the same cluster.

@mkuzmin mkuzmin added the problem label May 3, 2018
@lifeisfoo
Copy link
Author

I don't have access to vCenter, only to vShpere and here I don't see any useful log or event.

  1. I can't create a VM at a different host because I'm using on shared vSphere where I can use only that cluster with that resource pool.

  2. The vshpere-clone builder works nicely on the same cluster.

@mkuzmin
Copy link
Contributor

mkuzmin commented May 4, 2018

Then start with a config with a minimal set of parameters, and add them one by one.

{
  "builders": [
    {
      "type": "vsphere-iso",

      "vcenter_server":      "vcenter.vsphere65.test",
      "username":            "root",
      "password":            "jetbrains",
      "insecure_connection": "true",

      "vm_name": "example-ubuntu-{{timestamp}}",
      "host":     "esxi-1.vsphere65.test",

      "disk_size": 1024,

      "communicator": "none"
    }
  ]
}

@mkuzmin
Copy link
Contributor

mkuzmin commented May 4, 2018

need to warn you in advance. book_command is not compatible with vSphere 6.0 (#63).

@lifeisfoo
Copy link
Author

I receive the same error with this minimal file:

{
  "builders": [
    {
      "type": "vsphere-iso",

      "vcenter_server": "vcenter.example.com",
      "username": "user@example.com",
      "password": "xxx,

      "vm_name": "example-ubuntu-{{timestamp}}",

      "cluster": "Cluster3",
      "resource_pool": "resource-pool",
      "datastore": "Compellent pr1",

      "network": "pub",
      "network_card": "vmxnet3",

      "disk_size": 1024,

      "communicator": "none"
    }
  ]
}

I can't use a host directly and also the network is required, otherwise I receive this error:

error creating vm: default network resolves to multiple instances, please specify

@lifeisfoo
Copy link
Author

Does the unexpected EOF error could be caused by insufficient permissions of the vSphere user?
Does there exist a list of required permissions for vsphere-iso builder?

@mkuzmin
Copy link
Contributor

mkuzmin commented May 6, 2018

There is a discussion about permissions in #97

@mkuzmin
Copy link
Contributor

mkuzmin commented May 13, 2018 •
edited
Loading

I've just seen a similar error, but in a different piece of the code.
Please post extended output using PACKER_LOG=1 environment variable.

@lifeisfoo
Copy link
Author 

$ PACKER_LOG=1 ./packer build minimal-iso.json                
2018/05/13 23:22:50 [INFO] Packer version: 1.2.2
2018/05/13 23:22:50 Packer Target OS/Arch: darwin amd64
2018/05/13 23:22:50 Built with Go Version: go1.10
2018/05/13 23:22:50 [DEBUG] Discovered plugin: vsphere-clone = /Users/alessandro/Dev/packer/packer-builder-vsphere-clone.macos
2018/05/13 23:22:50 [DEBUG] Discovered plugin: vsphere-iso = /Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos
2018/05/13 23:22:50 Detected home directory from env var: /Users/alessandro
2018/05/13 23:22:50 [DEBUG] Discovered plugin: vsphere-clone = /Users/alessandro/Dev/packer/packer-builder-vsphere-clone.macos
2018/05/13 23:22:50 [DEBUG] Discovered plugin: vsphere-iso = /Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos
2018/05/13 23:22:50 Using internal plugin for parallels-pvm
2018/05/13 23:22:50 Using internal plugin for file
2018/05/13 23:22:50 Using internal plugin for hyperv-vmcx
2018/05/13 23:22:50 Using internal plugin for amazon-instance
2018/05/13 23:22:50 Using internal plugin for amazon-ebs
2018/05/13 23:22:50 Using internal plugin for amazon-ebssurrogate
2018/05/13 23:22:50 Using internal plugin for openstack
2018/05/13 23:22:50 Using internal plugin for parallels-iso
2018/05/13 23:22:50 Using internal plugin for profitbricks
2018/05/13 23:22:50 Using internal plugin for amazon-chroot
2018/05/13 23:22:50 Using internal plugin for cloudstack
2018/05/13 23:22:50 Using internal plugin for oneandone
2018/05/13 23:22:50 Using internal plugin for oracle-classic
2018/05/13 23:22:50 Using internal plugin for virtualbox-iso
2018/05/13 23:22:50 Using internal plugin for vmware-vmx
2018/05/13 23:22:50 Using internal plugin for digitalocean
2018/05/13 23:22:50 Using internal plugin for null
2018/05/13 23:22:50 Using internal plugin for oracle-oci
2018/05/13 23:22:50 Using internal plugin for triton
2018/05/13 23:22:50 Using internal plugin for docker
2018/05/13 23:22:50 Using internal plugin for googlecompute
2018/05/13 23:22:50 Using internal plugin for scaleway
2018/05/13 23:22:50 Using internal plugin for azure-arm
2018/05/13 23:22:50 Using internal plugin for lxc
2018/05/13 23:22:50 Using internal plugin for ncloud
2018/05/13 23:22:50 Using internal plugin for qemu
2018/05/13 23:22:50 Using internal plugin for amazon-ebsvolume
2018/05/13 23:22:50 Using internal plugin for hyperv-iso
2018/05/13 23:22:50 Using internal plugin for virtualbox-ovf
2018/05/13 23:22:50 Using internal plugin for vmware-iso
2018/05/13 23:22:50 Using internal plugin for alicloud-ecs
2018/05/13 23:22:50 Using internal plugin for lxd
2018/05/13 23:22:50 Using internal plugin for ansible-local
2018/05/13 23:22:50 Using internal plugin for chef-solo
2018/05/13 23:22:50 Using internal plugin for converge
2018/05/13 23:22:50 Using internal plugin for powershell
2018/05/13 23:22:50 Using internal plugin for ansible
2018/05/13 23:22:50 Using internal plugin for chef-client
2018/05/13 23:22:50 Using internal plugin for puppet-server
2018/05/13 23:22:50 Using internal plugin for shell
2018/05/13 23:22:50 Using internal plugin for salt-masterless
2018/05/13 23:22:50 Using internal plugin for windows-shell
2018/05/13 23:22:50 Using internal plugin for file
2018/05/13 23:22:50 Using internal plugin for puppet-masterless
2018/05/13 23:22:50 Using internal plugin for shell-local
2018/05/13 23:22:50 Using internal plugin for windows-restart
2018/05/13 23:22:50 Using internal plugin for alicloud-import
2018/05/13 23:22:50 Using internal plugin for checksum
2018/05/13 23:22:50 Using internal plugin for manifest
2018/05/13 23:22:50 Using internal plugin for amazon-import
2018/05/13 23:22:50 Using internal plugin for artifice
2018/05/13 23:22:50 Using internal plugin for shell-local
2018/05/13 23:22:50 Using internal plugin for vsphere-template
2018/05/13 23:22:50 Using internal plugin for atlas
2018/05/13 23:22:50 Using internal plugin for compress
2018/05/13 23:22:50 Using internal plugin for docker-push
2018/05/13 23:22:50 Using internal plugin for googlecompute-export
2018/05/13 23:22:50 Using internal plugin for vagrant
2018/05/13 23:22:50 Using internal plugin for vagrant-cloud
2018/05/13 23:22:50 Using internal plugin for docker-import
2018/05/13 23:22:50 Using internal plugin for docker-save
2018/05/13 23:22:50 Using internal plugin for docker-tag
2018/05/13 23:22:50 Using internal plugin for vsphere
2018/05/13 23:22:50 Detected home directory from env var: /Users/alessandro
2018/05/13 23:22:50 Attempting to open config file: /Users/alessandro/.packerconfig
2018/05/13 23:22:50 [WARN] Config file doesn't exist: /Users/alessandro/.packerconfig
2018/05/13 23:22:50 Packer config: &{DisableCheckpoint:false DisableCheckpointSignature:false PluginMinPort:10000 PluginMaxPort:25000 Builders:map[vsphere-clone:/Users/alessandro/Dev/packer/packer-builder-vsphere-clone.macos amazon-instance:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-instance qemu:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-qemu hyperv-iso:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-hyperv-iso oneandone:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-oneandone oracle-classic:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-oracle-classic digitalocean:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-digitalocean vsphere-iso:/Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos parallels-pvm:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-parallels-pvm file:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-file openstack:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-openstack parallels-iso:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-parallels-iso oracle-oci:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-oracle-oci triton:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-triton lxc:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-lxc ncloud:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-ncloud vmware-iso:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-vmware-iso alicloud-ecs:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-alicloud-ecs amazon-ebssurrogate:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebssurrogate amazon-chroot:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-chroot cloudstack:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-cloudstack docker:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-docker scaleway:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-scaleway null:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-null lxd:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-lxd hyperv-vmcx:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-hyperv-vmcx profitbricks:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-profitbricks azure-arm:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-azure-arm virtualbox-ovf:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-ovf vmware-vmx:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-vmware-vmx amazon-ebsvolume:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebsvolume amazon-ebs:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebs virtualbox-iso:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-iso googlecompute:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-googlecompute] PostProcessors:map[artifice:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-artifice shell-local:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-shell-local compress:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-compress docker-push:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-push vagrant-cloud:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant-cloud manifest:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-manifest amazon-import:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-amazon-import googlecompute-export:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-googlecompute-export docker-tag:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-tag alicloud-import:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-alicloud-import checksum:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-checksum atlas:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-atlas docker-import:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-import docker-save:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-save vsphere-template:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vsphere-template vagrant:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant vsphere:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vsphere] Provisioners:map[powershell:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-powershell file:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-file chef-solo:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-chef-solo ansible:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-ansible converge:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-converge chef-client:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-chef-client puppet-server:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-puppet-server salt-masterless:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-salt-masterless puppet-masterless:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-puppet-masterless shell-local:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell-local windows-restart:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-windows-restart ansible-local:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-ansible-local windows-shell:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-windows-shell shell:/Users/alessandro/Dev/packer/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell]}
2018/05/13 23:22:50 Detected home directory from env var: /Users/alessandro
2018/05/13 23:22:50 Setting cache directory: /Users/alessandro/Dev/packer/packer_cache
2018/05/13 23:22:50 Detected home directory from env var: /Users/alessandro
2018/05/13 23:22:50 Loading builder: vsphere-iso
2018/05/13 23:22:50 Creating plugin client for path: /Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos
2018/05/13 23:22:50 Starting plugin: /Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos []string{"/Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos"}
2018/05/13 23:22:50 Waiting for RPC address for: /Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos
2018/05/13 23:22:50 packer-builder-vsphere-iso.macos: 2018/05/13 23:22:50 Plugin minimum port: 10000
2018/05/13 23:22:50 packer-builder-vsphere-iso.macos: 2018/05/13 23:22:50 Plugin maximum port: 25000
2018/05/13 23:22:50 packer-builder-vsphere-iso.macos: 2018/05/13 23:22:50 Plugin address: unix /var/folders/hj/4bv2wd4n6bvb3xh0wy6_4kv80000gn/T/packer-plugin433217406
2018/05/13 23:22:50 packer-builder-vsphere-iso.macos: 2018/05/13 23:22:50 Waiting for connection...
2018/05/13 23:22:50 packer-builder-vsphere-iso.macos: 2018/05/13 23:22:50 Serving a plugin connection...
2018/05/13 23:22:50 ui: vsphere-iso output will be in this color.
2018/05/13 23:22:50 ui: 
2018/05/13 23:22:50 Build debug mode: false
2018/05/13 23:22:50 Force build: false
2018/05/13 23:22:50 On error: 
2018/05/13 23:22:50 Preparing build: vsphere-iso
vsphere-iso output will be in this color.

2018/05/13 23:22:50 Waiting on builds to complete...
2018/05/13 23:22:50 Starting build run: vsphere-iso
2018/05/13 23:22:50 Running builder: vsphere-iso
2018/05/13 23:22:50 [INFO] (telemetry) Starting builder vsphere-iso
2018/05/13 23:22:51 ui: ==> vsphere-iso: Creating VM...
==> vsphere-iso: Creating VM...
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: Build 'vsphere-iso' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso: unexpected EOF

==> Builds finished but no artifacts were created.
panic: runtime error: invalid memory address or nil pointer dereference
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1798ae9]
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: 
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: goroutine 81 [running]:
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/vmware/govmomi/object.DistributedVirtualPortgroup.EthernetCardBackingInfo(0xc42019eb40, 0x2b, 0xc420300b00, 0xc4204af2e0, 0x1b, 0xc4204af440, 0x11, 0x1e36a80, 0xc42018c2c0, 0x1e32e80, ...)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /mnt/agent/work/e79dadccf0bca3f7/src/github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/vmware/govmomi/object/distributed_virtual_portgroup.go:47 +0x249
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/driver.addNetwork(0xc4201915f0, 0xc42018d980, 0x3, 0x4, 0xc4206717a0, 0xc42018d980, 0x3, 0x4, 0x0, 0x0)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /mnt/agent/work/e79dadccf0bca3f7/src/github.com/jetbrains-infra/packer-builder-vsphere/driver/vm.go:445 +0xaa
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/driver.(*Driver).CreateVM(0xc4201915f0, 0xc4206717a0, 0xe, 0x1c8c8a0, 0xc4201915f0)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /mnt/agent/work/e79dadccf0bca3f7/src/github.com/jetbrains-infra/packer-builder-vsphere/driver/vm.go:116 +0x295
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/iso.(*StepCreateVM).Run(0xc420196028, 0x1e36a80, 0xc42018c2c0, 0x1e35a40, 0xc4201902a0, 0x0)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /mnt/agent/work/e79dadccf0bca3f7/src/github.com/jetbrains-infra/packer-builder-vsphere/iso/step_create.go:65 +0x3c7
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/helper/multistep.(*BasicRunner).Run(0xc42018c280, 0x1e35a40, 0xc4201902a0)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /mnt/agent/work/e79dadccf0bca3f7/src/github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/helper/multistep/basic_runner.go:72 +0x22f
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/iso.(*Builder).Run(0xc4201a50a0, 0x1e378c0, 0xc420192120, 0x1e32840, 0xc4201940c0, 0x1e36d00, 0xc420196018, 0x10b4b0c, 0x1a44ba0, 0x1a44ba0, ...)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /mnt/agent/work/e79dadccf0bca3f7/src/github.com/jetbrains-infra/packer-builder-vsphere/iso/builder.go:96 +0x9d1
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/packer/rpc.(*BuilderServer).Run(0xc4201a50c0, 0x1, 0xc4203be01c, 0x0, 0x0)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /mnt/agent/work/e79dadccf0bca3f7/src/github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/packer/rpc/builder.go:94 +0x1ce
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: reflect.Value.call(0xc420184660, 0xc4201aa150, 0x13, 0x1d215e2, 0x4, 0xc42007ff18, 0x3, 0x3, 0xc420366000, 0xc420388b08, ...)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /usr/local/go/src/reflect/value.go:447 +0x969
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: reflect.Value.Call(0xc420184660, 0xc4201aa150, 0x13, 0xc420391f18, 0x3, 0x3, 0x1, 0x1141d3e, 0x1)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /usr/local/go/src/reflect/value.go:308 +0xa4
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: net/rpc.(*service).call(0xc42019c700, 0xc4201a0410, 0xc4201b63c0, 0xc4201b63d0, 0xc420286800, 0xc4201a5300, 0x1a717c0, 0xc4203be018, 0x18a, 0x1a44ba0, ...)
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /usr/local/go/src/net/rpc/server.go:384 +0x14e
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos: created by net/rpc.(*Server).ServeCodec
2018/05/13 23:22:52 packer-builder-vsphere-iso.macos:   /usr/local/go/src/net/rpc/server.go:480 +0x43a
2018/05/13 23:22:52 /Users/alessandro/Dev/packer/packer-builder-vsphere-iso.macos: plugin process exited
2018/05/13 23:22:52 [INFO] (telemetry) ending vsphere-iso
2018/05/13 23:22:52 ui error: Build 'vsphere-iso' errored: unexpected EOF
2018/05/13 23:22:52 Builds completed. Waiting on interrupt barrier...
2018/05/13 23:22:52 machine readable: error-count []string{"1"}
2018/05/13 23:22:52 ui error: 
==> Some builds didn't complete successfully and had errors:
2018/05/13 23:22:52 machine readable: vsphere-iso,error []string{"unexpected EOF"}
2018/05/13 23:22:52 ui error: --> vsphere-iso: unexpected EOF
2018/05/13 23:22:52 ui: 
==> Builds finished but no artifacts were created.
2018/05/13 23:22:52 [INFO] (telemetry) Finalizing.
2018/05/13 23:22:53 waiting for all plugin processes to complete...

@xenithorb
Copy link

This is definitely permissions related. We really need to sort out the minimally viable permissions for this plugin. Got this when trying to replicate #97 multi-role approach.

@xenithorb
Copy link

This goes away when you do this from #97:

vCenter top-level w/ propagate:
Datastore > Browse Datastore, and Low Level File Operations (did not want to have to configure perms at this level, but had trouble getting it to work without it)

@thor
Copy link

thor commented Aug 13, 2018 •
edited
Loading

After working around #104 and #119, I experience this bug as well. While waiting for an IP, while the VM is working through its installation, I get the dreaded unexpected EOF:

...
2018/08/13 09:35:27 packer-builder-vsphere-iso: 2018/08/13 09:35:27 Special code '<enter>' found, replacing with: CodeReturnEnter
2018/08/13 09:35:27 ui: ==> remote: Waiting for IP...
==> remote: Waiting for IP...
==> remote: Power off VM...
2018/08/13 09:40:27 ui: ==> remote: Power off VM...
2018/08/13 09:40:28 packer-builder-vsphere-iso: 2018/08/13 09:40:28 Deleting floppy disk: /tmp/packer148744132
2018/08/13 09:40:28 ui: ==> remote: Destroying VM...
==> remote: Destroying VM...
2018/08/13 09:40:29 [INFO] (telemetry) ending vsphere-iso
2018/08/13 09:40:29 ui error: Build 'remote' errored: Post https://vcsa-host.server.no/sdk: EOF
2018/08/13 09:40:29 Builds completed. Waiting on interrupt barrier...
2018/08/13 09:40:29 machine readable: error-count []string{"1"}
2018/08/13 09:40:29 ui error: 
==> Some builds didn't complete successfully and had errors:
2018/08/13 09:40:29 machine readable: remote,error []string{"Post https://vcsa-host.server.no/sdk: EOF"}
2018/08/13 09:40:29 ui error: --> remote: Post https://vcsa-host.server.no/sdk: EOF
2018/08/13 09:40:29 ui: 
==> Builds finished but no artifacts were created.
2018/08/13 09:40:29 [INFO] (telemetry) Finalizing.
Build 'remote' errored: Post https://vcsa-host.server.no/sdk: EOF

==> Some builds didn't complete successfully and had errors:
--> remote: Post https://vcsa-host.server.no/sdk: EOF

Is there a way to get any more information? I haven't been able to pinpoint it as a problem with #97 per @xenithorb's suggestion, as we already have Low level operations set on our vCenter. In fact, in this case it was run with an administrator account.

Any ideas as to how to debug what goes wrong, or what it tries to send or read when the EOF occurs? Which file to even begin in? Could it simply be reading a fixed set from the stream? Does it expire at a point and needs to be reestablished?

@thor
Copy link

thor commented Oct 1, 2018

@mkuzmin We've looked closer into this bug, and we think it's a lot more specific than originally thought to be:

To reproduce it, simply create an installation takes more than 5 minutes (exactly) to finish the installation and yield an IP address.
The unexpected EOF happens because the connection to the vCSA is terminated exactly five minutes after the plugin (via govmomi) starts to wait for an update on the guest IP address property.
A colleague of mine timed it, and we can consistently reproduce it. If we manage to trim the installation phase to something that takes less than 5 minutes, then everything is fine, but if we can't acquire an IP address within five minutes, it will terminate the connection and return the unexpected EOF error.

We looked into the context object, and in the end also the http package used by govmomi -- is there a default timeout there that's five minutes, somewhere? As far as I can see, there's no default timeout in the http package itself, so without looking deeply into the govmomi library, I guess it's in there somewhere.

@thiagoalves
Copy link

I think that my user is lacking some permissions in the remote vSphere, but at least I would expect a clearer error message.

acordeon:packer talves$ PACKER_DEBUG=1 packer build remote.json
vsphere-iso output will be in this color.

==> vsphere-iso: Creating VM...
Build 'vsphere-iso' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso: unexpected EOF

==> Builds finished but no artifacts were created.
acordeon:packer talves$ PACKER_DEBUG=1 packer build remote.json
vsphere-iso output will be in this color.

^CBuild 'vsphere-iso' finished.
Cleanly cancelled builds after being interrupted.
acordeon:packer talves$ PACKER_LOG=1 packer build remote.json
2018/10/20 09:13:42 [INFO] Packer version: 1.3.1
2018/10/20 09:13:42 Packer Target OS/Arch: darwin amd64
2018/10/20 09:13:42 Built with Go Version: go1.11
2018/10/20 09:13:42 [DEBUG] Discovered plugin: vsphere-clone = /usr/local/bin/packer-builder-vsphere-clone.macos
2018/10/20 09:13:42 [DEBUG] Discovered plugin: vsphere-iso = /usr/local/bin/packer-builder-vsphere-iso.macos
2018/10/20 09:13:42 Detected home directory from env var: /Users/talves
2018/10/20 09:13:42 Using internal plugin for amazon-ebssurrogate
2018/10/20 09:13:42 Using internal plugin for file
2018/10/20 09:13:42 Using internal plugin for ncloud
2018/10/20 09:13:42 Using internal plugin for oracle-classic
2018/10/20 09:13:42 Using internal plugin for virtualbox-ovf
2018/10/20 09:13:42 Using internal plugin for alicloud-ecs
2018/10/20 09:13:42 Using internal plugin for cloudstack
2018/10/20 09:13:42 Using internal plugin for hyperv-vmcx
2018/10/20 09:13:42 Using internal plugin for qemu
2018/10/20 09:13:42 Using internal plugin for vmware-iso
2018/10/20 09:13:42 Using internal plugin for amazon-chroot
2018/10/20 09:13:42 Using internal plugin for amazon-ebsvolume
2018/10/20 09:13:42 Using internal plugin for hyperv-iso
2018/10/20 09:13:42 Using internal plugin for parallels-iso
2018/10/20 09:13:42 Using internal plugin for azure-arm
2018/10/20 09:13:42 Using internal plugin for oracle-oci
2018/10/20 09:13:42 Using internal plugin for parallels-pvm
2018/10/20 09:13:42 Using internal plugin for amazon-instance
2018/10/20 09:13:42 Using internal plugin for googlecompute
2018/10/20 09:13:42 Using internal plugin for oneandone
2018/10/20 09:13:42 Using internal plugin for scaleway
2018/10/20 09:13:42 Using internal plugin for virtualbox-iso
2018/10/20 09:13:42 Using internal plugin for amazon-ebs
2018/10/20 09:13:42 Using internal plugin for lxd
2018/10/20 09:13:42 Using internal plugin for lxc
2018/10/20 09:13:42 Using internal plugin for openstack
2018/10/20 09:13:42 Using internal plugin for vmware-vmx
2018/10/20 09:13:42 Using internal plugin for digitalocean
2018/10/20 09:13:42 Using internal plugin for docker
2018/10/20 09:13:42 Using internal plugin for null
2018/10/20 09:13:42 Using internal plugin for profitbricks
2018/10/20 09:13:42 Using internal plugin for triton
2018/10/20 09:13:42 Using internal plugin for ansible
2018/10/20 09:13:42 Using internal plugin for powershell
2018/10/20 09:13:42 Using internal plugin for puppet-masterless
2018/10/20 09:13:42 Using internal plugin for puppet-server
2018/10/20 09:13:42 Using internal plugin for shell-local
2018/10/20 09:13:42 Using internal plugin for windows-shell
2018/10/20 09:13:42 Using internal plugin for ansible-local
2018/10/20 09:13:42 Using internal plugin for chef-client
2018/10/20 09:13:42 Using internal plugin for shell
2018/10/20 09:13:42 Using internal plugin for file
2018/10/20 09:13:42 Using internal plugin for salt-masterless
2018/10/20 09:13:42 Using internal plugin for windows-restart
2018/10/20 09:13:42 Using internal plugin for chef-solo
2018/10/20 09:13:42 Using internal plugin for converge
2018/10/20 09:13:42 Using internal plugin for artifice
2018/10/20 09:13:42 Using internal plugin for compress
2018/10/20 09:13:42 Using internal plugin for docker-import
2018/10/20 09:13:42 Using internal plugin for docker-push
2018/10/20 09:13:42 Using internal plugin for docker-save
2018/10/20 09:13:42 Using internal plugin for vsphere-template
2018/10/20 09:13:42 Using internal plugin for alicloud-import
2018/10/20 09:13:42 Using internal plugin for docker-tag
2018/10/20 09:13:42 Using internal plugin for googlecompute-export
2018/10/20 09:13:42 Using internal plugin for googlecompute-import
2018/10/20 09:13:42 Using internal plugin for vagrant
2018/10/20 09:13:42 Using internal plugin for vagrant-cloud
2018/10/20 09:13:42 Using internal plugin for checksum
2018/10/20 09:13:42 Using internal plugin for shell-local
2018/10/20 09:13:42 Using internal plugin for amazon-import
2018/10/20 09:13:42 Using internal plugin for manifest
2018/10/20 09:13:42 Using internal plugin for vsphere
2018/10/20 09:13:42 Detected home directory from env var: /Users/talves
2018/10/20 09:13:42 Attempting to open config file: /Users/talves/.packerconfig
2018/10/20 09:13:42 [WARN] Config file doesn't exist: /Users/talves/.packerconfig
2018/10/20 09:13:42 Packer config: &{DisableCheckpoint:false DisableCheckpointSignature:false PluginMinPort:10000 PluginMaxPort:25000 Builders:map[oracle-oci:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-oracle-oci googlecompute:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-googlecompute scaleway:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-scaleway lxd:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-lxd triton:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-triton alicloud-ecs:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-alicloud-ecs qemu:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-qemu vmware-vmx:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-vmware-vmx vsphere-clone:/usr/local/bin/packer-builder-vsphere-clone.macos virtualbox-ovf:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-ovf amazon-chroot:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-chroot azure-arm:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-azure-arm lxc:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-lxc oracle-classic:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-oracle-classic cloudstack:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-cloudstack hyperv-vmcx:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-hyperv-vmcx ncloud:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-ncloud amazon-ebsvolume:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebsvolume docker:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-docker vsphere-iso:/usr/local/bin/packer-builder-vsphere-iso.macos amazon-instance:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-instance amazon-ebs:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebs openstack:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-openstack digitalocean:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-digitalocean vmware-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-vmware-iso parallels-pvm:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-parallels-pvm oneandone:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-oneandone virtualbox-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-iso profitbricks:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-profitbricks parallels-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-parallels-iso file:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-file hyperv-iso:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-hyperv-iso null:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-null amazon-ebssurrogate:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-builder-amazon-ebssurrogate] PostProcessors:map[docker-tag:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-tag vagrant:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant vsphere:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vsphere vagrant-cloud:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant-cloud shell-local:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-shell-local artifice:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-artifice docker-import:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-import docker-push:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-push vsphere-template:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vsphere-template googlecompute-export:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-googlecompute-export manifest:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-manifest compress:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-compress docker-save:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-docker-save googlecompute-import:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-googlecompute-import checksum:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-checksum amazon-import:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-amazon-import alicloud-import:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-alicloud-import] Provisioners:map[ansible-local:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-ansible-local salt-masterless:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-salt-masterless windows-restart:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-windows-restart powershell:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-powershell shell-local:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell-local windows-shell:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-windows-shell ansible:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-ansible file:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-file converge:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-converge puppet-masterless:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-puppet-masterless chef-client:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-chef-client shell:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell puppet-server:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-puppet-server chef-solo:/usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-chef-solo]}
2018/10/20 09:13:42 Detected home directory from env var: /Users/talves
2018/10/20 09:13:42 Setting cache directory: /Users/talves/projects/aurea-ms-central-vmware/packer/packer_cache
2018/10/20 09:13:42 Detected home directory from env var: /Users/talves
2018/10/20 09:13:42 Loading builder: vsphere-iso
2018/10/20 09:13:42 Creating plugin client for path: /usr/local/bin/packer-builder-vsphere-iso.macos
2018/10/20 09:13:42 Starting plugin: /usr/local/bin/packer-builder-vsphere-iso.macos []string{"/usr/local/bin/packer-builder-vsphere-iso.macos"}
2018/10/20 09:13:42 Waiting for RPC address for: /usr/local/bin/packer-builder-vsphere-iso.macos
2018/10/20 09:13:42 packer-builder-vsphere-iso.macos: 2018/10/20 09:13:42 Plugin minimum port: 10000
2018/10/20 09:13:42 packer-builder-vsphere-iso.macos: 2018/10/20 09:13:42 Plugin maximum port: 25000
2018/10/20 09:13:42 packer-builder-vsphere-iso.macos: 2018/10/20 09:13:42 Plugin address: unix /var/folders/y6/1q4067c908qd9zntlp56q1s80000gn/T/packer-plugin240477232
2018/10/20 09:13:42 packer-builder-vsphere-iso.macos: 2018/10/20 09:13:42 Waiting for connection...
2018/10/20 09:13:42 packer-builder-vsphere-iso.macos: 2018/10/20 09:13:42 Serving a plugin connection...
2018/10/20 09:13:42 Loading provisioner: shell
2018/10/20 09:13:42 Plugin could not be found. Checking same directory as executable.
2018/10/20 09:13:42 Current exe path: /usr/local/bin/packer
2018/10/20 09:13:42 Creating plugin client for path: /usr/local/bin/packer
2018/10/20 09:13:42 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-provisioner-shell"}
2018/10/20 09:13:42 Waiting for RPC address for: /usr/local/bin/packer
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 [INFO] Packer version: 1.3.1
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Packer Target OS/Arch: darwin amd64
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Built with Go Version: go1.11
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Detected home directory from env var: /Users/talves
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Attempting to open config file: /Users/talves/.packerconfig
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 [WARN] Config file doesn't exist: /Users/talves/.packerconfig
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Packer config: &{DisableCheckpoint:false DisableCheckpointSignature:false PluginMinPort:10000 PluginMaxPort:25000 Builders:map[] PostProcessors:map[] Provisioners:map[]}
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Detected home directory from env var: /Users/talves
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Setting cache directory: /Users/talves/projects/aurea-ms-central-vmware/packer/packer_cache
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Detected home directory from env var: /Users/talves
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 args: []string{"packer-provisioner-shell"}
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Plugin minimum port: 10000
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Plugin maximum port: 25000
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Plugin address: unix /var/folders/y6/1q4067c908qd9zntlp56q1s80000gn/T/packer-plugin497960173
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Waiting for connection...
2018/10/20 09:13:42 packer: 2018/10/20 09:13:42 Serving a plugin connection...
2018/10/20 09:13:42 Build debug mode: false
2018/10/20 09:13:42 Force build: false
2018/10/20 09:13:42 On error: 
2018/10/20 09:13:42 Preparing build: vsphere-iso
vsphere-iso output will be in this color.

2018/10/20 09:13:42 Waiting on builds to complete...
2018/10/20 09:13:42 Starting build run: vsphere-iso
2018/10/20 09:13:42 Running builder: vsphere-iso
2018/10/20 09:13:42 [INFO] (telemetry) Starting builder vsphere-iso
==> vsphere-iso: Creating VM...
Build 'vsphere-iso' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso: unexpected EOF

==> Builds finished but no artifacts were created.
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: panic: runtime error: invalid memory address or nil pointer dereference
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x175ab59]
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: goroutine 10 [running]:
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/vmware/govmomi/object.DistributedVirtualPortgroup.EthernetCardBackingInfo(0xc42003a5a0, 0x5f, 0xc4200f5080, 0xc4204b2e60, 0x1b, 0xc42019e630, 0x10, 0x1de0d20, 0xc4200320a0, 0x1ddd160, ...)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/go/src/github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/vmware/govmomi/object/distributed_virtual_portgroup.go:47 +0x249
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/driver.addNetwork(0xc42018c810, 0xc420397b40, 0x3, 0x4, 0xc42053b7d0, 0xc420397b40, 0x3, 0x4, 0x0, 0x0)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/go/src/github.com/jetbrains-infra/packer-builder-vsphere/driver/vm.go:453 +0xaa
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/driver.(*Driver).CreateVM(0xc42018c810, 0xc42053b7d0, 0xe, 0x1c53b20, 0xc42018c810)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/go/src/github.com/jetbrains-infra/packer-builder-vsphere/driver/vm.go:122 +0x2a8
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/iso.(*StepCreateVM).Run(0xc4203a9c20, 0x1de0ce0, 0xc420396900, 0x1ddfc60, 0xc42039f170, 0x0)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/go/src/github.com/jetbrains-infra/packer-builder-vsphere/iso/step_create.go:49 +0x28f
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/helper/multistep.(*BasicRunner).Run(0xc4203968c0, 0x1ddfc60, 0xc42039f170)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/go/src/github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/helper/multistep/basic_runner.go:72 +0x22f
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/iso.(*Builder).Run(0xc420192080, 0x1de1ae0, 0xc4203c64c0, 0x1ddcb60, 0xc4203a9c10, 0x1de0f60, 0xc420190170, 0x10b40cc, 0x1a099e0, 0x1a099e0, ...)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/go/src/github.com/jetbrains-infra/packer-builder-vsphere/iso/builder.go:99 +0xa8b
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/packer/rpc.(*BuilderServer).Run(0xc4201920a0, 0x1, 0xc420032c80, 0x0, 0x0)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/go/src/github.com/jetbrains-infra/packer-builder-vsphere/vendor/github.com/hashicorp/packer/packer/rpc/builder.go:94 +0x1ce
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: reflect.Value.call(0xc4202c01e0, 0xc420190078, 0x13, 0x1ce7415, 0x4, 0xc420079f18, 0x3, 0x3, 0xc42017e000, 0xc420393708, ...)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/usr/local/go/src/reflect/value.go:447 +0x969
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: reflect.Value.Call(0xc4202c01e0, 0xc420190078, 0x13, 0xc4203b1718, 0x3, 0x3, 0x1, 0x114157e, 0x1)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/usr/local/go/src/reflect/value.go:308 +0xa4
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: net/rpc.(*service).call(0xc42017e100, 0xc42018a140, 0xc42019e060, 0xc42019e070, 0xc4202d0180, 0xc420192340, 0x1a367a0, 0xc420032c5c, 0x18a, 0x1a099e0, ...)
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/usr/local/go/src/net/rpc/server.go:384 +0x14e
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: created by net/rpc.(*Server).ServeCodec
2018/10/20 09:13:53 packer-builder-vsphere-iso.macos: 	/usr/local/go/src/net/rpc/server.go:480 +0x43a
2018/10/20 09:13:53 [INFO] (telemetry) ending vsphere-iso
2018/10/20 09:13:53 ui error: Build 'vsphere-iso' errored: unexpected EOF
2018/10/20 09:13:53 Builds completed. Waiting on interrupt barrier...
2018/10/20 09:13:53 machine readable: error-count []string{"1"}
2018/10/20 09:13:53 ui error: 
==> Some builds didn't complete successfully and had errors:
2018/10/20 09:13:53 machine readable: vsphere-iso,error []string{"unexpected EOF"}
2018/10/20 09:13:53 ui error: --> vsphere-iso: unexpected EOF
==> Builds finished but no artifacts were created.
2018/10/20 09:13:53 /usr/local/bin/packer-builder-vsphere-iso.macos: plugin process exited
2018/10/20 09:13:53 [INFO] (telemetry) Finalizing.
2018/10/20 09:13:55 waiting for all plugin processes to complete...
2018/10/20 09:13:55 /usr/local/bin/packer: plugin process exited

@tsborland
Copy link
Contributor

Spent some time on this today, and was able to build without globally inhereted permissions.

#97 (comment)

@thor
Copy link

thor commented Nov 5, 2018

That's good! From our own testing though, this issue isn't really an issue specific to permissions, but is a timeout on the vCSA side of things. Any build that results in waiting for longer than 5 minutes for any property to update (in this case, the IP address) will result in an unexpected EOF due to the timeout.

Currently we run our builds in debug mode as to get around the timeout, meaning we manually let it know that it can run, but it's quite tedious and not very sustainable.

@mkuzmin
Copy link
Contributor

mkuzmin commented Nov 5, 2018

unexpected EOF message is so vague, so we have several unrelated problems mixed in this issue.

I've updated README with a list of required permissions.

Our builds may take hours, and do work fine. The only known timeout is 30-minute sessions, and it's especially handled in

packer-builder-vsphere/driver/driver.go

Lines 41 to 56 in 6f7ec33

soapClient := soap.NewClient(vcenter_url, config.InsecureConnection)
vimClient, err := vim25.NewClient(ctx, soapClient)
if err != nil {
return nil, err
}
vimClient.RoundTripper = session.KeepAlive(vimClient.RoundTripper, 10*time.Minute)
client := &govmomi.Client{
Client: vimClient,
SessionManager: session.NewManager(vimClient),
}
err = client.SessionManager.Login(ctx, credentials)
if err != nil {
return nil, err
}
.

@mkuzmin
Copy link
Contributor

mkuzmin commented Nov 5, 2018

for test, try to change 10*time.Minute to 3.
You can build binaries by docker-compose run build

@jgibbarduk
Copy link

I have the exact same issue as @thor. Consistently at the 5 min mark after creating the VM and waiting for an IP. Packer decides to destroy the VM after getting the EOF message from vSphere 6.5

I've just tried @mkuzmin suggestion of rebuilding the plugin with the keepalive set to 3 mins. Unfortunately it had no effect.

The really strange thing was that this packer process was working perfectly up until about a week ago. I really have no idea what has happened. Any ideas would be great. Thanks

@thor
Copy link

thor commented Nov 11, 2018

@jgibbarduk I'm glad to hear we're not alone, so maybe we could throw some ideas back and forth.

  • Do you have a direct connection to your vCSA from the build machine? I ask, because I've been trying to exclude my odd proxy setup as a possible reason for the problems (HTTP to SOCKS5 over SSH), but couldn't find any keepalives there that matched with the 5 minute mark.
  • Have you recently upgraded your vCSA?
  • Have all your builds taken noticeably more than 5 minutes? (Ours take about 15-20.)

Thanks for taking a look @mkuzmin and pointing to ´driver.go´ -- I can't confirm changing that didn't help yet, but I'll this week. All I could find of related documentation for the fact that it always times out exactly when waiting for the IP address is that it's utilising the ´WaitForUpdatesEx´ function, which refers to the ´WaitOptions´ parameter which may have an unset ´maxWaitSeconds´ property that defers the timeout to some policy (see second paragraph for the property).

@jgibbarduk
Copy link

jgibbarduk commented Nov 12, 2018 •
edited
Loading

Do you have a direct connection to your vCSA from the build machine?

No, I go via a UTM/Firewall device. When I run the same packer process from a machine with a direct connection, the process works successfully.

Have you recently upgraded your vCSA?

Not that I know of. Our vCSA is not managed by me, but the admins have said there has been no recent change. This packer process work working up until a few weeks ago.

Have all your builds taken noticeably more than 5 minutes? (Ours take about 15-20.)

Yes, they have always been long than 5 mins. Especially some of the older windows platforms.

I still think that it is to do with the connection being terminated somehow after 5 mins, when going through our UTM.

@ToroNZ
Copy link

ToroNZ commented Jan 12, 2020

My setup:

Packer -> HAproxy (tcp) -> Caddy (http) -> vCenter API

The odd thing was that I errored out at the ~1min mark...

2020/01/11 06:18:50 [INFO] Packer version: 1.3.4
2020/01/11 06:18:50 Packer Target OS/Arch: linux amd64
2020/01/11 06:18:50 Built with Go Version: go1.11.4
[...]
==> vsphere-iso: Power on VM...
==> vsphere-iso: Waiting 10s for boot...
==> vsphere-iso: Typing boot command...
[...]
2020/01/11 06:19:23 packer-builder-vsphere-iso: 2020/01/11 06:19:23 Special code '<enter>' found, replacing with: CodeReturnEnter
               ^===== Last communication before going 'idle'
==> vsphere-iso: Waiting for IP...
==> vsphere-iso: Clear boot order...
==> vsphere-iso: Power off VM...
==> vsphere-iso: Destroying VM...
2020/01/11 06:20:24 [INFO] (telemetry) ending vsphere-iso
2020/01/11 06:20:24 ui error: Build 'vsphere-iso' errored: Post https://vcenter.com/sdk: EOF
               ^===== Timeout exactly 1m after
[...]

HAproxy had a default of 1min... setting it to 10min (in line with Caddy) for TCP connections solved this issue for me:

    timeout client          10m
    timeout server          10m

Success:

2020/01/12 06:19:46 [INFO] Packer version: 1.3.4
2020/01/12 06:19:46 Packer Target OS/Arch: linux amd64
2020/01/12 06:19:46 Built with Go Version: go1.11.4
[...]
==> vsphere-iso: Power on VM...
==> vsphere-iso: Waiting 10s for boot...
==> vsphere-iso: Typing boot command...
[...]
2020/01/12 06:20:17 packer-builder-vsphere-iso: 2020/01/12 06:20:17 Special code '<enter>' found, replacing with: CodeReturnEnter
==> vsphere-iso: Waiting for IP...
==> vsphere-iso: IP address: 1.1.1.1
2020/01/12 06:27:26 packer-builder-vsphere-iso: 2020/01/12 06:27:26 [INFO] Waiting for SSH, up to timeout: 5m0s
[...]
Build 'vsphere-iso' finished.

@luciantuce
Copy link

Hello guys! I am experiencing, I think, the same error with the packer version 1.7.9:

Build 'vsphere-iso' errored after 3 minutes 31 seconds: Post "https://vcenter/sdk": EOF.

The build machine from where the packer commands are being executed stays in a different DataCenter than the VCenter one (No FW Rules would be between them).

The awkward topic would be that in debug mode (command packer build --debug file.json ); it ends up with success (with template published to the vSphere).

Any ideas would be welcome!

@thor
Copy link

thor commented Jan 24, 2022

@luciantuce I think you'll need to take this issue to the new Packer plugin repository: https://github.com/hashicorp/packer-plugin-vsphere

@vrevelas vrevelas mentioned this issue Jan 31, 2022
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
problem
Milestone
No milestone
Development

No branches or pull requests
9 participants
@thor @thiagoalves @mkuzmin @lifeisfoo @jgibbarduk @ToroNZ @xenithorb @luciantuce @tsborland