amazon-ebssurrogate not able to lookup source volume id when using spot instances

[sergey-safarov]

Overview of the Issue

Issue related to #8777 and reproduced on current master.

When using spot instances with the amazon-ebssurrogate builder, the builder is not able to lookup the id of the source volume(s). This appears to be because it issues the describe instance call to AWS after it has stopped the EC2 instance, and not before.

Reproduction Steps

Try and use spot instances with amazon-ebssurrogate, you'll get the following error regardless of the value of delete_on_termination in the launch_block_device_mappings config:

==> amazon-ebssurrogate: 1 error occurred:
==> amazon-ebssurrogate: 	* Volume ID for device /dev/xvdf not found
==> amazon-ebssurrogate:
==> amazon-ebssurrogate:

Simply replacing the spot_* configuration options with instance_type works.

Packer version

[safarov@safarov-dell intetics]$ packer version
Packer v1.6.0-dev (19e4afa728c02104d2c0571e6c6fb86278009a75)

Simplified Packer Buildfile

Fails:

{
  "builders": [
    {
      "ami_name": "packer debug {{ user `BuildTime` }}",
      "ami_virtualization_type": "hvm",

      "type": "amazon-ebssurrogate",
      "spot_instance_types": [
        "c5.large",
        "c5.xlarge"
      ],
      "spot_price": "auto",

      "launch_block_device_mappings": [
        {
          "volume_type": "gp2",
          "device_name": "{{ user `DeviceName` }}",
          "delete_on_termination": false,
          "volume_size": 20
        }
      ],

      "ami_root_device": {
        "delete_on_termination": true,
        "device_name": "/dev/xvda",
        "source_device_name": "{{ user `DeviceName` }}",
        "volume_size": 20,
        "volume_type": "gp2"
      }
    }
  ]
}

Works:

{
  "builders": [
    {
      "ami_name": "packer debug {{ user `BuildTime` }}",
      "ami_virtualization_type": "hvm",

      "type": "amazon-ebssurrogate",
      "instance_type": "c5.large",

      "launch_block_device_mappings": [
        {
          "volume_type": "gp2",
          "device_name": "{{ user `DeviceName` }}",
          "delete_on_termination": false,
          "volume_size": 20
        }
      ],

      "ami_root_device": {
        "delete_on_termination": true,
        "device_name": "/dev/xvda",
        "source_device_name": "{{ user `DeviceName` }}",
        "volume_size": 20,
        "volume_type": "gp2"
      }
    }
  ]
}

Operating system and Environment details

Tested on Linux.

Log Fragments and crash.log files

2020/06/02 22:13:45 packer-builder-amazon-ebssurrogate plugin: [DEBUG] Opening new ssh session
2020/06/02 22:13:45 packer-builder-amazon-ebssurrogate plugin: [DEBUG] starting remote command: rm -f /tmp/script_6506.sh
2020/06/02 22:13:45 packer-builder-amazon-ebssurrogate plugin: [INFO] RPC endpoint: Communicator ended with: 0
2020/06/02 22:13:45 [INFO] RPC client: Communicator ended with: 0
2020/06/02 22:13:45 [INFO] RPC endpoint: Communicator ended with: 0
2020/06/02 22:13:45 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2020/06/02 22:13:45 packer-builder-amazon-ebssurrogate plugin: [DEBUG] Opening new ssh session
2020/06/02 22:13:45 packer-builder-amazon-ebssurrogate plugin: [DEBUG] starting remote command: rm -f
2020/06/02 22:13:45 packer-builder-amazon-ebssurrogate plugin: [INFO] RPC endpoint: Communicator ended with: 0
2020/06/02 22:13:45 [INFO] RPC client: Communicator ended with: 0
2020/06/02 22:13:45 [INFO] RPC endpoint: Communicator ended with: 0
2020/06/02 22:13:45 packer-provisioner-shell plugin: [INFO] RPC client: Communicator ended with: 0
2020/06/02 22:13:45 [INFO] (telemetry) ending shell
==> amazon-ebssurrogate: 	* Volume ID for device /dev/sdf not found
==> amazon-ebssurrogate: 
==> amazon-ebssurrogate:
==> amazon-ebssurrogate: 1 error occurred:
==> amazon-ebssurrogate: 	* Volume ID for device /dev/sdf not found
==> amazon-ebssurrogate: 
==> amazon-ebssurrogate:
==> amazon-ebssurrogate: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebssurrogate: Terminating the source AWS instance...
==> amazon-ebssurrogate: Cleaning up any extra volumes...
==> amazon-ebssurrogate: No volumes to clean up, skipping
==> amazon-ebssurrogate: Deleting temporary security group...
==> amazon-ebssurrogate: Deleting temporary keypair...
2020/06/02 22:14:06 [INFO] (telemetry) ending amazon-ebssurrogate
	* Volume ID for device /dev/sdf not found

Build 'amazon-ebssurrogate' errored: 1 error occurred:

	* Volume ID for device /dev/sdf not found
2020/06/02 22:14:06 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:

2020/06/02 22:14:06 machine readable: amazon-ebssurrogate,error []string{"1 error occurred:\n\t* Volume ID for device /dev/sdf not found\n\n"}
	* Volume ID for device /dev/sdf not found

==> Builds finished but no artifacts were created.

2020/06/02 22:14:06 [INFO] (telemetry) Finalizing.

==> Some builds didn't complete successfully and had errors:
--> amazon-ebssurrogate: 1 error occurred:
	* Volume ID for device /dev/sdf not found



==> Builds finished but no artifacts were created.

[sergey-safarov]

Just found when the issue is happening removed target volume and not deleted VM root volume.
So think packer delete target volume instead of VM root volume when used spot instances.

[ghost]

This issue has been automatically migrated to hashicorp/packer-plugin-amazon#5 because it looks like an issue with that plugin. If you believe this is not an issue with the plugin, please reply to hashicorp/packer-plugin-amazon#5.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.