No error message for aws_instance creation failure

[aaronbrashears]

I was creating an aws_instance resource with an attached ebs_block_device. On apply, the error message from terriform was:

* Error waiting for instance (i-81795976) to become ready: unexpected state 'terminated', wanted target 'running'

Going into the AWS UI to test, I found that I was bumping up against the ebs iops limit which returns the move useful message:

Error creating volume
Maximum number of active volumes iops, 100000, exceeded.

I would like to see that error message during the terraform operation.

[mitchellh]

Hm, interesting. I wonder where this error message appears. Where did you see it in the console? We check all API responses for errors so this must mean it is coming from some non-intuitive endpoint.

[aaronbrashears]

During the apply after some things refresh, I see:

aws_instance.replayer: Creating...
  ami:                                               "" => "ami-5189a661"
  ...
  ebs_block_device.#:                                "" => "1"
  ebs_block_device.4023988449.delete_on_termination: "" => "1"
  ebs_block_device.4023988449.device_name:           "" => "/dev/xvdf"
  ebs_block_device.4023988449.encrypted:             "" => "<computed>"    
  ebs_block_device.4023988449.iops:                  "" => "20000"
  ebs_block_device.4023988449.snapshot_id:           "" => "<computed>"
  ebs_block_device.4023988449.volume_size:           "" => "2000"
  ebs_block_device.4023988449.volume_type:           "" => "io1"
  ...

and after an moment:

aws_instance.replayer: Error: 1 error(s) occurred:

* Error waiting for instance (i-81795976) to become ready: unexpected state 'terminated', wanted target 'running'

[pikeas]

I believe @mitchellh is asking which section of the AWS console displayed the error.

[mstuparu]

I just encountered something similar when I was trying to launch a instance with a 3 terabyte standard volume. No obvious error was thrown instance just got terminated, Then i used volume_type = "gp2" which fixed it.

[mmell]

I see this error now. I hit CTRL-C in the middle of an apply and then started apply again. In the AWS UI I saw two instances being created when there should have been one (the cluster_member below). I manually terminated the older one. The second apply completed successfully. Now, on a third apply I see

* aws_instance.cluster_member: Error waiting for instance (i-e088de24) to become ready: unexpected state 'terminated', wanted target 'running'

I resolved the issue by manually replacing the instance id in the tfstate file.
[Update] After I manually replaced the instance id in the tfstate file with the id from the second instance a destroy completed successfully. But even if I delete the tfstate & backup files, I still see the problem in the next apply. The offending instance id has changed and it is listed in the EC2 UI as terminated.

[Update 2] I still see the problem when I run apply even if I destroy everything, delete the tfstate* files, and rename "aws_instance.cluster_member".

[Update 3 & Resolution] In aws_instance.cluster_member I was creating volumes with delete_on_termination = "false". After I manually deleted all of the available volumes the issue disappeared. Now I can apply successfully.

resource "aws_instance" "cluster_member" {
  ebs_block_device = {
    delete_on_termination = "false" # **the source of the problem**
    ...

[RyanBowlby-Reflektion]

What's the status on this one? Bubbling up that error would prove useful.

[mengesb]

I get a different issue. delete_on_termination doesn't seem to matter, however to see the error message I receive from AWS, I go to one of the terminated instances and get the following:

State transition reason: Client.InvalidParameterCombination: Could not create volume with size 40GiB and iops 5000 from snapshot 'snap-67dd9c7a'

I'm creating a 40G root volume with 5k IOPS on Ubuntu 14.04 LTS 64-bit LVM (ami-415f6d2b)

[mengesb]

Looks like this is more related to the AMIs compatibility. Seems that I should probably be using ami-aac24fbd instead

[mengesb]

So the root of my issue is there's math involved in creating the volume itself. Doing direct aws ec2 create-volume commands, it shows there's a ratio of size in GiB and IOPS 1:30:

$ aws ec2 create-volume --size 20 --snapshot-id snap-b619c24e --region us-east-1 --availability-zone us-east-1a --volume-type io1 --iops 1000

An error occurred (InvalidParameterValue) when calling the CreateVolume operation: Iops to volume size ratio of 50.000000 is too high; maximum is 30

The maximum IOPS for a 20 GiB volume would be 600 (20 * 30 = 600). To get to a minimum of 1000 I'd have to create a volume 33GiB in size or larger. 10k IOPs would require a volume 334GiB in size or larger.

Not sure that this check can be included in terraform, however the error messages are not that grand even considering the 'state transition reason' message.

[mengesb]

Is it possible to not make the iops parameter required with io1 volume_types? Could we just assume ${volume_size * 30} for iops unless the user overrides this value?

[mylons]

I actually work with @aaronbrashears ! and ran into this issue as well just today.

➜  repo git:(airflow_tweaks) ✗ terraform version
Terraform v0.6.14

I believe, for whatever reason, my ebs block:

  ebs_block_device = {
    device_name = "/dev/xvdi"
    volume_size = "${var.ebs_size}"
    #volume_type = "gp2"
  }

is prefering iops as the volume_type instead of gp2. Once I uncomment gp2 it will work. It's quite likely that this aws account's iops limit is hit.

[ajvb]

Just hit this today.

Terraform output:

* aws_instance.dbserver: Error waiting for instance (i-01a31aa8844145a8e) to become ready: unexpected state 'terminated', wanted target 'running'. last error: %!s(<nil>)

Terraform version:

$ terraform version
Terraform v0.7.4

The reason was hitting an EBS-related limit. When viewing the terminated EC2 instance in the console, this was the error message:

State transition reason: Client.VolumeLimitExceeded: Volume limit exceeded

[rmmcnulty9]

I was getting this error too. But for me the cause was the AMI I was using (Ubuntu 16.04) did not allow the device name I gave it. It took a long time to debug because Terraform and AWS had very unhelpful error messages. I went with a /dev/sdb for my device name and it worked perfectly.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.