New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EC2 instance with EBS root volume destroyed & recreated on each apply #2905

Open
hashibot opened this Issue Jan 9, 2018 · 8 comments

Comments

Projects
None yet
9 participants
@hashibot

hashibot commented Jan 9, 2018

This issue was originally opened by @mrubin as hashicorp/terraform#17059. It was migrated here as a result of the provider split. The original body of the issue is below.


I am seeing the below issue on Terraform v0.11.1. I understand that there are a number of previous issues in this area, but I have not been able to get past this by looking at the suggestions there.

I am using the following AMI:

$ aws ec2 describe-images --image-ids ami-5583d42f                                                                                                                                      {
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2018-01-03T19:04:51.000Z",
            "ImageId": "ami-5583d42f",
            "ImageLocation": "amazon/amzn-ami-hvm-2017.09.1.20180103-x86_64-gp2",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "137112412989",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "Encrypted": false,
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-084cb269d55295d27",
                        "VolumeSize": 8,
                        "VolumeType": "gp2"
                    }
                }
            ],
            "Description": "Amazon Linux AMI 2017.09.1.20180103 x86_64 HVM GP2",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "ImageOwnerAlias": "amazon",
            "Name": "amzn-ami-hvm-2017.09.1.20180103-x86_64-gp2",
            "RootDeviceName": "/dev/xvda",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm"
        }
    ]
}

My terraform configuration for this resource looks like this:

resource "aws_instance" "jenkins" {
  ebs_block_device {
    device_name           = "/dev/xvda"
    volume_size           = 100
    volume_type           = "gp2"
    delete_on_termination = false
    encrypted             = false
    snapshot_id           = "snap-084cb269d55295d27"
  }

I added the encrypted and snapshot_id based on suggestions in prior issues, but it is not making a difference. I am seeing the following from Terraform:

-/+ aws_instance.jenkins (new resource required)
      id:                                               "i-05e55cfa524407bed" => <computed> (forces new resource)
      ami:                                              "ami-5583d42f" => "ami-5583d42f"
      ebs_block_device.#:                               "0" => "1"
      ebs_block_device.745701089.delete_on_termination: "" => "false" (forces new resource)
      ebs_block_device.745701089.device_name:           "" => "/dev/xvda" (forces new resource)
      ebs_block_device.745701089.encrypted:             "" => "false" (forces new resource)
      ebs_block_device.745701089.snapshot_id:           "" => "snap-084cb269d55295d27" (forces new resource)
      ebs_block_device.745701089.volume_size:           "" => "100" (forces new resource)
      ebs_block_device.745701089.volume_type:           "" => "gp2" (forces new resource)
      ephemeral_block_device.#:                         "0" => <computed>

Note the ebs_block_device.#: "0" => "1". Does anyone see what I'm doing wrong?

Thank you!

@mkingbe

This comment has been minimized.

mkingbe commented Feb 6, 2018

any info regarding this issue ?

@ageekymonk

This comment has been minimized.

ageekymonk commented Feb 21, 2018

I tried with the terraform 0.11.3 and provider version 1.9.0. I could not reproduce it.

@kmaragon

This comment has been minimized.

kmaragon commented Mar 27, 2018

Using:

Terraform v0.11.5

  • provider.aws v1.12.0
-/+ aws_ebs_volume.scylladb-volumes (new resource required)
      id:                                        "vol-************" => <computed> (forces new resource)
      arn:                                       "arn:aws:ec2:us-west-2:***********:volume/vol-**********" => <computed>
      availability_zone:                         "us-west-2c" => "${element(aws_instance.scylladb-node.*.availability_zone, count.index)}" (forces new resource)
      encrypted:                                 "false" => <computed>
      iops:                                      "150" => "150"
      kms_key_id:                                "" => <computed>
      size:                                      "15" => "15"
      snapshot_id:                               "" => <computed>
      type:                                      "io1" => "io1"

-/+ aws_instance.scylladb-node (new resource required)
      id:                                        "i-**********" => <computed> (forces new resource)
      ami:                                       "ami-f369f08b" => "ami-f369f08b"
      associate_public_ip_address:               "true" => "true"
      availability_zone:                         "us-west-2c" => <computed>
      ebs_block_device.#:                        "1" => <computed>
      ebs_optimized:                             "true" => "true"
      ephemeral_block_device.#:                  "0" => <computed>
      get_password_data:                         "false" => "false"
      instance_state:                            "running" => <computed>
      instance_type:                             "r4.large" => "r4.large"
      ipv6_address_count:                        "" => <computed>
      ipv6_addresses.#:                          "0" => <computed>
      network_interface.#:                       "0" => <computed>
      network_interface_id:                      "eni-*****" => <computed>
      password_data:                             "" => <computed>
      placement_group:                           "" => <computed>
      primary_network_interface_id:              "eni-********" => <computed>
      root_block_device.#:                       "1" => "1"
      root_block_device.0.delete_on_termination: "true" => "true"
      root_block_device.0.volume_id:             "vol-*************8" => <computed>
      root_block_device.0.volume_size:           "10" => "10"
      root_block_device.0.volume_type:           "gp2" => "gp2"
      security_groups.#:                         "0" => "1" (forces new resource)
      security_groups.2560785339:                "" => "sg-********" (forces new resource)
      source_dest_check:                         "true" => "true"
      subnet_id:                                 "subnet-********" => "subnet-**********"
      tenancy:                                   "default" => <computed>
      user_data:                                 "9c0d6daebe40fd449371695fe02590ac7273d128" => "9c0d6daebe40fd449371695fe02590ac7273d128"
      vpc_security_group_ids.#:                  "1" => <computed>

  + aws_instance.test-node
      id:                                        <computed>
      ami:                                       "ami-da801ca2"
      associate_public_ip_address:               "true"
      availability_zone:                         <computed>
      ebs_block_device.#:                        <computed>
      ebs_optimized:                             "true"
      ephemeral_block_device.#:                  <computed>
      get_password_data:                         "false"
      instance_state:                            <computed>
      instance_type:                             "t2.medium"
      ipv6_address_count:                        <computed>
      ipv6_addresses.#:                          <computed>
      network_interface.#:                       <computed>
      network_interface_id:                      <computed>
      password_data:                             <computed>
      placement_group:                           <computed>
      primary_network_interface_id:              <computed>
      private_dns:                               <computed>
      private_ip:                                <computed>
      public_dns:                                <computed>
      public_ip:                                 <computed>
      root_block_device.#:                       <computed>
      security_groups.#:                         "1"
      security_groups.2560785339:                "sg-********"
      source_dest_check:                         "true"
      subnet_id:                                 "subnet-********"
      tenancy:                                   <computed>
      user_data:                                 "393e2803118d31a7121f7947d21a4f39917b4063"
      volume_tags.%:                             <computed>
      vpc_security_group_ids.#:                  <computed>

-/+ aws_volume_attachment.volumes-attachment (new resource required)
      id:                                        "vai-***********" => <computed> (forces new resource)
      device_name:                               "xvdh" => "xvdh"
      instance_id:                               "i-*****************" => "${element(aws_instance.scylladb-node.*.id, count.index)}" (forces new resource)
      volume_id:                                 "vol-****************" => "${element(aws_ebs_volume.scylladb-volumes.*.id, count.index)}" (forces new resource)

(With account specific stuff redacted).

This is absolutely consistently reproducible. My workaround right now is to put the recreated stuff in a separate file and just rename it to a non-.tf and remove it from the tfstate unless I want to make changes.

@kmaragon

This comment has been minimized.

kmaragon commented Mar 29, 2018

Actually never mind. My issue looks to have been caused by the fact that
security_groups always requires a recreation of the instance if using vpc'd instances
whereas
vpc_security_group_ids doesn't

@WasStoNed

This comment has been minimized.

WasStoNed commented Apr 6, 2018

Hi, I'm seeing this behaviour, I've added the vpc_security_group_ids and this has removed one of the restores. I've played around a bit and notice that the root_block_device forces a restore if the size changes and the ebs_block_device is always requesting a new resource regardless of change.

Started with no root_block_device but added just to see behaviour and other posts saying you need to specify.

My config looks like this for the ebs;

ebs_block_device {
device_name = "/dev/sda1"
volume_size = "15"
volume_type = "gp2"
delete_on_termination = "true"
}

any suggestion please or is this actually a bug?

@noah-trilling

This comment has been minimized.

noah-trilling commented Apr 10, 2018

+1

@joey-clypd

This comment has been minimized.

joey-clypd commented Apr 30, 2018

I'm also seeing this. I've configured the creation of an ebs_block_device:

resource "aws_instance" "<snip>" {
  ami                     = "<snip>"
  instance_type           = "m4.xlarge"
  subnet_id               = "<snip>"
  iam_instance_profile    = "<snip>"
  key_name                = "<snip>"
  disable_api_termination = true

  ebs_block_device {
    device_name = "${var.block_device_name}"
    snapshot_id = "${var.device_snapshot_id}"
    volume_size = 40
    volume_type = "gp2"
  }

  vpc_security_group_ids = ["<snip>"]
}

But what shows up in terraform.tfstate is a root_block_device:

            "resources": {
                "aws_instance.name": {
                    "type": "aws_instance",
                    "depends_on": [
                        "module.ami_filter"
                    ],
                    "primary": {
                        "id": "<snip>",
                        "attributes": {
                            "ami": "<snip>",
                            "associate_public_ip_address": "false",
                            "availability_zone": "us-east-1d",
                            "credit_specification.#": "1",
                            "credit_specification.0.cpu_credits": "standard",
                            "disable_api_termination": "true",
                            "ebs_block_device.#": "0",
                            "ebs_optimized": "false",
                            "ephemeral_block_device.#": "0",
                            "get_password_data": "false",
                            "iam_instance_profile": "<snip>",
                            "id": "<snip>",
                            "instance_state": "running",
                            "instance_type": "m4.xlarge",
                            "ipv6_addresses.#": "0",
                            "key_name": "<snip>",
                            "monitoring": "false",
                            "network_interface.#": "0",
                            "network_interface_id": "<snip>",
                            "password_data": "",
                            "placement_group": "",
                            "primary_network_interface_id": "<snip>",
                            "private_dns": "<snip>",
                            "private_ip": "<snip>",
                            "public_dns": "",
                            "public_ip": "",
                            "root_block_device.#": "1",
                            "root_block_device.0.delete_on_termination": "true",
                            "root_block_device.0.iops": "120",
                            "root_block_device.0.volume_id": "<snip>",
                            "root_block_device.0.volume_size": "40",
                            "root_block_device.0.volume_type": "gp2",
                            "security_groups.#": "0",
                            "source_dest_check": "true",
                            "subnet_id": "<snip>",
                            "tenancy": "default",
                            "volume_tags.%": "0",
                            "vpc_security_group_ids.#": "1",
                            "vpc_security_group_ids.4171218715": "<snip>"
                        },
                        "meta": {
                            "<snip>": {
                                "create": 600000000000,
                                "delete": 1200000000000,
                                "update": 600000000000
                            },
                            "schema_version": "1"
                        },
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                },

Is this a mismatch between what terraform thinks it's updating the state to be and what state actually gets supplied by terraform refresh?

@bflad

This comment has been minimized.

Contributor

bflad commented May 9, 2018

Hi folks 👋 it looks like there are a couple of various issues reported here. If there is anything we can do to improve the resource documentation, please reach out. I believe the resource code handles each of the below as stated, but please let me know if that is not the case.

If you are trying to reference the root EBS block device (e.g. device name listed as the RootDeviceName of the AMI), it needs to be declared in your Terraform configuration as the root_block_device argument, not a ebs_block_device argument. There is likely room for improving the resource handling in this case.

If you are trying to create a second EBS block device, separate from the root EBS block device, it needs to have a device name that is different than the root EBS block device name.

If you are trying to resize the root EBS block device without recreation, that issue can be tracked here: #768.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment