New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic IP assignment #557

Closed
vacquah opened this Issue Nov 9, 2014 · 10 comments

Comments

Projects
None yet
@vacquah

vacquah commented Nov 9, 2014

I noticed that eips are not assigned when I add the "remote-exec" provisioner to my instance code. when I comment out the provisioner portion of the code, the eip is assgined correctly. Have been able to reproduce this several times. Here is the code:

https://gist.github.com/e481ea8b37d478296396

@sethvargo sethvargo changed the title from eip assignment to Elastic IP assignment Nov 19, 2014

@CpuID

This comment has been minimized.

Show comment
Hide comment
@CpuID

CpuID Jan 15, 2015

Contributor

Yep this seems to cause some issues. I think it's a resource ordering issue.

Pretty sure Terraform is trying to complete the provisioners within the aws_instance resources before moving on to add the aws_eip resources. This means that instances are stuck with a private only IP during the provisioning phase. If your provisioner does anything that requires external network access it will hang during. This then causes Terraform to never get to the aws_eip resources.

Ideally what needs to be done, is the aws_eip resources should be executed right after the instances come online but before the aws_instance provisioners commence.

Contributor

CpuID commented Jan 15, 2015

Yep this seems to cause some issues. I think it's a resource ordering issue.

Pretty sure Terraform is trying to complete the provisioners within the aws_instance resources before moving on to add the aws_eip resources. This means that instances are stuck with a private only IP during the provisioning phase. If your provisioner does anything that requires external network access it will hang during. This then causes Terraform to never get to the aws_eip resources.

Ideally what needs to be done, is the aws_eip resources should be executed right after the instances come online but before the aws_instance provisioners commence.

@brycekahle

This comment has been minimized.

Show comment
Hide comment
@brycekahle

brycekahle Jan 22, 2015

I know it is backwards, but have you tried putting the provisioner on the EIP resource?

brycekahle commented Jan 22, 2015

I know it is backwards, but have you tried putting the provisioner on the EIP resource?

@mdulghier

This comment has been minimized.

Show comment
Hide comment
@mdulghier

mdulghier Jan 24, 2015

Putting the remote-exec provisioner on the aws_eip resource actually works. You just have to define the connection, e.g.

resource "aws_eip" "example-ip" {
    instance = "${aws_instance.example.id}"
    vpc = true
    connection {
        host = "${aws_eip.example-ip.public_ip}"
        user = "ubuntu"
        key_file = "${var.ssh_key}"
    }
    provisioner "remote-exec" {
        script = "setup.sh"
    }
}

mdulghier commented Jan 24, 2015

Putting the remote-exec provisioner on the aws_eip resource actually works. You just have to define the connection, e.g.

resource "aws_eip" "example-ip" {
    instance = "${aws_instance.example.id}"
    vpc = true
    connection {
        host = "${aws_eip.example-ip.public_ip}"
        user = "ubuntu"
        key_file = "${var.ssh_key}"
    }
    provisioner "remote-exec" {
        script = "setup.sh"
    }
}
@duggan

This comment has been minimized.

Show comment
Hide comment
@duggan

duggan Feb 10, 2015

Contributor

When updating security groups (or other instance-destructive action), this means any provisioning steps will have to be run manually, as they are not invoked under the instance resource as they otherwise would be.

Contributor

duggan commented Feb 10, 2015

When updating security groups (or other instance-destructive action), this means any provisioning steps will have to be run manually, as they are not invoked under the instance resource as they otherwise would be.

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Feb 18, 2015

Member

I agree with @brycekahle that it sounds backwards but it actually sounds the best for me that in this case you should run the provisioner on the EIP and setup the connection to talk to the EIP. It isn't ideal but this would be a pretty hard problem to fix at the moment.

Member

mitchellh commented Feb 18, 2015

I agree with @brycekahle that it sounds backwards but it actually sounds the best for me that in this case you should run the provisioner on the EIP and setup the connection to talk to the EIP. It isn't ideal but this would be a pretty hard problem to fix at the moment.

@duggan

This comment has been minimized.

Show comment
Hide comment
@duggan

duggan Feb 25, 2015

Contributor

There's also a side effect here that the public DNS of the instance is incorrect for provisioning, meaning one can't make use of the (very convenient) split horizon DNS resolution of AWS (where externally it resolves to the elastic IP, but resolves to the private interface within the network).

Edit:

For what it's worth, a workaround for this is to query the EC2 metadata API on the provisioner, like:

provisioner "remote-exec" {
                inline = [
                        "sudo sed -i s/ADVERTISED_HOST/$(curl http://169.254.169.254/latest/meta-data/public-hostname)/ /etc/foo.conf",
                ]
        }
Contributor

duggan commented Feb 25, 2015

There's also a side effect here that the public DNS of the instance is incorrect for provisioning, meaning one can't make use of the (very convenient) split horizon DNS resolution of AWS (where externally it resolves to the elastic IP, but resolves to the private interface within the network).

Edit:

For what it's worth, a workaround for this is to query the EC2 metadata API on the provisioner, like:

provisioner "remote-exec" {
                inline = [
                        "sudo sed -i s/ADVERTISED_HOST/$(curl http://169.254.169.254/latest/meta-data/public-hostname)/ /etc/foo.conf",
                ]
        }
@phinze

This comment has been minimized.

Show comment
Hide comment
@phinze

phinze Nov 23, 2015

Member

Checking in on some old issues here. The fact that provisioners for a given resource before its dependencies are processed is pretty inherent to the way the dependency graph works. Today we'd recommend the same solution of dropping the provisioners later in the graph, either by adding them to the aws_eip resource directly or via a null_resource placed afterwards.

These options can be considered first-class strategies for accomplishing this, and I can't picture any way for us to implicitly make the assignment occur before the instance's provisioners run, so I'm going to close this issue. Feel free to follow up if you think this should still be open. 👍

Member

phinze commented Nov 23, 2015

Checking in on some old issues here. The fact that provisioners for a given resource before its dependencies are processed is pretty inherent to the way the dependency graph works. Today we'd recommend the same solution of dropping the provisioners later in the graph, either by adding them to the aws_eip resource directly or via a null_resource placed afterwards.

These options can be considered first-class strategies for accomplishing this, and I can't picture any way for us to implicitly make the assignment occur before the instance's provisioners run, so I'm going to close this issue. Feel free to follow up if you think this should still be open. 👍

@phinze phinze closed this Nov 23, 2015

@rcoh

This comment has been minimized.

Show comment
Hide comment
@rcoh

rcoh Aug 11, 2016

FYI: if you key the null_resource off of the instance.id you run into the same issue :-/
You need to set the connection to the aws_eip.xxx.public_ip_address

rcoh commented Aug 11, 2016

FYI: if you key the null_resource off of the instance.id you run into the same issue :-/
You need to set the connection to the aws_eip.xxx.public_ip_address

@hfuops

This comment has been minimized.

Show comment
Hide comment
@hfuops

hfuops Aug 15, 2016

@rcoh , unfortunately I ran into the issue as you mentioned :(, and a morning of time was lost due to this.
Here is the final sample I've tested successfully. FYI, hope this could ease the pain of someone who cloud meet the same problem in future.

resource "aws_eip" "demo" {
  instance = "${aws_instance.TS-demo.id}"
  depends_on = ["aws_instance.TS-demo"]
  vpc      = true
}

resource "aws_instance" "TS-demo" {
  ami                    = "${lookup(var.aws_opsman_ami,var.region)}"
  instance_type          = "m3.medium"
  key_name               = "${var.key_name}"
  subnet_id              = "${aws_subnet.public.id}"
  vpc_security_group_ids = ["${aws_security_group.all_pass.id}"]

  tags {
    Name = "demo"
  }

  provisioner "local-exec" {
    command = "echo ${aws_instance.TS-demo.private_ip} > private_ips"
  }

}
resource "null_resource" "preparation" {
    triggers {
        instance = "${aws_instance.TS-demo.id}"
    }
  connection {
    host        ="${aws_eip.demo.public_ip}"   # don't forget  this option.
    user        = "ubuntu"
    timeout     = "30s"
    private_key = "${file("./skyfree.pem")}"
    agent = false
  }

  provisioner "file" {
    source      = "./tfvars"
    destination = "/tmp/"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get -y update",
    ]
  }

}

hfuops commented Aug 15, 2016

@rcoh , unfortunately I ran into the issue as you mentioned :(, and a morning of time was lost due to this.
Here is the final sample I've tested successfully. FYI, hope this could ease the pain of someone who cloud meet the same problem in future.

resource "aws_eip" "demo" {
  instance = "${aws_instance.TS-demo.id}"
  depends_on = ["aws_instance.TS-demo"]
  vpc      = true
}

resource "aws_instance" "TS-demo" {
  ami                    = "${lookup(var.aws_opsman_ami,var.region)}"
  instance_type          = "m3.medium"
  key_name               = "${var.key_name}"
  subnet_id              = "${aws_subnet.public.id}"
  vpc_security_group_ids = ["${aws_security_group.all_pass.id}"]

  tags {
    Name = "demo"
  }

  provisioner "local-exec" {
    command = "echo ${aws_instance.TS-demo.private_ip} > private_ips"
  }

}
resource "null_resource" "preparation" {
    triggers {
        instance = "${aws_instance.TS-demo.id}"
    }
  connection {
    host        ="${aws_eip.demo.public_ip}"   # don't forget  this option.
    user        = "ubuntu"
    timeout     = "30s"
    private_key = "${file("./skyfree.pem")}"
    agent = false
  }

  provisioner "file" {
    source      = "./tfvars"
    destination = "/tmp/"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get -y update",
    ]
  }

}
@afeld

This comment has been minimized.

Show comment
Hide comment
@afeld

afeld Nov 25, 2017

Contributor

/cc #2585

Contributor

afeld commented Nov 25, 2017

/cc #2585

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment