Provisionning multiple resources depending on one another

[ghost]

I am trying to create 3 EC2 instances for a Mesos cluster. I would then like to save the private_ip of all 3 instances in each of them:

resource "aws_instance" "mesos-master" {
    count = 3
    ami = "ami-cf35f3a4"
    instance_type = "t2.micro"

    provisioner "remote-exec" {
        inline = [
          "echo ${count.index+1} > /tmp/myid",
          "echo ${aws_instance.mesos-master.self.private_ip} > /tmp/myip",
          "echo ${element(aws_instance.mesos-master.*.private_ip, count.index)} > /tmp/masterips"
        ]
    }
}

Which gives me the following error when I run terraform plan:

connection info cannot contain splat variable referencing itself

I also tried this:

resource "aws_instance" "mesos-master" {
    count = 3
    ami = "ami-cf35f3a4"
    instance_type = "t2.micro"

    provisioner "remote-exec" {
        inline = [
          "echo ${count.index+1} > /tmp/myid",
          "echo ${aws_instance.mesos-master.self.private_ip} > /tmp/myip",
          "echo ${aws_instance.mesos-master.0.private_ip} > /tmp/masterip-1",
          "echo ${aws_instance.mesos-master.1.private_ip} > /tmp/masterip-2",
          "echo ${aws_instance.mesos-master.2.private_ip} > /tmp/masterip-3",
        ]
    }
}

Which does nothing and just hangs. Probably because it's stuck in a dependency cycle.
Is there a way to work around this and somehow wait for all instances to be created before provisioning all of them?

[phinze]

Good question! Pretty sure there's a way to accomplish this. I'll circle back when I get the chance to fiddle around with some examples and post my findings.

[nrcxcia]

Set the private_ip's to static values with variable lookups.

variable "instance_ips" {
  default = {
    "0" = "10.11.12.100"
    "1" = "10.11.12.101"
    "2" = "10.11.12.102"
  }
}

I think maybe provisioner should exist outside of resource.

[phinze]

Managed to accomplish this behavior with a "null_resource" - moving the provisioners to that allowed them to properly depend on the whole collection of EC2 instances booting first.

module "ami" {
  source = "github.com/terraform-community-modules/tf_aws_ubuntu_ami/ebs"
  region = "us-west-2"
  distribution = "trusty"
  instance_type = "t2.micro"
}

resource "aws_instance" "mesos-master" {
  count         = 3
  ami           = "${module.ami.ami_id}"
  instance_type = "t2.micro"
  key_name      = "tftest"
}

resource "null_resource" "configure-mesos-ips" {
  count = 3

  connection {
    user = "ubuntu"
    host = "${element(aws_instance.mesos-master.*.public_ip, count.index)}"
  }

  provisioner "remote-exec" {
    inline = [
      "echo '${element(aws_instance.mesos-master.*.private_ip, count.index)}' > /tmp/myip",
      "echo '${join(",", aws_instance.mesos-master.*.private_ip)}' > /tmp/masterips",
    ]
  }
}

This results in each machine having /tmp/myip with its private IP and /tmp/masterips with a comma separated list of master ips.

Let me know if this meets your needs, @mickael1787!

[ghost]

Thanks @phinze ! Didn't know about the "null_resource".

[kaelumania]

@phinze this does not work when one of the aws_instances is down and you reapply, then the provisioner for this instance is not being reexecuted. You can use the trigger option, but this results in executing the provisioner for ALL aws_instances not only for the one recreated.

[phinze]

Hi @kaelumania - once #3449 is fixed, that trigger strategy should work properly for you. 👍

[dhrsharma]

You can also use instance metadata endpoints in your user data, like LOCAL_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)

[birdypme]

Hello, you can also use the 'self' variable inside provisioners: terraform doc: interpolation "attributes of your own resource"

In your case, this fixes accessing the ip for the current resource, but you still need a null_resource to build the full list of ips:

resource "aws_instance" "mesos-master" {
    count = 3
    ami = "ami-cf35f3a4"
    instance_type = "t2.micro"

    provisioner "remote-exec" {
        inline = [
          "echo ${count.index+1} > /tmp/myid",
          "echo ${self.private_ip} > /tmp/myip",
        ]
    }
}

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.