AWS_Instance Always Recreates

[ahollings]

We used to deploy firewalls by injecting dynamic userdata into the AWS_INSTANCE resource with terraform, Whenever we deployed platform configuration with terraform the firewalls would only rebuild if their userdata configuration changed. This has been tested many times.

Recently new configuration took us past the 16K limit for userdata, now we save the firewall dynamic configuration in S3 and point the firewall at it a with JSON userdata which is much smaller. The firewalls deploy fine but repeated deployments failed to deploy because terraform was not seeing a change in the userdata.

To fix this we added the sha() value of the configuration data stored in S3 into the JSON boot text, thereby ensuring if the configuration changes the JSON boot userdata changes which will cause a rebuild.

Problem observed is the firewalls always destroy and recreate, it is definatly USERDATA which is causing this, using the lifecycle{} to ingore userdata causes the firewalls not to rebuild.

We have checked the content of the json and it's not changing but every time we deploy the firewalls rebuild. We have outputed the raw userdata and done a byte check and the sha value doesn't change unless we change configuration.

We have also tried doing this using a null resource with the same results.

Any help / ideas would be much appreciated.

terraform 12.3
terraform-providers/datadog 2.5.0
hashicorp/null 2.1.2
hashicorp/template 2.1.2
hashicorp/aws 2.45.0

FILE resource_fortigate_config.json

{
"bucket":"${waypoint_bucket}",
"region":"${region}",
"license":"",
"config":"fortigate_configuration.conf",
"versionSHA":"${sha1}"
}

data template_file config_fortigate_a_template_main {
template = "${file("resource_fortigate_config.conf")}"

DYNAMIC CONTENT

}

STORE FIREWALL CONFIGURATION IN ENCRYPTED S3 BUCKET (REMOVING CR\LF)

resource "aws_s3_bucket_object" fortigate_a_s3_main {
bucket = "${data.aws_s3_bucket.waypoint_bucket.id}"
key = "fortigate_configuration.conf"
content = "${replace(data.template_file.config_fortigate_a_template_main.rendered, "/\r\n/", "\n")}"
content_type = "text/plain"
force_destroy = false
}

READ FIREWALL CONFIGURATION, ABSTRACTED FROM ENCRYPTED S3 BUCKET

THIS IS USED TO CALCUATE SHA VALUE TO TRIGGER DEPLOYMENT ON CONFIGURATION CHANGE IN S3

data "aws_s3_bucket_object" fortigate_a_s3_main_read {
bucket = "${data.aws_s3_bucket.waypoint_bucket.id}"
key = "fortigate_configuration.conf"
depends_on = ["aws_s3_bucket_object.fortigate_a_s3_main"]
}

FIRWALL LAUNCH JSON NB. STORES CONFIGURATION SHA USED TO RECREATE AWS_INSTANCE IF THE CONFIGURATION CHANGES

data "template_file" "firewall_1a_main" {
template = "${file("resource_fortigate_config.json")}"
vars = {
waypoint_bucket = "${data.aws_s3_bucket.waypoint_bucket.id}"
region = "${var.amazon_region}"
sha1 = "${sha1(data.aws_s3_bucket_object.fortigate_a_s3_main_read.body)}"
}
}

resource "aws_instance" "firewall_1a_main" {
availability_zone = "${var.amazon_region}a"
ami = "${data.aws_ami.fortigate.image_id}"
instance_type = "t2.small"
disable_api_termination = false
monitoring = true
key_name = "A-Keypair"
iam_instance_profile = "${aws_iam_instance_profile.firewall_iam_profile_main.id}"
user_data = "${data.template_file.firewall_1a_main.rendered}"
depends_on = [ "aws_s3_bucket_object.fortigate_a_s3_main" ]
}

#####################################
Example of two builds and AWS_INSTANCE recreates both times. note the firewall_1a_main_firewall_boot output is the userdata which is not changing.

Build V1013
firewall_1a_main_firewall_boot = {
"bucket":"xxxxxxxxxxxx",
"region":"eu-west-1",
"license":"",
"config":"fortigate_configuration.conf",
"versionSHA":"ee35717f0c9b86c91524467667e0730ce8eddfa2"
}
firewall_1a_main_sha = ee35717f0c9b86c91524467667e0730ce8eddfa2
firewall_1a_main_userdata = 2c977815b8079bef7fbb211fbc40b12b3bebc4f2

Build v1012
firewall_1a_main_firewall_boot =
{
"bucket":"xxxxxxxxxxxx",
"region":"eu-west-1",
"license":"",
"config":"fortigate_configuration.conf",
"versionSHA":"ee35717f0c9b86c91524467667e0730ce8eddfa2"
}

firewall_1a_main_sha = ee35717f0c9b86c91524467667e0730ce8eddfa2
firewall_1a_main_userdata = 2c977815b8079bef7fbb211fbc40b12b3bebc4f2

[justinretzolk]

Hey @ahollings 👋 Thank you very much for taking the time to file this issue! Given that there's been a number of AWS provider releases since you filed it, can you confirm whether you're still experiencing this behavior?