Userdata scripts in EC2 do not get updated correctly

[jwlogemann]

Hi,
I've made some changes to my user-data script, which do not get applied by terraform to the EC2 instances. It appears to cache some old version of the script, and keeps applying that, instead of looking for local changes.

Terraform Version

Terraform v0.12.12

Terraform Configuration Files

ec2.tf:
resource "aws_instance" "ec2_instance" {
...
user_data = templatefile("${path.module}/../shared-templates/user-data.sh", {
log_group = aws_cloudwatch_log_group.ec2_log_group.name
additional_user_data = var.additional_user_data
})
...
}
first part of the user-data script:
#!/bin/bash
#Save script output
set -x
exec > >(tee /var/log/user-data.log|logger -t user-data ) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'

OS_NAME=$(cat /etc/os-release | grep ^NAME | cut -d '"' -f2 | cut -d ' ' -f1)
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
LOG_GROUP="${log_group}"
REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep '"region"' | cut -d" -f4)

Relevant part of tfstate:
{
"module": "module.b2b_db2",
"mode": "data",
"type": "template_file",
"name": "ec2_userdata",
"provider": "provider.template",
"instances": [
{
"schema_version": 0,
"attributes": {
"filename": null,
"id": "353e47d81936964543a43f677ac73701fec92eca5ff52074599ce675270b685e",
"rendered": "#!/bin/sh\n# Stream instance logs to CloudWatch Logs\nset -x\nOS_NAME=$(cat /etc/os-release|grep ^NAME|cut -d '"' -f2|cut -d ' ' -f1)\ngrep '/var/log/cfn-hup.log' /etc/awslogs/awslogs.conf\nif [ $? -ne 0 ]; then\n INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)\n LOG_GROUP="/ec2/b2b-DB2-instance-log-group"\n REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep '\"region\"' | cut -d\" -f4)\n # install the awslogs package\n if [$OS_NAME == "Redhat"] || [ $OS_NAME == "CentOS" ] || [ $OS_NAME == "Amazon" ];then\n yum install -y aws-cli awslogs\n elif [ $OS_NAME == "Ubuntu"]; then\n apt-get update \u0026\u0026 apt-get install awscli awslogs\n else\n echo "unsupported OS"\n fi\nfi\n\n# update awscli.conf with regions where logs to be sent\ngrep 'region = ' /etc/awslogs/awscli.conf\nif [ $? -ne 0 ]; then\n echo "region = ${REGION}" \u003e\u003e /etc/awslogs/awscli.conf\n else\n sed -i "s/region = ./region = ${REGION}/g" /etc/awslogs/awscli.conf\nfi\n\n# adding other log files\n\nfor log in $(find /var -iname \.log -o -name messages|tr '\n' ' ');\ndo\n echo -e "\n[${log}]\\n \nfile = ${log}\\n \nlog_group_name = ${LOG_GROUP}\\n \nlog_stream_name = ${INSTANCE_ID}${log}\\n \ninitial_position = start_of_file\\n \ndatetime_format = %b %d %H:%M:%S\\n \nbuffer_duration = 5000" \u003e\u003e /etc/awslogs/awslogs.conf\ndone\n\n# enable awslogd service\nsystemctl enable awslogsd\n# restart awslogs service\nsystemctl restart awslogsd\n# enable awslogs service to start on system boot\nchkconfig awslogsd on\n# Additional user data\necho "Running additional user data - setting up DB2 instance."\n\necho "Mounting EFS data volume"\nmkdir /data\nmount -t efs -o tls fs-94c4ddcd:/ /data\n",
"template": "#!/bin/sh\n# Stream instance logs to CloudWatch Logs\nset -x\nOS_NAME=$(cat /etc/os-release|grep ^NAME|cut -d '"' -f2|cut -d ' ' -f1)\ngrep '/var/log/cfn-hup.log' /etc/awslogs/awslogs.conf\nif [ $? -ne 0 ]; then\n INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)\n LOG_GROUP="${log_group}"\n REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep '\"region\"' | cut -d\" -f4)\n # install the awslogs package\n if [$OS_NAME == "Redhat"] || [ $OS_NAME == "CentOS" ] || [ $OS_NAME == "Amazon" ];then\n yum install -y aws-cli awslogs\n elif [ $OS_NAME == "Ubuntu"]; then\n apt-get update \u0026\u0026 apt-get install awscli awslogs\n else\n echo "unsupported OS"\n fi\nfi\n\n# update awscli.conf with regions where logs to be sent\ngrep 'region = ' /etc/awslogs/awscli.conf\nif [ $? -ne 0 ]; then\n echo "region = $${REGION}" \u003e\u003e /etc/awslogs/awscli.conf\n else\n sed -i "s/region = ./region = $${REGION}/g" /etc/awslogs/awscli.conf\nfi\n\n# adding other log files\n\nfor log in $(find /var -iname \.log -o -name messages|tr '\n' ' ');\ndo\n echo -e "\n[$${log}]\\n \nfile = $${log}\\n \nlog_group_name = $${LOG_GROUP}\\n \nlog_stream_name = $${INSTANCE_ID}$${log}\\n \ninitial_position = start_of_file\\n \ndatetime_format = %b %d %H:%M:%S\\n \nbuffer_duration = 5000" \u003e\u003e /etc/awslogs/awslogs.conf\ndone\n\n# enable awslogd service\nsystemctl enable awslogsd\n# restart awslogs service\nsystemctl restart awslogsd\n# enable awslogs service to start on system boot\nchkconfig awslogsd on\n# Additional user data\n${additional_user_data}\n",
"vars": {
"additional_user_data": "echo "Running additional user data - setting up DB2 instance."\n\necho "Mounting EFS data volume"\nmkdir /data\nmount -t efs -o tls fs-94c4ddcd:/ /data",
"log_group": "/ec2/b2b-DB2-instance-log-group"
}
},

Note that the userdata script is completely different. The local one starts with #!/bin/bash, and the tfstate one with #!/bin/sh for instance

Debug Output

https://drive.google.com/file/d/1IRYu14QYnKhjlPW5S2G4JDDM9MzaEeV_/view?usp=sharing

Expected Behavior

New user-data should have been applied

Actual Behavior

Terraform did not change anything

Steps to Reproduce

1. terraform init
2. terraform apply

[jwlogemann]

I also tried to do a 'terraform destroy' and then a new 'terraform apply' but that doesn't make a difference

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#10696 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-aws#10696.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.