error finding matching route for Route table and destination CIDR block randomly

[kiich]

This is a very specific setup and it probably is the reason why i am seeing it as i can not reproduce this problem when i run it on my laptop and so on (classic excuse!! :-)).

The setup

We have a docker container that is running a deploy script which is an ansible playbook. As part of this deploy ansible playbook, we ssh to a target server where we then run terraform on.

When i run the ansible playbook on my laptop on its own, the terraform runs fine and completes as success.

But when the ansible playbook is ran inside the docker container (everything else downstream is the same as when i ran on laptop), i see error messages "error finding matching route for Route table and destination CIDR block" where the route it complains about is random.

The terraform file simply does this:
# following block takes care of igw and vpg creation resource "aws_route" "igw_or_vpg_route" { count = "${length(compact(split(",", var.cidr_block))) * var.CREATE_ROUTES}" route_table_id = "${var.route_table_id}" destination_cidr_block = "${element(split(",", var.cidr_block),count.index)}" gateway_id = "${var.gateway_id}" }
with variables like these:
CREATE_ROUTES = 1 route_table_id = "${module.routetable.route_table_id}" gateway_id = "${module.vpc.vpg_id}" cidr_block = "11.111.0.0/16,11.112.0.0/16,11.113.0.0/16,11.114.0.0/16,11.115.0.0/16,11.116.0.0/12"

and it fails with:
`

• aws_route.igw_or_vpg_route.0:
  error finding matching route for Route table (rtb-36514b52) and destination CIDR block (11.111.0.0/16)
• aws_route.igw_or_vpg_route.1:
  error finding matching route for Route table (rtb-36514b52) and destination CIDR block (11.112.0.0/16)
• aws_route.igw_or_vpg_route.4:
  error finding matching route for Route table (rtb-36514b52) and destination CIDR block (11.114.0.0/16)
  `

The route it fails on is random it seems.

I can reproduce this error all the time on the docker container but never when i run the ansible play on my laptop so i appreciate it probably is not terraform issue but i wanted to know what kind of condition can cause this.

FYI if i set:
-parallelism=1

as part of my terraform apply, this makes it work inside the container so i know it is to do with some kind of thread conditions/timing issue.

Also after getting the terraform error, when i go back to the console in AWS, the routes are actually created fine.

Terraform Version

Terraform v0.6.15

Affected Resource(s)

• aws_route

Expected Behavior

All routes created successfully and terraform apply finishing successfully.

Actual Behavior

All routes gets created successfully in AWS console but terraform apply failed with the following error:

• aws_route.igw_or_vpg_route:
  error finding matching route for Route table (rtb-36514b52) and destination CIDR block (0.0.0.0/0)
• aws_route.igw_or_vpg_route.0:
  error finding matching route for Route table (rtb-36514b52) and destination CIDR block (11.111.0.0/16)
• aws_route.igw_or_vpg_route.1:
  error finding matching route for Route table (rtb-36514b52) and destination CIDR block (11.112.0.0/16)
• aws_route.igw_or_vpg_route.4:
  error finding matching route for Route table (rtb-36514b52) and destination CIDR block (11.114.0.0/16)

Steps to Reproduce

1. terraform apply

Important Factoids

We call a module that creates the routes in route table.

[kiich]

Linking possible workaround and might be related issue
#5335

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.