S3 and Dynamodb endpoints keep getting replaced

[aaditi30]

Problem Summary:

Provider Version: 3.31
Terraform version: v0.14.11

Creating s3 and dynamodb vpc endpoints.

locals{
  s3_service_name = data.aws_vpc_endpoint_service.s3[0].service_name
  vpc_id          = aws_vpc.vpc[0].id
}

resource "aws_vpc_endpoint_service" "s3" {
  count = var.create_vpc ? 1 : 0
  service     = "s3"
  type        = "Gateway"
}

resource "aws_vpc_endpoint" "s3_endpoint" {
  count = var.create_vpc ? 1 : 0
  vpc_id            = local.vpc_id
  service_name      = local.s3_service_name
}

resource "aws_vpc_endpoint" "dynamodb_endpoint" {
  vpc_id       = aws_vpc.vpc[0].id
  service_name = data.aws_vpc_endpoint_service.dynamodb_endpoint_service.service_name
}

Each time we run terraform plan the endpoint service is finding new endpoint in the backend and these endpoints keep getting replaced.

Output

module.vpc.data.aws_vpc_endpoint_service.dynamodb_endpoint_service will be read during apply
  (config refers to values not yet known)
 <= data "aws_vpc_endpoint_service" "dynamodb_endpoint_service"  {
      ~ acceptance_required           = false -> (known after apply)
      ~ arn                           = "**arn:aws:ec2:us-west-2:XXX:vpc-endpoint-service/vpce-svc-06e332dbde3bXX"** -> (known after apply)
      ~ availability_zones            = [
          - "us-west-2a",
          - "us-west-2b",
          - "us-west-2c",
          - "us-west-2d",
        ] -> (known after apply)
      ~ base_endpoint_dns_names       = [
          - "dynamodb.us-west-2.amazonaws.com",
        ] -> (known after apply)
      ~ id                            = "3114548494" -> (known after apply)
      ~ manages_vpc_endpoints         = false -> (known after apply)
      ~ owner                         = "amazon" -> (known after apply)
      + private_dns_name              = (known after apply)
      ~ service_id                    = "vpce-svc-06e332dbde3bSSS" -> (known after apply)
      ~ service_name                  = "com.amazonaws.us-west-2.dynamodb" -> (known after apply)
      ~ service_type                  = "Gateway" -> (known after apply)
      ~ tags                          = {} -> (known after apply)
      ~ vpc_endpoint_policy_supported = true -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  module.vpc.aws_vpc_endpoint.dynamodb_endpoint must be replaced
-/+ resource "aws_vpc_endpoint" "dynamodb_endpoint" {
      ~ arn                   = "**arn:aws:ec2:us-west-2:XXX:vpc-endpoint/vpce-07779dbede027a3bf**" -> (known after apply)
      ~ cidr_blocks           = [
          - "XXX",
          - "XX",
        ] -> (known after apply)
      ~ dns_entry             = [] -> (known after apply)
      ~ id                    = "vpce-07779dbede027a3bf" -> (known after apply)
      ~ network_interface_ids = [] -> (known after apply)
      ~ owner_id              = "XXX" -> (known after apply)

We can clearly see the service arn changing in the data call aws_vpc_endpoint_service for every plan output.

Tried adding the aws_vpc_endpoint_service data resource, adding the service arn and vpc arn in local

Referred issues:

#13383
#3777

Expected Behavior:

Once apply resource should not get modified unless there is a specific change made in the resource.

Actual Behavior

The resource gets modified on every apply.

[justinretzolk]

Hi @aaditi30 👋 Thank you for taking the time to raise this! I see that the aws_vpc_endpoint_service data source has the following note next to it in the plan log:

module.vpc.data.aws_vpc_endpoint_service.dynamodb_endpoint_service will be read during apply
  (config refers to values not yet known)

Can you supply the configuration for this data source as well? Based on the plan log, it appears that what is happening is that there is an argument for the data source that is not known at plan time, and so the data source must be read during the apply phase on each run (see the Data Resource Behavior documentation for more information). Because the aws_vpc_endpoint depends on this data source, the service_name argument won't be known until apply time. When the service_name argument is changed, the resource must be replaced, leading to the aws_vpc_endpoint resource being replaced on each run.

[zack-is-cool]

This is happening for us sporadically. It doesn't happen all the time. Sometimes we apply and suddenly all of our VPC endpoints need to be replaced. We aren't changing anything going into the VPC module or relating to the endpoints. Sidenote - we are deploying in govcloud.

We're using this module:

https://github.com/defenseunicorns/terraform-aws-vpc/blob/main/main.tf#L130-L256 which feeds into this ->
https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/master/modules/vpc-endpoints/main.tf#L11-L21

truncated output, but it does this for all of our endpoints that get fed from that aws_vpc_endpoint_service data that feed into the aws_vpc_endpoint resource

         ~ requester_managed     = false -> (known after apply)                                                                                                                  
           ~ route_table_ids       = [] -> (known after apply)                                                                                                                     
           ~ service_name          = "com.amazonaws.us-gov-west-1.ssm" # forces replacement -> (known after apply) # forces replacement                                            
           ~ state                 = "available" -> (known after apply)