error modifying Target Group Attributes: TargetGroupNotFound

[123BLiN]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

• terraform 0.15.3
• aws provider 3.42

Affected Resource(s)

• aws_alb_target_group

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_alb_target_group" "test" {
  name                 = "test"
  deregistration_delay = 15
  port                 = 8888
  protocol             = "HTTP"
  target_type          = "instance"
  vpc_id               = "[vpc_id]"

  health_check {
    healthy_threshold = 2
    interval          = 5
    matcher           = "200"
    path              = "/health"
    port              = 8888
    timeout           = 2
  }

  stickiness {
    type            = "lb_cookie"
    cookie_duration = 3600
    enabled         = true
  }

  lifecycle {
    create_before_destroy = true
  }
}

Panic Output

Error: error modifying Target Group Attributes: TargetGroupNotFound: Target groups 'arn:aws:elasticloadbalancing:us-west-2:xxxxxxxxx:targetgroup/sdfsdfdgdfgdfbdf323r34r/fsfs44wgdg4g4g4' not found

Steps to Reproduce

It fails irregularly without any pattern, almost not possible to reproduce intentionally, some inconsistency is present probably on AWS side, I've found a lot of improvements for this behaviour, for example for tags, but in that case we have the same but only for attributes

References

Maybe related to this change #18634
Looks similar to #17280 but for attributes

Notes

The other issue is that terraform exit code is 0 on this - but I'll create another issue in terraform core for this probably.

[ryanrupp]

Looks to be the same issue as #20454 or related at least - eventual consistency problem so AWS doesn't think the just created target group exists

[eccentrichead]

Is there any progress in terms of fixes? In my current setup, the target group error occurs randomly during the NLB target creation. The terraform logs says, that those targetgroups are created but it fails

aws_lb.deveks-sb-NLB: Still creating... [1m50s elapsed]
aws_lb.deveks-sb-NLB: Still creating... [2m0s elapsed]
aws_lb.deveks-sb-NLB: Still creating... [2m10s elapsed]
aws_lb.deveks-sb-NLB: Still creating... [2m20s elapsed]
aws_lb.deveks-sb-NLB: Still creating... [2m30s elapsed]
aws_lb.deveks-sb-NLB: Creation complete after 2m34s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:loadbalancer/net/accessplatform-2-sb-nlb-zugf/76c7c582a7302c52]
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-udp["xxxxx"]: Creating...
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:targetgroup/tg-xxxxx-zugf/071ec5ffc81bd057]
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:targetgroup/tg-xxxxx-zugf/05c057f18540bc9e]
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:targetgroup/tg-xxxxx-zugf/fc78021225300385]
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:targetgroup/tg-xxxxx-zugf/f52d4ee4a11a6378]
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:targetgroup/tg-xxxxx-zugf/f7a8ca4931b2e0d9]
aws_lb_target_group.nlb-sb-targetgroups-tcp["xxxxx"]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:targetgroup/tg-xxxxx-zugf/6e3343b979e024b3]
aws_lb_target_group.nlb-sb-targetgroups-udp["xxxxx"]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:xxxxx:targetgroup/tg-xxxxx-zugf/d3e366d238e31db4]

but on the console, these targets are not seen under the NLBs. So the subsequent stage of mapping it to a listener and targetgroupbindings (EKS env in our case) it fails with the error. This has been randomly observed with RHOCP installer as well, in the past and also recently in the upcoming 4.12 version: https://issues.redhat.com/browse/OCPBUGS-4158.