Unable to set target attribute proxy_protocol_v2 to false for aws-load-balancer-nlb-target-type: "ip"

[niravshah2705]

Describe the bug
Unable to connect to mongodb from TCP connection. Manually chaning the value for proxy_protocol_v2 works.

Steps to reproduce
Setup Mongodb using bitnami helm chart:

  type: LoadBalancer
  port: 27017
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"  
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "27017"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "${subnets}"
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "false"
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=false
    service.beta.kubernetes.io/aws-load-balancer-alpn-policy: None
architecture: standalone

resource "helm_release" "mongodb" {
  name       = "mongodb"
  namespace  = var.stage
  repository = "oci://registry-1.docker.io/bitnamicharts/"
  chart      = "mongodb"
  version    = "15.1.4"
  values = [
    templatefile(
      "templates/mongo.yaml", {
        subnets = "subnet-1,subnet-2"
      }
    )
  ]
}

Checking logs for aws loadbalancer always shows that proxy_protocol_v2 set to true

{"level":"info","ts":"2024-04-21T16:50:03Z","logger":"controllers.service","msg":"successfully built model","model":"{\"id\":\"latest/mongodb\",\"resources\":{\"AWS::EC2::SecurityGroup\":{\"ManagedLBSecurityGroup\":{\"spec\":{\"groupName\":\"k8s-latest-mongodb-6213e0c2a2\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"ingress\":[{\"ipProtocol\":\"tcp\",\"fromPort\":27017,\"toPort\":27017,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]}]}}},\"AWS::ElasticLoadBalancingV2::Listener\":{\"27017\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":27017,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/latest/mongodb:27017/status/targetGroupARN\"}}]}}]}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-latest-mongodb-4c15c92d4a\",\"type\":\"network\",\"scheme\":\"internal\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-05b2f08f761724973\"},{\"subnetID\":\"subnet-0d3cf964eb7c7b7b1\"}],\"securityGroups\":[{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"},\"sg-081e82f792a73cf8e\"]}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"latest/mongodb:27017\":{\"spec\":{\"name\":\"k8s-latest-mongodb-7cda93f04b\",\"targetType\":\"ip\",\"port\":1,\"protocol\":\"TCP\",\"ipAddressType\":\"ipv4\",\"healthCheckConfig\":{\"port\":27017,\"protocol\":\"TCP\",\"intervalSeconds\":10,\"timeoutSeconds\":10,\"healthyThresholdCount\":3,\"unhealthyThresholdCount\":3},\"targetGroupAttributes\":[{\"key\":\"preserve_client_ip.enabled\",\"value\":\"true\"},{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"true\"}]}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"latest/mongodb:27017\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-latest-mongodb-7cda93f04b\",\"namespace\":\"latest\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/latest/mongodb:27017/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"mongodb\",\"port\":27017},\"networking\":{\"ingress\":[{\"from\":[{\"securityGroup\":{\"groupID\":\"sg-081e82f792a73cf8e\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":\"mongodb\"},{\"protocol\":\"TCP\",\"port\":27017}]}]},\"ipAddressType\":\"ipv4\"}}}}}}}"}
{"level":"info","ts":"2024-04-21T16:50:04Z","logger":"controllers.service","msg":"modifying targetGroup attributes","stackID":"latest/mongodb","resourceID":"latest/mongodb:27017","arn":"arn:aws:elasticloadbalancing:us-east-1:566211916844:targetgroup/k8s-latest-mongodb-7cda93f04b/fa17eb782f213667","change":{"preserve_client_ip.enabled":"true"}}
{"level":"info","ts":"2024-04-21T16:50:04Z","logger":"controllers.service","msg":"modified targetGroup attributes","stackID":"latest/mongodb","resourceID":"latest/mongodb:27017","arn":"arn:aws:elasticloadbalancing:us-east-1:566211916844:targetgroup/k8s-latest-mongodb-7cda93f04b/fa17eb782f213667"}
{"level":"debug","ts":"2024-04-21T16:50:04Z","logger":"controllers.service","msg":"Res and Sdk Listener don't have SSL Policy set, skip updating extra certs for non-TLS listener."}
{"level":"info","ts":"2024-04-21T16:50:04Z","logger":"controllers.service","msg":"successfully deployed model","service":{"namespace":"latest","name":"mongodb"}}
{"level":"debug","ts":"2024-04-21T16:50:04Z","logger":"events","msg":"Successfully reconciled","type":"Normal","object":{"kind":"Service","namespace":"latest","name":"mongodb","uid":"5d2c2a45-c44b-4d74-9b02-677e939b32fa","apiVersion":"v1","resourceVersion":"4416344"},"reason":"SuccessfullyReconciled"}

tcp connection works.
If we go to AWS UI & change value manually we are able to connect mongodb using mongosh command, else it is giving below error!

mongosh mongodb://mymongodns.dns.com:27017
Warning: Could not access file: ENOENT: no such file or directory, mkdir '/.mongodb/mongosh'
Current Mongosh Log ID:	662546ce3582fd3722ef634a
Connecting to:		mongodb://mymongodns.dns.com:27017/?directConnection=true&appName=mongosh+2.2.3

MongoServerSelectionError: Server selection timed out after 30000 ms

Expected outcome
TCP communication work for mongodb with IaaC too.

Environment
Production

• AWS Load Balancer controller version: 2.7.2
• Kubernetes version: 1.29
• Using EKS (yes/no), if so version? yes, 1.29

Additional Context:
I tried changing value via cli, then also faced same issue.

aws elbv2 modify-target-group-attributes \
                                       --target-group-arn arn:aws:elasticloadbalancing:regin:account:targetgroup/mongodb-abc/fa17eb782f213667 \
                                       --attributes 'Key=proxy_protocol_v2.enabled,Value=false'