GatewayRoute Selector Inconsistent Behavior

[egkelly]

Describe the bug
I have two virtual gateways, one internet-facing and the other internal-facing deployed within the same namespace. I have this setup in two separate environments.

In one environment, the gateway route selector is working as expected; my internal-only gateway route is properly bound to the internal virtual gateway, and all other gateway routes default to the internet-facing virtual gateway. In the other environment however, the internal virtual gateway is ignored, and all gateway routes bind to the internet-facing virtual gateway including my internal gateway route. Both environments are configured in the same way with the same selectors, as shown below:

Broken Env Gateway Route:

apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
  labels:
    argocd.argoproj.io/instance: app-staging
    gateway: ingress-gw-internal
    ...
    ...
    ...
  virtualGatewayRef:
    name: ingress-gw-external
    namespace: staging
status:
  gatewayRouteARN: >-
    arn:aws:appmesh:us-east-1:XXXXXXX:mesh/XXXXXX/virtualGateway/ingress-gw-external_staging/gatewayRoute/gatewayroute_staging1

Broken Env Virtual Gateway:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualGateway
  labels:
    argocd.argoproj.io/instance: my-mesh
  name: ingress-gw-internal
  namespace: staging
spec:
  ...
  ...
  gatewayRouteSelector:
    matchLabels:
      gateway: ingress-gw-internal
  meshRef:
    name: my-mesh
  namespaceSelector:
    matchLabels:
      gateway: ingress-gw-internal
  podSelector:
    matchLabels:
      app.kubernetes.io/name: ingress-gw-internal

Working Env Gateway Route

apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
  labels:
    argocd.argoproj.io/instance: app-production
    gateway: ingress-gw-internal
    ...
    ...
    ...
  virtualGatewayRef:
    name: ingress-gw-internal
    namespace: production
status:
  gatewayRouteARN: >-
    arn:aws:appmesh:us-east-1:XXXXXXX:mesh/XXXXXX/virtualGateway/ingress-gw-internal_production/gatewayRoute/gatewayroute_production

Working Env Virtual Gateway:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualGateway
  labels:
    argocd.argoproj.io/instance: my-mesh
  name: ingress-gw-internal
  namespace: production
spec:
  ...
  ...
  gatewayRouteSelector:
    matchLabels:
      gateway: ingress-gw-internal
  meshRef:
    name: my-mesh
  namespaceSelector:
    matchLabels:
      gateway: ingress-gw-internal
  podSelector:
    matchLabels:
      app.kubernetes.io/name: ingress-gw-internal

Steps to reproduce
I'm not sure how to reproduce this. It's cropped up several times with little apparent trend. I have fixed this after deleting the entire set of virtual gateways and gateway routes and redeploying several times, but often times it will go back to ignoring the gateway route selector if I need to redeploy after getting it working again.

Expected outcome
The internal gateway route should bind to the internal virtual gateway according to its gatewayrouteselector, and all other gateway routes should default to the virtual gateway with no gatewayrouteselector configured.

Environment

• App Mesh controller version v1.10.0
• Envoy version v1.24.0.0-prod
• Kubernetes version 1.24
• Using EKS (yes/no), if so version? eks.4