v0.0.16 does not delete an outdated Rule in a Listener it manages, breaks the Service

[klyubin]

We've upgraded the Controller from v0.0.15 to v0.0.16 and now one of the Lattice Services the Controller manages has an extraneous Rule not described in any K8S resources. I think what's happening is that the naming conventions of the resources created by the Controller changed and it doesn't recognize the rules its former self-created before the upgrade. Regardless of the mechanism, the issue is that the v0.0.16 Controller does not ensure that the Lattice resources correspond to their K8S resource counterparts.

The first rule (Priority 1) below is the extraneous rule -- nowhere to be found in the K8S resources for this Lattice Service. The resulting service does not work because the Lattice Target Groupk8s-traffic-exemplar-traffic-exemplar is no longer even managed by the Controller and thus contains stale IPs.

Listener: traffic-exemplar-traffic-exemplar-4433-https
Routing:

• k8s-1693522631-rule-1:
  • Condition: Path (Prefix): / (case sensitive)
  • Action: Forward to k8s-traffic-exemplar-traffic-exemplar: 1 (100%)
  • Priority: 1
• k8s-1693522637-rule-1:
  • Condition: Header (Exact): target-group-type Value: staging-health-aws (case insensitive), Path (Prefix): / (case sensitive)
  • Action: Forward to k8s-traffic-exemplar-health-aws-traffic-exemplar-https-http2: 1 (100%)
  • Priority: 3
• k8s-1693522633-rule-1:
  • Condition: Path (Prefix): / (case sensitive)
  • Action: Forward to k8s-traffic-exemplar-traffic-exemplar-https-http2 : 1 (100%)
  • Priority: 2

K8S Services:

apiVersion: v1
kind: Service
metadata:
  name: traffic-exemplar
  namespace: traffic-exemplar
spec:
  clusterIP: None
  ports:
  - port: 4433
    protocol: TCP
    targetPort: 4433
  selector:
    app: traffic-exemplar
    envoy-target-group-type: staging
---
apiVersion: v1
kind: Service
metadata:
  name: traffic-exemplar-health-aws
  namespace: traffic-exemplar
spec:
  clusterIP: None
  ports:
  - port: 4433
    protocol: TCP
    targetPort: 4433
  selector:
    app: traffic-exemplar
    envoy-target-group-type: staging-health-aws

K8S HTTPRoutes:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: traffic-exemplar
  namespace: traffic-exemplar
spec:
  parentRefs:
  - name: square
    namespace: lattice
    sectionName: http
  - name: square
    namespace: lattice
    sectionName: https
  rules:
  - backendRefs:
    - kind: Service
      name: traffic-exemplar-health-aws
      port: 8080
      weight: 1
    matches:
    - headers:
      - name: target-group-type
        type: Exact
        value: development-health-aws
      path:
        type: PathPrefix
        value: /
  - backendRefs:
    - kind: Service
      name: traffic-exemplar
      port: 8080
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

K8S TargetGroupPolicies

apiVersion: application-networking.k8s.aws/v1alpha1
kind: TargetGroupPolicy
metadata:
  name: traffic-exemplar
  namespace: traffic-exemplar
spec:
  targetRef:
    group: ""
    kind: Service
    name: traffic-exemplar
  protocol: HTTPS
  protocolVersion: HTTP2
  healthCheck:
    enabled: true
    intervalSeconds: 10
    timeoutSeconds: 5
    healthyThresholdCount: 3
    unhealthyThresholdCount: 2
    path: "/_status"
    protocolVersion: HTTP2
    statusMatch: "200,403"
---
apiVersion: application-networking.k8s.aws/v1alpha1
kind: TargetGroupPolicy
metadata:
  name: traffic-exemplar-health-aws
  namespace: traffic-exemplar
spec:
  targetRef:
    group: ""
    kind: Service
    name: traffic-exemplar-health-aws
  protocol: HTTPS
  protocolVersion: HTTP2
  healthCheck:
    enabled: true
    intervalSeconds: 10
    timeoutSeconds: 5
    healthyThresholdCount: 3
    unhealthyThresholdCount: 2
    path: "/_status"
    protocolVersion: HTTP2
    statusMatch: "200,403"

[mikhail-aws]

we need to investigate it more, thank you for filling issue

[solmonk]

The controller is likely stuck at target registration here - backendRef port 8080 on your HTTPRoute means that you are only using targets that belong to the service port 8080, yet the services I see do not have any port on 4433. Have you tried changing backendRef ports to 4433 and check if it is happening?

The controller being stuck is being addressed in #386

[klyubin]

Thanks. Unfortunately, I deleted that broken Lattice Service after filing this ticket. So, I cannot reproduce without downgrading the Controller which I'd rather not do. In the end, I think regardless of whether ports are correct, there should be at most 2 Rules in any Listener(s) the above setup creates. Why would there be more than 2? There are no K8S resources here which suggest the need to have more than 2 Rules anywhere.

[solmonk]

In your case the controller cannot reach "the end state" - because of the issue mentioned above.

Our intended process of handling target group name change is:

1. New TGs with new naming schemes created
2. New rules created with new TGs
3. Stale rules are deleted
4. Stale TGs are deleted

What we are seeing here is most likely a state between 2 and 3.

[solmonk]

I tried a few setup to reproduce the issue but I was mostly stuck between step 1 and 2 instead. Along with the bugfix, I will continue to explore if there is any other chance of rule mismatch.

[graehren]

This should be resolved by our upcoming v0.0.18 release.

[graehren]

This should be resolved with PR #457 which will be available in our next release.