BGP advertisement not working for LoadBalancer with externalTrafficPolicy: Local

[xoxys]

Expected Behavior

Successful connection to services within the CIDR of the serviceClusterIPs while using the externalTrafficPolicy: Cluster service.

Current Behavior

I have 3 LoadBalancer services on my cluster, the external IP addresses are allocated by MetallLB. A calico BGPConfiguration is used to advertise the CIDR used for the external addresses via BGP. While I can connect to both nging-ingress services via the external IP (both use externalTrafficPolicy: Local), I cannot connect to the mqtt service, which uses externalTrafficPolicy: Cluster. After changing externalTrafficPolicy: Cluster to externalTrafficPolicy: Local in the service homeassistant/mosquitto I can successfully connect to 10.168.65.47:8883.

Services and connection tests:

❯ kubectl get service -A | grep Load
nginx-public       ingress-nginx-public-controller              LoadBalancer   172.17.216.223   10.168.65.45   80:30199/TCP,443:30776/TCP   25d
homeassistant      mosquitto                                    LoadBalancer   172.17.213.36    10.168.65.47   8883:30891/TCP               10d
nginx-private      ingress-nginx-private-controller             LoadBalancer   172.17.33.119    10.168.65.46   80:31795/TCP,443:32707/TCP   25d

❯ nc -v 10.168.65.47 8883 -w 1
Ncat: Version 7.94 ( https://nmap.org/ncat )
Ncat: TIMEOUT.

❯ nc -v 10.168.65.45 443 -w 1
Ncat: Version 7.94 ( https://nmap.org/ncat )
Ncat: Connected to 10.168.65.45:443.

BGPConfiguration:

---
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  creationTimestamp: "2024-03-25T20:51:17Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: cluster-core-calico-bgp
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: default
  resourceVersion: "9847279"
  uid: 8a1b23b5-bf35-474c-80b5-7e3254c972c6
spec:
  asNumber: 64513
  serviceLoadBalancerIPs:
  - cidr: 10.168.65.0/26

Test after externalTrafficPolicy: Local was set:

{
  "allocateLoadBalancerNodePorts": true,
  "clusterIP": "172.17.213.36",
  "clusterIPs": [
    "172.17.213.36"
  ],
  "externalTrafficPolicy": "Local",
  "healthCheckNodePort": 31120,
  "internalTrafficPolicy": "Cluster",
  "ipFamilies": [
    "IPv4"
  ],
  "ipFamilyPolicy": "SingleStack",
  "ports": [
    {
      "name": "mqtt",
      "nodePort": 30891,
      "port": 8883,
      "protocol": "TCP",
      "targetPort": 8883
    }
  ],
  "selector": {
    "app.kubernetes.io/instance": "mosquitto",
    "app.kubernetes.io/name": "mosquitto"
  },
  "sessionAffinity": "None",
  "type": "LoadBalancer"
}

❯ nc -v 10.168.65.47 8883 -w 1
Ncat: Version 7.94 ( https://nmap.org/ncat )
Ncat: Connected to 10.168.65.47:8883.

Your Environment

• Calico version v3.27.3
• Orchestrator version (e.g. kubernetes, mesos, rkt): K3s 1.28
• Operating System and version: Rocky Linux 9.2

[xoxys]

I have also tried:

  serviceLoadBalancerIPs:
  - cidr: 10.168.65.45/32
  - cidr: 10.168.65.46/32
  - cidr: 10.168.65.47/32

But that also just works if the service is configured to use externalTrafficPolicy: Local.

What works (even if that looks wrong to me) is:

serviceExternalIPs:
  - cidr: 10.168.65.47/32
  serviceLoadBalancerIPs:
  - cidr: 10.168.65.45/32
  - cidr: 10.168.65.46/32

With this configuration, connections to all 3 services are working as expected.

[caseydavenport]

The main difference between the traffic flow for cluster / local is that cluster traffic will be SNAT'd upon arrival in the cluster - I wonder if perhaps there is something dropping that traffic. Do you have any network policy on the destination pods? Or cloud firewall rules that might impact cross node traffic?