BGP Advertised Path Attributes fails to match CiliumLoadBalancerIPPool

[dswaffordcw]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

This bug report is a follow-up from https://cilium.slack.com/archives/C53TG4J4R/p1716491937260819

I attempted to configure the Advertised Paths feature using https://docs.cilium.io/en/stable/network/bgp-control-plane/#advertised-path-attributes. Before configuring the Advertised Paths feature, I had established BGP peering and a BGP path was being announced. This was seen in:

 cilium bgp routes
(Defaulting to `available ipv4 unicast` routes, please see help for more options)

Node                              VRouter   Prefix          NextHop    Age      Attrs
bgp-cplane-dev-v4-control-plane   65001     100.64.8.2/32   10.0.1.1   41m37s   [{Origin: i} {AsPath: 65000 65002} {Nexthop: 10.0.1.1}]
bgp-cplane-dev-v4-worker          65002     100.64.8.2/32   0.0.0.0    46m52s   [{Origin: i} {Nexthop: 0.0.0.0}]

For the Advertised Paths feature, I had configured:

      advertisedPathAttributes:
      - selectorType: CiliumLoadBalancerIPPool
        selector:
          matchLabels:
            environment: production
        communities:
          standard:
          - 65001:100

With this configuration, no BGP communities were present on the announced path.

The full configuration is below:

cat bgpp.yaml

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: worker
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: bgp-cplane-dev-v4-worker
  virtualRouters:
  - localASN: 65002
    serviceSelector:
      matchExpressions:
       - { key: app, operator: In, values: [hello-node, hello-node2, hello-node3] }
    serviceAdvertisements:
      - LoadBalancerIP # <-- default
      #- ClusterIP      # <-- options
      #- ExternalIP     # <-- options
    neighbors:
    - peerASN: 65000
      peerAddress: 10.0.2.1/32
      gracefulRestart:
        enabled: true
        restartTimeSeconds: 60
      advertisedPathAttributes:
      - selectorType: CiliumLoadBalancerIPPool
        selector:
          matchLabels:
            environment: vrf-mgmt
        communities:
          standard:
          - 65001:100
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "vrf-mgmt"
  labels:
    environment: vrf-mgmt
spec:
  blocks:
  - cidr: "100.64.8.0/22"
---

There were no route policies observed on the Cilium nodes:

$  k exec cilium-vj8nj -n kube-system -- cilium-dbg bgp route-policies
VRouter   Policy Name   Type   Match Peers   Match Prefixes (Min..Max Len)   RIB Action   Path Actions

$  k exec cilium-lj9b5 -n kube-system -- cilium-dbg bgp route-policies
VRouter   Policy Name   Type   Match Peers   Match Prefixes (Min..Max Len)   RIB Action   Path Actions

Other related information was:

$  k get ippools --show-labels
NAME                DISABLED   CONFLICTING   IPS AVAILABLE   AGE   LABELS
vrf-core-services   false      False         254             64m   <none>
vrf-customers       false      False         254             64m   <none>
vrf-mgmt            false      False         1020            64m   environment=vrf-mgmt

$  k get services --show-labels
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE   LABELS
hello-node     LoadBalancer   10.2.157.150   100.64.8.2    8080:32293/TCP   67m   app=hello-node
hello-node-2   LoadBalancer   10.2.88.35     100.64.8.1    8080:32414/TCP   64m   app=hello-node-2
kubernetes     ClusterIP      10.2.0.1       <none>        443/TCP          71m   component=apiserver,provider=kubernetes

In the Slack thread, Harsimran Pabla pointed out that the existing BGP reconciler references a deprecated CRD field (cidrs) for CiliumLoadBalancerIPPool. In https://github.com/cilium/cilium/blob/main/pkg/bgpv1/manager/reconciler/route_policy.go, he pointed out:

            for _, cidrBlock := range pool.Spec.Cidrs {

As a test, for the CiliumLoadBalancerIPPool I changed:

spec:
  blocks:
  - cidr: "100.64.8.0/22"

to:

spec:
  cidrs:
  - "100.64.8.0/22"

This corrected my issue! The configured BGP community, via the Advertised Paths Feature, was now present on the announcement:

$  cilium bgp routes
(Defaulting to `available ipv4 unicast` routes, please see help for more options)

Node                              VRouter   Prefix          NextHop    Age      Attrs
bgp-cplane-dev-v4-control-plane   65001     100.64.8.2/32   10.0.1.1   7s       [{Origin: i} {AsPath: 65000 65002} {Nexthop: 10.0.1.1} {Communities: 65001:100}]
bgp-cplane-dev-v4-worker          65002     100.64.8.2/32   0.0.0.0    59m12s   [{Origin: i} {Nexthop: 0.0.0.0}]

$  k exec cilium-vj8nj -n kube-system -- cilium-dbg bgp route-policies
VRouter   Policy Name                                   Type     Match Peers   Match Prefixes (Min..Max Len)   RIB Action   Path Actions
65002     10.0.2.1/32-CiliumLoadBalancerIPPool-52e...   export   10.0.2.1/32   100.64.8.0/22 (32..32)          none         AddCommunities: [65001:100]

$  k exec cilium-vj8nj -n kube-system -- cilium-dbg bgp route-policies
VRouter   Policy Name                                   Type     Match Peers   Match Prefixes (Min..Max Len)   RIB Action   Path Actions
65002     10.0.2.1/32-CiliumLoadBalancerIPPool-52e...   export   10.0.2.1/32   100.64.8.0/22 (32..32)          none         AddCommunities: [65001:100]

This bug request is being opened to capture the necessary changes to https://github.com/cilium/cilium/blob/main/pkg/bgpv1/manager/reconciler/route_policy.go to allow one to specify either the deprecated (cidrs) or current field (blocks).

Cilium Version

$  cilium version
cilium-cli: v0.16.7-40-g9316d0ac compiled with go1.22.2 on linux/amd64
cilium image (default): v1.15.5
cilium image (stable): v1.15.5
cilium image (running): 1.16.0-dev

Kernel Version

$  uname -a
Linux hostname 6.5.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

$  k version
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.15-enhanced-describe-dirty", GitCommit:"ac2e2baa7d4039cc4c68f2e869e4edbe2d60b305", GitTreeState:"dirty", BuildDate:"2023-03-02T00:33:46Z", GoVersion:"go1.20.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.0", GitCommit:"7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a", GitTreeState:"clean", BuildDate:"2024-05-13T22:00:36Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/amd64"}

Regression

No response

Sysdump

cilium-sysdump-20240523-133822.zip

Relevant log output

No response

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct