Dynamic BGP communities from services

[ahmet2mir]

Hello,

We are looking for a way to dynamically assign communities using labels and not "hardcoded" communities from config.

In our network design (vm/physical):

• each tenant (namespace) had their own BGP communities (a "publisher" community to identify route source and one subscriber community so other tenant could use this community to make the route available)
• each tenant are isolated thanks to the VRF we don't need to make anything (no iptables) to block requests from other tenants, paquets won't route (and it's faster than anything)
• cross tenant communications is done via ExaBGP (L3 LB) (exabgp will announce the route to "subscribers" tenants that are allowed to retrieve the service)
  • a tenant could use exabgp directly making a L3 LB
  • a tenant could use a custom haproxy setup (with indirect exabgp) making a L7 LB

To make an analogy with Kube:

• pod : vm
• service: exabgp
• ingress: haproxy

Thanks to MetalLB we are able to make this L3 service but we can't add communities on service definitions, it will be very useful if we could set them with labels like:

Define MetalLB config:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
        ...
    bgp-communities:
      publisher-tenant-a: "1:10"
      subscriber-tenant-a: "1001:10"
      publisher-tenant-b: "1:20"
      subscriber-tenant-b: "1001:20"
      publisher-tenant-c: "1:30"
      subscriber-tenant-c: "1001:30"
      publisher-tenant-d: "1:40"
      subscriber-tenant-d: "1001:40"
    address-pools:
      - name: tenant-a
        protocol: bgp
        addresses:
          - "10.1.0.0/23"
        bgp-advertisements:
          - communities:
              - publisher-tenant-a
              - subscriber-tenant-a

On service side override communities with metallb.universe.tf/address-pool-bgp-advertisements

apiVersion: v1
kind: Service
metadata:
  name: hello-service
  annotations:
    metallb.universe.tf/address-pool: tenant-a
    # tenant d is excluded
    metallb.universe.tf/address-pool-bgp-advertisements: publisher-tenant-a subscriber-tenant-a subscriber-tenant-b subscriber-tenant-c
    # or in string array
    metallb.universe.tf/address-pool-bgp-advertisements: '["publisher-tenant-a", "subscriber-tenant-a", "subscriber-tenant-b", "subscriber-tenant-c"]'
spec:
  selector:
    app: hello
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8080
  type: LoadBalancer

If not defined, use default communities from config.

[fedepaol]

Hi @ahmet2mir , there is a currently opened proposal to bind specific pools to specific services (#942).
Since the communities are bound to a given address pool, I have the feeling that this could solve your issue
(bind a service to a specific pool with specific communities). Does it make sense?

[ahmet2mir]

Hi @fedepaol

Lot of information in this proposal :)

From what I understand, if you specify the community on "address-pools" side, user can't change it "per service".

Like PR #593 for peers, I like the idea of "from annotation/labels" but for address-pools too.

[fedepaol]

Hi @fedepaol

Lot of information in this proposal :)

From what I understand, if you specify the community on "address-pools" side, user can't change it "per service".

Like PR #593 for peers, I like the idea of "from annotation/labels" but for address-pools too.

Indeed, it won't be possible directly to change communities in a service definition, but by binding a specific address pool to a given service, you would be able to associate the communities to the service indirectly.

[fedepaol]

Closing this as the whole configuration rework happened. It is now possible to attach an ipaddresspool to a tenant, and a bgpadvertisement with a given set of communities to that ipaddresspool only, covering the requested scenario