Supported way to customize BGP export filters

[danderson]

In metallb/metallb#114, I explored how to make my k8s BGP load-balancer interoperate gracefully with Calico clusters that peer with external BGP routers. I've documented my findings at https://master--metallb.netlify.com/configuration/calico/ and metallb/metallb#114 (comment)

Current Behavior

In my setup, I want to peer Calico with external BGP routers. I additionally want to peer Calico on each machine with another node agent running on localhost, so that I can push additional routes into Calico and have it redistribute them to the external routers.

Currently, this is not possible, because Calico's BIRD configurations filter out routes that aren't part of the subnets that have been allocated by Calico. So, even though I can peer with my node agent and push routes into Calico's BIRD instances, those routes do not get propagated to other peers.

See the documentation I wrote on integrating with Romana for some diagrams and more explanations of the topology I'm trying to build.

Expected Behavior

Calico should have a documented, supported way of customizing BGP export filters. projectcalico/calicoctl#1138 added some support for custom filters, and the issue was closed successfully. However, #292 states that this implementation is problematic, and as such it's not officially supported or documented.

Context

I am trying to make Calico and MetalLB integrate nicely with each other, by setting up a BGP topology like the one I documented for Romana integration. Basically, I want Calico to peer with the outside world, but also with another node agent that pushes routes into Calico for redistribution.

Setting up BGP sessions to/from localhost is notoriously tricky, but with the right set of options, it's possible. Inflexible route filters in Calico is one of the issues I encountered.

Your Environment

• Calico version: 2.6.3
• Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.9.1
• Operating System and version: Debian testing
• Link to your project (optional): https://github.com/google/metallb

[caseydavenport]

@danderson thanks for raising this and for the detailed write up.

As you discovered, such a feature was a part of older Calico releases, but it wasn't officially supported and required detailed knowledge of the etcd data model.

I'd very much like to extend Calico's data model to add this concept - I'm going to tentatively add this to the Calico v3.1.0 milestone.

As a strawman, it might make sense to have something like global/per-node BGPExportFilter object similar to how the BGPConfiguration object works today.

e.g.

kind: BGPExportFilter
metadata:
  name: default
spec:
  rules:
  - action: Accept
    net: 10.0.0.0/24

We'd need to work out the right set of filter criteria to expose in the first iteration.

CC @neiljerram @robbrockbank @liljenstolpe

[danderson]

The BGPExportFilter object you describe would do what I need. For MetalLB integration, the only filter criteria I need is the equivalent of Bird's ~, i.e. "is this_cidr contained inside other_cidr?"

[enitlas]

Exposing filters through a Calico abstraction seems to fix this specific issue.

There's definitely a larger conversation to be had around exposing more of the BIRD functionality in general. Certain users are going to have use cases around more advanced routing functions on the node, but if Calico is running the routing daemon, it's monopolizing routing on the node. These users could have a need to be able to configure the node in ways that may not be reasonable to maintain a Calico abstraction for. One possible solution could be to allow users to pass config snippets directly into bird.cfg via a configmap-like mechanism.

[caseydavenport]

There's definitely a larger conversation to be had around exposing more of the BIRD functionality in general.

Yep, definitely happy to have that discussion!

I tend to think of BIRD as a Calico implementation detail, and so exposing BIRD config directly to the user breaks an important abstraction. For example, Calico already includes a community-supported GoBGP integration and baking BIRD constructs into the Calico API may hinder that effort.

[enitlas]

Calico should not be responsible for validating or formatting advanced config snippets. It could pass it to bird.cfg or to GoBGP's config mechanism (I'm not familiar with it). I totally agree that building the Calico API for the full featureset of multiple routing daemons is not a reasonable pursuit.

[caseydavenport]

The BGPExportFilter object you describe would do what I need. For MetalLB integration, the only filter criteria I need is the equivalent of Bird's ~, i.e. "is this_cidr contained inside other_cidr?"

For this, I think it should be just a matter of defining an IPPool object and setting spec.Disabled to true.

Calico will allow export of routes within that IP pool, and disabling it means it won't be used to assign pod IP addresses.