New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supported way to customize BGP export filters #1604

Open
danderson opened this Issue Jan 15, 2018 · 5 comments

Comments

Projects
None yet
3 participants
@danderson

danderson commented Jan 15, 2018

In google/metallb#114, I explored how to make my k8s BGP load-balancer interoperate gracefully with Calico clusters that peer with external BGP routers. I've documented my findings at https://master--metallb.netlify.com/configuration/calico/ and google/metallb#114 (comment)

Current Behavior

In my setup, I want to peer Calico with external BGP routers. I additionally want to peer Calico on each machine with another node agent running on localhost, so that I can push additional routes into Calico and have it redistribute them to the external routers.

Currently, this is not possible, because Calico's BIRD configurations filter out routes that aren't part of the subnets that have been allocated by Calico. So, even though I can peer with my node agent and push routes into Calico's BIRD instances, those routes do not get propagated to other peers.

See the documentation I wrote on integrating with Romana for some diagrams and more explanations of the topology I'm trying to build.

Expected Behavior

Calico should have a documented, supported way of customizing BGP export filters. projectcalico/calicoctl#1138 added some support for custom filters, and the issue was closed successfully. However, #292 states that this implementation is problematic, and as such it's not officially supported or documented.

Context

I am trying to make Calico and MetalLB integrate nicely with each other, by setting up a BGP topology like the one I documented for Romana integration. Basically, I want Calico to peer with the outside world, but also with another node agent that pushes routes into Calico for redistribution.

Setting up BGP sessions to/from localhost is notoriously tricky, but with the right set of options, it's possible. Inflexible route filters in Calico is one of the issues I encountered.

Your Environment

  • Calico version: 2.6.3
  • Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.9.1
  • Operating System and version: Debian testing
  • Link to your project (optional): https://github.com/google/metallb
@caseydavenport

This comment has been minimized.

Member

caseydavenport commented Jan 16, 2018

@danderson thanks for raising this and for the detailed write up.

As you discovered, such a feature was a part of older Calico releases, but it wasn't officially supported and required detailed knowledge of the etcd data model.

I'd very much like to extend Calico's data model to add this concept - I'm going to tentatively add this to the Calico v3.1.0 milestone.

As a strawman, it might make sense to have something like global/per-node BGPExportFilter object similar to how the BGPConfiguration object works today.

e.g.

kind: BGPExportFilter
metadata:
  name: default
spec:
  rules:
  - action: Accept
    net: 10.0.0.0/24

We'd need to work out the right set of filter criteria to expose in the first iteration.

CC @neiljerram @robbrockbank @liljenstolpe

@caseydavenport caseydavenport added this to the Calico v3.1.0 milestone Jan 16, 2018

@danderson

This comment has been minimized.

danderson commented Jan 18, 2018

The BGPExportFilter object you describe would do what I need. For MetalLB integration, the only filter criteria I need is the equivalent of Bird's ~, i.e. "is this_cidr contained inside other_cidr?"

@enitlas

This comment has been minimized.

enitlas commented Jan 18, 2018

Exposing filters through a Calico abstraction seems to fix this specific issue.

There's definitely a larger conversation to be had around exposing more of the BIRD functionality in general. Certain users are going to have use cases around more advanced routing functions on the node, but if Calico is running the routing daemon, it's monopolizing routing on the node. These users could have a need to be able to configure the node in ways that may not be reasonable to maintain a Calico abstraction for. One possible solution could be to allow users to pass config snippets directly into bird.cfg via a configmap-like mechanism.

@caseydavenport

This comment has been minimized.

Member

caseydavenport commented Jan 18, 2018

There's definitely a larger conversation to be had around exposing more of the BIRD functionality in general.

Yep, definitely happy to have that discussion!

I tend to think of BIRD as a Calico implementation detail, and so exposing BIRD config directly to the user breaks an important abstraction. For example, Calico already includes a community-supported GoBGP integration and baking BIRD constructs into the Calico API may hinder that effort.

@enitlas

This comment has been minimized.

enitlas commented Jan 18, 2018

Calico should not be responsible for validating or formatting advanced config snippets. It could pass it to bird.cfg or to GoBGP's config mechanism (I'm not familiar with it). I totally agree that building the Calico API for the full featureset of multiple routing daemons is not a reasonable pursuit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment