Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supported way to customize BGP export filters #1604

Open
danderson opened this issue Jan 15, 2018 · 7 comments
Open

Supported way to customize BGP export filters #1604

danderson opened this issue Jan 15, 2018 · 7 comments

Comments

@danderson
Copy link

@danderson danderson commented Jan 15, 2018

In metallb/metallb#114, I explored how to make my k8s BGP load-balancer interoperate gracefully with Calico clusters that peer with external BGP routers. I've documented my findings at https://master--metallb.netlify.com/configuration/calico/ and metallb/metallb#114 (comment)

Current Behavior

In my setup, I want to peer Calico with external BGP routers. I additionally want to peer Calico on each machine with another node agent running on localhost, so that I can push additional routes into Calico and have it redistribute them to the external routers.

Currently, this is not possible, because Calico's BIRD configurations filter out routes that aren't part of the subnets that have been allocated by Calico. So, even though I can peer with my node agent and push routes into Calico's BIRD instances, those routes do not get propagated to other peers.

See the documentation I wrote on integrating with Romana for some diagrams and more explanations of the topology I'm trying to build.

Expected Behavior

Calico should have a documented, supported way of customizing BGP export filters. projectcalico/calicoctl#1138 added some support for custom filters, and the issue was closed successfully. However, #292 states that this implementation is problematic, and as such it's not officially supported or documented.

Context

I am trying to make Calico and MetalLB integrate nicely with each other, by setting up a BGP topology like the one I documented for Romana integration. Basically, I want Calico to peer with the outside world, but also with another node agent that pushes routes into Calico for redistribution.

Setting up BGP sessions to/from localhost is notoriously tricky, but with the right set of options, it's possible. Inflexible route filters in Calico is one of the issues I encountered.

Your Environment

  • Calico version: 2.6.3
  • Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.9.1
  • Operating System and version: Debian testing
  • Link to your project (optional): https://github.com/google/metallb
@caseydavenport
Copy link
Member

@caseydavenport caseydavenport commented Jan 16, 2018

@danderson thanks for raising this and for the detailed write up.

As you discovered, such a feature was a part of older Calico releases, but it wasn't officially supported and required detailed knowledge of the etcd data model.

I'd very much like to extend Calico's data model to add this concept - I'm going to tentatively add this to the Calico v3.1.0 milestone.

As a strawman, it might make sense to have something like global/per-node BGPExportFilter object similar to how the BGPConfiguration object works today.

e.g.

kind: BGPExportFilter
metadata:
  name: default
spec:
  rules:
  - action: Accept
    net: 10.0.0.0/24

We'd need to work out the right set of filter criteria to expose in the first iteration.

CC @neiljerram @robbrockbank @liljenstolpe

@danderson
Copy link
Author

@danderson danderson commented Jan 18, 2018

The BGPExportFilter object you describe would do what I need. For MetalLB integration, the only filter criteria I need is the equivalent of Bird's ~, i.e. "is this_cidr contained inside other_cidr?"

@enitlas
Copy link

@enitlas enitlas commented Jan 18, 2018

Exposing filters through a Calico abstraction seems to fix this specific issue.

There's definitely a larger conversation to be had around exposing more of the BIRD functionality in general. Certain users are going to have use cases around more advanced routing functions on the node, but if Calico is running the routing daemon, it's monopolizing routing on the node. These users could have a need to be able to configure the node in ways that may not be reasonable to maintain a Calico abstraction for. One possible solution could be to allow users to pass config snippets directly into bird.cfg via a configmap-like mechanism.

@caseydavenport
Copy link
Member

@caseydavenport caseydavenport commented Jan 18, 2018

There's definitely a larger conversation to be had around exposing more of the BIRD functionality in general.

Yep, definitely happy to have that discussion!

I tend to think of BIRD as a Calico implementation detail, and so exposing BIRD config directly to the user breaks an important abstraction. For example, Calico already includes a community-supported GoBGP integration and baking BIRD constructs into the Calico API may hinder that effort.

@enitlas
Copy link

@enitlas enitlas commented Jan 18, 2018

Calico should not be responsible for validating or formatting advanced config snippets. It could pass it to bird.cfg or to GoBGP's config mechanism (I'm not familiar with it). I totally agree that building the Calico API for the full featureset of multiple routing daemons is not a reasonable pursuit.

@caseydavenport
Copy link
Member

@caseydavenport caseydavenport commented Mar 1, 2019

The BGPExportFilter object you describe would do what I need. For MetalLB integration, the only filter criteria I need is the equivalent of Bird's ~, i.e. "is this_cidr contained inside other_cidr?"

For this, I think it should be just a matter of defining an IPPool object and setting spec.Disabled to true.

Calico will allow export of routes within that IP pool, and disabling it means it won't be used to assign pod IP addresses.

@smekkley
Copy link

@smekkley smekkley commented Jul 12, 2019

I rambled a little in projectcalico/confd#172 about keeping the source ip while exporting cluster ip address range.

But I think this BGP export filters feature would give me the most reliable way to keep the source IP address while you expose the service on bare-metal installation using externalTrafficPolicy: Local.

With openstack, calico can create a floating ip but with bare-metal installation, metallb is the only option to have HA lb. It would make a lot of people happy if calico just supported type: Loadbalancer and assign VIP, but I don't think it's a scope of this project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants