Do you use MetalLB? Tell us!

[danderson]

This is not an issue so much as a lightweight way of gathering information on who is using MetalLB. This is mostly to satisfy our curiosity, but might also help us decide how to evolve the project.

So, if you use MetalLB for something, please chime in here and tell us more!

[rsanders]

We're not using it yet, but our K8S clusters are 100% metallic, running in our data centers and on customer premises. We're going to be trying this out in the new year.

Our deployment would in most cases have this peer directly with our routers. We're a mostly Cisco shop. Which BGP implementations have you tested with?

[danderson]

So far, I've only tested with OSS BGP implementations, mainly BIRD. I used to have some Juniper and Cisco hardware that could speak BGP, but I got rid of them :(.

With that said, currently MetalLB speaks a very conventional, "has existed for ever" dialect of BGP, so it should work fine. I foresee two possible issues interacting with Cisco's implementation:

• Some old Cisco devices use an unusual encoding for capability advertisements in the OPEN message. AFAICT, they stopped doing this in the late 1990s, so it shouldn't be a problem. If it is a problem, it's a trivial patch to fix it.
• Cisco may refuse to speak to a BGPv4 speaker that uses the original route encoding for IPv4 (which the RFCs say is still 100% correct), and instead requires all route advertisements to use the MP-BGP encoding. I'm in the middle of making MetalLB speak MP-BGP for dual-stack support anyway, so hopefully by the time you test MetalLB this should not be a problem.

If you hit any interop problems, please file bugs! Ideally with a pcap that captures the initial exchange of BGP OPEN messages and a few UPDATE messages, so that I can add regression tests.

[xnaveira]

A colleague brought my attention to this project recently. We're running bare metal k8s on bgp enabled hosts and pods so this looks very interesting. Unfortunately we are using 32-bit AS numbers in our setup so we can't test it right away but it looks promising!

[danderson]

@xnaveira MetalLB supports 32-bit AS numbers! It's the one BGP extension that's implemented in v0.1.0. v0.2.0 will also support MP-BGP, so if you use Quagga or Cisco devices, you'll want to wait for 0.2.0.

[xnaveira]

That is awesome @danderson , i'll give it a try asap then.

[tangjoe]

I saw a post in reddit.com about MetalLB and this is what I had been looking for a long time for my local K8S on Mac. Took about one day following the tutorial and experiment, finally I have it up and running. Now I can deploy service in k8s with LoadBalancer without "pending". Awesome.

[halfa]

We have successfully deployed a patched version of MetalLB on a staging K8S cluster at @naitways, peered with JunOS hardware appliances. Awesome! :)

[danderson]

\o/ What patching did you need to interface with JunOS? I'm going to spin up a virtual JunOS later today and do interop testing, but it sounds like you already found the issues?

[jacobsmith928]

@danderson happy to support some bare metal and diverse switching (Juniper) + local and global BGP testing at Packet.net. Maybe take advantage of https://github.com/cncf/cluster ?

[zcourts]

We've just deployed MetalLB to our new test env. on a 1.9 K8s and eagerly watching #7 and #83. We've got a use case where any and all pods can be publicly routable so we're looking at having an IPv6 only cluster and each pod/svc getting IPv6.

We need non-HTTP svc so planning to use IPs to identify pods when SNI/TLS isn't available (of which we're launching with thousands this summer and expect that to grow to 10s of in 12-18 months.) Aligning some things with K8s 1.10 release for beta IPv6 support and will probably be running alpha versions of stuff in test leading up to launch.
FWIW we use OVH and each server comes with a /64 IPv6 block so when this is in, being able to draw on that pool from each K8s compute node will be ideal. As it stands we have no Go lang expertise but if we can contrib in any other way do let us know. We're comfy with C++/Java/Scala and I'll probably be learning Go this yr since we're committed to K8s.

[hameno]

I just tried this in ARP mode in my k8s-cluster @ home. Works so far, thanks for this great project 👍
I may also deploy this at work in the future for an internal development cluster.