Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do you use MetalLB? Tell us! #5

Open
danderson opened this issue Nov 28, 2017 · 213 comments
Open

Do you use MetalLB? Tell us! #5

danderson opened this issue Nov 28, 2017 · 213 comments
Labels

Comments

@danderson
Copy link
Contributor

danderson commented Nov 28, 2017

This is not an issue so much as a lightweight way of gathering information on who is using MetalLB. This is mostly to satisfy our curiosity, but might also help us decide how to evolve the project.

So, if you use MetalLB for something, please chime in here and tell us more!

@rsanders
Copy link

rsanders commented Dec 1, 2017

We're not using it yet, but our K8S clusters are 100% metallic, running in our data centers and on customer premises. We're going to be trying this out in the new year.

Our deployment would in most cases have this peer directly with our routers. We're a mostly Cisco shop. Which BGP implementations have you tested with?

@danderson
Copy link
Contributor Author

danderson commented Dec 1, 2017

So far, I've only tested with OSS BGP implementations, mainly BIRD. I used to have some Juniper and Cisco hardware that could speak BGP, but I got rid of them :(.

With that said, currently MetalLB speaks a very conventional, "has existed for ever" dialect of BGP, so it should work fine. I foresee two possible issues interacting with Cisco's implementation:

  • Some old Cisco devices use an unusual encoding for capability advertisements in the OPEN message. AFAICT, they stopped doing this in the late 1990s, so it shouldn't be a problem. If it is a problem, it's a trivial patch to fix it.
  • Cisco may refuse to speak to a BGPv4 speaker that uses the original route encoding for IPv4 (which the RFCs say is still 100% correct), and instead requires all route advertisements to use the MP-BGP encoding. I'm in the middle of making MetalLB speak MP-BGP for dual-stack support anyway, so hopefully by the time you test MetalLB this should not be a problem.

If you hit any interop problems, please file bugs! Ideally with a pcap that captures the initial exchange of BGP OPEN messages and a few UPDATE messages, so that I can add regression tests.

@xnaveira
Copy link
Contributor

xnaveira commented Dec 4, 2017

A colleague brought my attention to this project recently. We're running bare metal k8s on bgp enabled hosts and pods so this looks very interesting. Unfortunately we are using 32-bit AS numbers in our setup so we can't test it right away but it looks promising!

@danderson
Copy link
Contributor Author

danderson commented Dec 5, 2017

@xnaveira MetalLB supports 32-bit AS numbers! It's the one BGP extension that's implemented in v0.1.0. v0.2.0 will also support MP-BGP, so if you use Quagga or Cisco devices, you'll want to wait for 0.2.0.

@xnaveira
Copy link
Contributor

xnaveira commented Dec 5, 2017

That is awesome @danderson , i'll give it a try asap then.

@tangjoe
Copy link

tangjoe commented Dec 8, 2017

I saw a post in reddit.com about MetalLB and this is what I had been looking for a long time for my local K8S on Mac. Took about one day following the tutorial and experiment, finally I have it up and running. Now I can deploy service in k8s with LoadBalancer without "pending". Awesome.

@halfa
Copy link

halfa commented Dec 8, 2017

We have successfully deployed a patched version of MetalLB on a staging K8S cluster at @naitways, peered with JunOS hardware appliances. Awesome! :)

@danderson
Copy link
Contributor Author

danderson commented Dec 8, 2017

\o/ What patching did you need to interface with JunOS? I'm going to spin up a virtual JunOS later today and do interop testing, but it sounds like you already found the issues?

@jacobsmith928
Copy link

jacobsmith928 commented Jan 8, 2018

@danderson happy to support some bare metal and diverse switching (Juniper) + local and global BGP testing at Packet.net. Maybe take advantage of https://github.com/cncf/cluster ?

@zcourts
Copy link

zcourts commented Jan 14, 2018

We've just deployed MetalLB to our new test env. on a 1.9 K8s and eagerly watching #7 and #83. We've got a use case where any and all pods can be publicly routable so we're looking at having an IPv6 only cluster and each pod/svc getting IPv6.

We need non-HTTP svc so planning to use IPs to identify pods when SNI/TLS isn't available (of which we're launching with thousands this summer and expect that to grow to 10s of in 12-18 months.) Aligning some things with K8s 1.10 release for beta IPv6 support and will probably be running alpha versions of stuff in test leading up to launch.
FWIW we use OVH and each server comes with a /64 IPv6 block so when this is in, being able to draw on that pool from each K8s compute node will be ideal. As it stands we have no Go lang expertise but if we can contrib in any other way do let us know. We're comfy with C++/Java/Scala and I'll probably be learning Go this yr since we're committed to K8s.

@hameno
Copy link

hameno commented Jan 31, 2018

I just tried this in ARP mode in my k8s-cluster @ home. Works so far, thanks for this great project 👍
I may also deploy this at work in the future for an internal development cluster.

@aphistic
Copy link

aphistic commented Feb 18, 2018

Just jumping in as well. I'm using ARP mode in my home lab with a cluster I set up following Kubernetes the Hard Way to learn Kubernetes. I'm using Weave Net to handle my cluster networking and running on XenServer VMs. I haven't gotten metallb running correctly yet but I'm working on it. :)

@pfcurtis
Copy link

pfcurtis commented Feb 21, 2018

We are using MetalLB on a 30 node k8x cluster (internally) in production. So far, no issues.

@pehlert
Copy link

pehlert commented Feb 22, 2018

Just wanted to say thank you for this project! I have spent hours trying to figure out several issues with keepalived-vip before stumbling across MetallLB.. Installed it in 5mins and it just works (and is a more elegant approach, too). Time will tell how stable it is, but no issues whatsoever so far!

@ewoutp
Copy link

ewoutp commented Mar 11, 2018

Running it on both a 3 OrangePI PC2 cluster and a 3 VM CoreOS cluster with k8s 1.9.
Works like a charm!
Love the ease of installation.

@ChristofferNicklassonLenswayGroup

Hi, we will use this when we will launch our ongoing k8s project.
and i am also using this on my cluster at home.
just love it :)

@ebauman
Copy link

ebauman commented Mar 27, 2018

This project is lovely.

I'm using it on my home cluster, and plan to use it on my cluster at work.
We have a old-school deployment at my employer which doesn't afford me the flexibility to setup such niceties as BGP (or really any routing protocol that isn't EIGRP). Most everything that we do is layer 2 separated, so I felt left in the dust by people who got to just specify type: LoadBalancer and off they went.

This project makes k8s fit into my org.

@joestafford
Copy link

joestafford commented Apr 2, 2018

<3
Using this LB implementation in my home lab to learn k8s and look forward to using it in a project at work!

@szabopaul
Copy link

szabopaul commented Apr 8, 2018

Using this project on my homelab to facilitate Ansible configuration of single container pods is a breeze. Amazing work!

@mpackard
Copy link

mpackard commented Apr 13, 2018

We are using this to try out Kubernetes on our own hardware. Thanks for creating it.

@fsdaniel
Copy link

fsdaniel commented Apr 18, 2018

Running it against SRX and MX juniper hardware.

Thanks for making it!

@uablrek
Copy link
Contributor

uablrek commented Apr 19, 2018

I am evaluating metallb for use in "bare-VM" rather than bare-metal. There is however no difference, the problems are the same.
For testing I "install" metallb by starting the 2 binaries directly on the node vms, i.e not in pods. Due to the simple and elegant architecture this works perfectly fine.

I have learned a lot about Kubernetes by studying metallb. Thanks!

@schmitch
Copy link

schmitch commented Apr 25, 2018

my company started trying out metallb.
we first tried to use it via layer 2 (we use calicoctl and had configured a bgp peer (that was not in use))
however only one client could connect to the service, we had no idea, why. maybe ARP packets were filtered somewhere.

I then removed the bgp peering from calico and used it with metallb which finally worked.
it's really cool to have a metallb.

however it's sad that metallb does not have some kind of LVS way of attaching IPs in layer2 mode, which won't use ARP requests, so it would be useful and less error prone in most networks.

@michaelfig
Copy link

michaelfig commented May 1, 2018

I'm getting my feet wet in Kubernetes at my company. Metallb has proven really useful for repeatable configuration of Ingresses (using nginx-ingress) on available layer2 IPs. Thanks for this useful software!

@FireDrunk
Copy link
Contributor

FireDrunk commented May 1, 2018

I'm running it at home, in my testing kubernetes cluster (which is running weave). It's connected to my pfSense router via BGP. This setup would be perfect in a datacenter setup with a specific IP space for DMZ.
If anyone wants the configs, I'd be happy to share.

Thanks for an awesome piece of software!

@nrbrt
Copy link

nrbrt commented May 7, 2018

I am running it in my home-lab and switched to Metallb after using Keepalived-vip for a while and not finding it stable enough. Keepalived-vip would work fine for a while and then lock up, forcing me to manually delete the "master" pod, after which things would start working again. I hope my worries are over now, using Metallb.

@champtar
Copy link
Contributor

champtar commented Aug 20, 2021

@MehrCurry please do not hijack this issue and either go on Slack or create a new issue

@fpiesche
Copy link

fpiesche commented Sep 9, 2021

I'm using MetalLB together with an nginx ingress controller for my personal mixed-architecture SBC cluster, consisting of:

  • 1x Raspberry Pi 2 (armv7)
  • 2x Raspberry Pi 3 (arm64)
  • 2x Raspberry Pi 4 (arm64)
  • 1x Macbook Air 2015 (amd64)
  • 1x Rock Pi X (amd64)

All working perfectly lovely! :)

@scepeda78
Copy link

scepeda78 commented Sep 16, 2021

Microk8s plus knative, the last needs a load balancer in bare metal for solving names.

@emoritzx
Copy link

emoritzx commented Nov 14, 2021

I use microk8s + metallb in my homelab cluster comprised of old desktop PCs that have been re-purposed into Linux servers.

Also, I just learned about allow-shared-ip and it has made my life so much easier to manage my services :)

@flakeman2
Copy link

flakeman2 commented Dec 18, 2021

I'm using k3s + metalLB (layer2) in a homelab of Raspberry Pi's. Thanks for the awesome work on this project!

@MysticalMount
Copy link

MysticalMount commented Dec 21, 2021

I use in a homelab 3 x node Proxmox with K3OS VM's running MetalLB, tried ingress, this time round on my new cluster doing away with it in favour of MetalLB only!

@akharlamov
Copy link

akharlamov commented Dec 31, 2021

I use MetalLB on our clusters deployed in Hetzner DCs.

@fire833
Copy link

fire833 commented Jan 14, 2022

I have MetalLB (L2) deployed to my home K8s cluster, it works great to expose a LoadBalancer for my nginx-ingress controller along with a few other TCP services. Thanks for your great work!

@bjornbouetsmith
Copy link

bjornbouetsmith commented Feb 19, 2022

I have MetalLB deployed - thank god I found somthing like this, since it makes it so much more "HA" than having to have the load balancer outside of my cluster and pointing it to specific node ip's.

@vosdev
Copy link

vosdev commented Feb 25, 2022

Been wanting to try out MetalLB since I started using K8s but with the lack of IPv6 support I never could. Am forced to using NodePort for all my services with an external LB..

Hope that soon I'll be able to say I am using MetalLB for my clusters! Watching releases for IPv6 support so I can try it out ASAP.

@fedepaol
Copy link
Member

fedepaol commented Feb 25, 2022

@vosdev we have ipv6 support both for L2 mode, and if you switch to FRR mode also for BGP in the latest release.

@wiljay
Copy link

wiljay commented Mar 25, 2022

I use metallb in my environment, it is easy and funny, but encounter some issue , when i delete svc ,IP still remain in the ipvsadm, which mean i will have to manually delete the ip forward.

@fauh45
Copy link

fauh45 commented Mar 25, 2022

Hi! I'm part of infrastructure team in my university server for a class. We need to make kubernetes in bare metal, and metallb surely is the easiest way to do load balancing with our cluster! We got no issues, with just little change in the default configuration it works out of the box!

@abessifi
Copy link

abessifi commented Apr 29, 2022

Nice project !
Running it with OpenShift.

@magnezone150
Copy link

magnezone150 commented Apr 29, 2022

Using Metallb on a Canary and Production K8 Clusters.
Wondering where I can learn more about how it elects which node is the primary because when preforming cluster upgrades Metallb goes down and causes service downtimes

@fedepaol
Copy link
Member

fedepaol commented Apr 29, 2022

@magnezone150 I guess you are talking about L2. I can add an issue to document how the election works.

@magnezone150
Copy link

magnezone150 commented Apr 29, 2022

@magnezone150 I guess you are talking about L2. I can add an issue to document how the election works.

Yes Exactly, Thank you.
Appreciate it, I'll describe my issue there.

@smanikkara
Copy link

smanikkara commented May 2, 2022

Using metallbon Openshift 4.9 4.10 clusters (on VMware)
In version 4.6 load balancer is not getting ip from pool (pending state)

@mateuszdrab
Copy link

mateuszdrab commented May 30, 2022

I'm using k3s + metalLB (layer2) in a homelab of Raspberry Pi's. Thanks for the awesome work on this project!

Same here, metallb on K3s on a mix of VMware VMs and Raspberry Pis. Using BGP mode due to need to support multiple subnets.

@thetredev
Copy link

thetredev commented Jun 20, 2022

Using MetalLB on a k0s cluster running on Arch Linux VMs. All smooth :)

@maxgio92
Copy link

maxgio92 commented Jul 5, 2022

I'm using MetalLB L2-mode on my home lab of 3-node Rpi k3s cluster.
It works like a charm, for some time now :-)

@Walternrg4u
Copy link

Walternrg4u commented Jul 6, 2022

@JunRamoneda
Copy link

JunRamoneda commented Jul 10, 2022

Using it on my k3s cluster deployed to raspberry pi 4s. Very satisfied with metallb

@Pseudow
Copy link

Pseudow commented Jul 19, 2022

Im using metallb on a debian-node cluster!

@BloodyIron
Copy link

BloodyIron commented Aug 2, 2022

Using MetalLB in Layer 2 mode on RKE1 k8s cluster, managed by Rancher (2.6.x+). It makes it so I don't have to reconfigure the LAN but still get rapid and automatic fail-over for a single inbound LAN IP. It's great! (once I wrapped my head around it)

@alowe7
Copy link

alowe7 commented Aug 19, 2022

I'm using MetalLB in Layer 2 mode on rhel8 k8s bare metal dev cluster. works great!

@k4nzdroid
Copy link

k4nzdroid commented Sep 11, 2022

Nice Project!!!
Using it on our selfhosted Kubernetes Cluster.
BGP mode for Development Env and Production Env. ARP mode for less common scenarios.

@LeadingMoominExpert
Copy link

LeadingMoominExpert commented Nov 30, 2022

I've been developing a bare metal k8s cluster at our office mostly for both educational purposes as well as for running some small internal apps. Metal LB in Layer 2 mode was a great solution for me to be able to deploy an ingress controller so I can use ingresses instead of having to use nodeports. Source repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests