Add MetalLB 0.13 support

[enkelprifti98]

It looks like the latest MetalLB releases aren’t working with the CCM anymore. I suspect the reason is there’s updated syntax with the newer MetalLB 0.13 releases and the CCM isn’t updated to support it.

With CCM 3.5.0 and MetalLB v0.13.4 I get this error on the MetalLB speaker pods:

{"caller":"main.go:282","error":"assigned IP not allowed by config","ips":["86.109.11.206"],"level":"error","msg":"IP allocated by controller not allowed by config","op":"setBalancer","ts":"2022-08-30T17:12:23Z"}

When I revert MetalLB to 0.12.1, it all works fine.

To replicate the issue:

1. Deploy Kubernetes cluster.
2. Install CCM 3.5.0.
3. Install MetalLB v0.13.4 as shown here. I'm using the native yaml deployment.

[displague]

@deitch mentioned elsewhere that we would benefit from a version compatibility matrix

• document kube-vip version compatibility matrix #226
• https://github.com/equinix/cloud-provider-equinix-metal#metallb
• https://github.com/equinix/cloud-provider-equinix-metal#version

@enkelprifti98 was 0.13 working with CCM 3.4? In #321 the docs were updated to call out that 0.13 was not known to work.

We are in the process of adding support for version 0.13.x.

Perhaps we can reword this issue to "Add MetalLB 0.13 support"? If the instructions ever advise install of an untagged or latest metallb version, then we should definitely correct that and that type of problem would align with the current "Latest MetalLB no longer working with CPEM" issue title.

[displague]

@enkelprifti98 are there features of MetalLB 0.13 that are uniquely interesting -- I'm wondering if we'll want to explore these features in the documentation and examples.

[enkelprifti98]

MetalLB 0.13 doesn't work with any CCM version AFAIK so i guess "Add MetalLB 0.13 support" is a more fitting title.

It seems like the 0.13 release is mostly just moving to a CRD based config model instead of using configmaps so I don't think there's anything there that would be interesting for Metal users in particular.

[cprivitere]

New 0.13+ Metal LB Features

• CRD support! A long awaited feature, MetalLB is now configurable via CRs. On top of that, validating webhooks will ensure the validity of the configuration upfront, without needing to check the logs. (PR #1237, PR #1245) Please note that the ConfigMap configuration is not supported anymore. Check the “Changes in behaviour” section for more details.
• It’s now possible to choose to advertise addresses in L2 mode, BGP mode, both or just allocate the IP without advertising it.
• Announcement node selector. It’s possible to choose which nodes to advertise from the IPs coming from a given pool (PR #1302)
• BGPPeer selector. For any IP allocated from a given IPAddressPool, it is possible to choose the subset of BGPPeers we want to advertise that IP to (PR #1171).
• Kustomize configuration overlays. We now provide various overlays that implement different configuration degrees (as opposed to having) one single manifest. (PR #1254)
• It’s now possible to store BGP passwords as secrets (as an alternative to plain text passwords). (PR #1264).
• LoadBalancerClass support: it’s possible to have MetalLB listen only to services with the provided load balancer class to comply with kubernetes loadbalancer class. (PR 1417).
• Helm Charts: optional annotations for PodMonitors and PrometheusRules (PR 1407)
• Multiprotocol BGP support: it’s possible to expose ipv4 addresses via a router connected via ipv6 and viceversa. It was already possible with FRR mode in the v0.12.x version, but now the feature is covered by tests too (PR 1444).
• Use cosign to sign the images (PR 1437)
• Added namespace validation for custom resources (PR 1523)
• Helm: added updateStrategy for controller and speakers (PR 1340)
• Expose the prometheus metrics securely via kube-rbac-proxy (PR 1545)
• Don’t deploy pod security policy, not supported in k8s 1.25+ (PR 1569)
• Layer2: Announce LB IPs from specific interfaces (PR 1536)
• Validate MetalLB supports mixed protocol services (Issue 1050 PR 1580)
• ConfigMapToCRs tool: align docs to match how to use it within the cluster (PR 1595)
• End to end tests: allow using external containers to execute the tests against a real cluster (PR 1604)
• Helm: add an option to set resources for speaker sidecar containers (PR 1622)
• Helm: add an option to set the validating webhooks failure policy (PR 1623)
• Removed the “experimental” wording from FRR mode declaring it being stable but less battle tested (PR 1636)

0.13+ breaking changes

• the biggest change is the introduction of CRDs and removing support for the configuration via ConfigMap. In order to ease the transition to the new configuration, we provide a conversion tool from ConfigMap to resources (see the “Backward compatibility” section from the main page).
• the internal architecture was radically changed in order to accomodate CRDs, so please do not hesitate to file an issue.
• The AvoidBuggyIPs flag was removed in order to reduce the api surface a bit. The same result can be achieved using ranges of IPs instead of the CIDR annotation.
• The metallb images from dockerhub are deprecated. From this release, only the images on quay.io will be supported and updated. The official images can be found under the quay.io metallb organization.

Obviously, the first one is the biggest impact to us as it breaks all the old configuration methods.