feat: add support for Equinix Metal Load Balancer #470

ctreatma · 2023-10-24T19:12:25Z

This updates CPEM to support the beta Load Balancer service provided by Equinix Metal. CPEM users can now configure CPEM to create external, managed Layer 4 load balancers for service load balancing and control plane load balancing.

This PR also adds a local development workflow using Tilt.

k8s-ci-robot · 2023-10-24T19:12:32Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ctreatma
Once this PR has been reviewed and has the lgtm label, please assign cheftako for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

internal/lbaas/v1/.gitignore

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

- add the front end port - use the front end port for the load balancer - point out the service annotations don't work Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

To avoid adding knowledge of specific load balancer implementers in `loadbalancers.go`, this moves the `GetLoadBalancer` logic for EMLB into `emlb.go`. In the future we can refactor the pre-existing `GetLoadBalancer` code in order to push that down to `metallb`, `kubevip`, `empty`, etc., keeping `loadbalancers.go` focused on the work of deciding which implementer to use and how to translate from the cloud-provider interface to our internal implementer interface.

When deploying a new LoadBalancer service, we would see a "Failed to update..." error message in the logs, and 2 load balancers would be created on the Equinix side, with only 1 of them reflected in the service annotations. This changes our `Update` call to update the k8s service object with a `Patch` call; by doing this we focus our k8s API calls on changing only the properties that our provider has modified, which reduces the chance that something else has concurrently made conflicting changes.

Co-authored-by: Charles Treatman <ctreatma@users.noreply.github.com> Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

After inspecting API responses in the Equinix Metal console, I saw that origin pool data is not nested under the corresponding port. Instead, `ports` is an array of ports, and `pools` is an array of arrays of pools. The top-level index of `pools` corresponds to the top-level index of `ports`, so `pools[i]` is an array of all of the origin pools assigned to `ports[i]`. The old code was looking for origin data in the wrong place, so it was failing to delete origins that are no longer needed. Now it is looking in the right place and successfully deleting old origins.

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

- Allow the control plane LoadBalancer service to be managed by the provider if `usesBGP` is false for the selected load balancer implementation - Add a reconciler for LBaaS-based control plane LoadBalancer - Add a `LoadBalancerID` config option to specify the ID of the load balancer to use for the control plane

displague · 2023-10-24T21:55:25Z

README.md

 - [kube-vip](#kube-vip)
 - [MetalLB](#metallb)
 - [empty](#empty)

 CCM does **not** deploy _any_ load balancers for you. It limits itself to managing the Equinix Metal-specific
 API calls to support a load balancer, and providing configuration for supported load balancers.

+##### Equinix Metal Load Balancer
+
+Equinix Metal Load Balancer (EMLB) is a beta service that is available to a limited number of Equinix Metal customers that provides managed layer 4 load balancers.


Suggested change

Equinix Metal Load Balancer (EMLB) is a beta service that is available to a limited number of Equinix Metal customers that provides managed layer 4 load balancers.

[Equinix Metal Load Balancer](https://github.com/equinix/lbaas-api-docs) (EMLB) is a [service under development](https://feedback.equinixmetal.com/network/p/load-balancer-as-a-service-lbaas), currently in [beta](https://deploy.equinix.com/enroll-beta-lab/), that provides managed layer 4 load balancers.

displague · 2023-10-24T21:59:51Z

README.md

-For the control plane nodes, the Equinix Metal CCM uses static Elastic IP assignment, via the Equinix Metal API, to tell the
-Equinix Metal network which control plane node should receive the traffic. For more details on the control plane
-load-balancer, see [this section](#control-plane-load-balancing).
-
 #### Service LoadBalancer Implementations


nit: We should link to this section above in the table where the loadbalancer setting is introduced without any options described.

8a048dc#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R224

displague · 2023-10-24T22:05:28Z

README.md

@@ -233,6 +233,7 @@ This section lists each configuration option, and whether it can be set by each
 | Kubernetes Service annotation to set EIP metro                                                                                                               |                | `METAL_ANNOTATION_EIP_METRO`            | `annotationEIPMetro`           | `"metal.equinix.com/eip-metro"`                              |
 | Kubernetes Service annotation to set EIP facility                                                                                                            |                | `METAL_ANNOTATION_EIP_FACILITY`         | `annotationEIPFacility`        | `"metal.equinix.com/eip-facility"`                           |
 | Tag for control plane Elastic IP                                                                                                                             |                | `METAL_EIP_TAG`                         | `eipTag`                       | No control plane Elastic IP                                  |
+| ID for control plane Equinix Metal Load Balancer                                                                                                                             |                | `METAL_LOAD_BALANCER_ID`                         | `loadBalancerID`                       | No control plane Equinix Metal Load Balancer                                  |


I think this option would fit with how we offer arguments in METAL_LOAD_BALANCER, as used by metallb:

METAL_LOAD_BALANCER=emlb://sv

METAL_LOAD_BALANCER=emlb://?loadbalancerID=123

METAL_LOAD_BALANCER=emlb://?existing-only-tagged=cpem

Perhaps we don't need a separate option for ID.

(This is conversational, not merge blocking)

I know there is no support for tags today.

I'm curious how well CPEM will handle the limitation of working with a single LoadBalancer and whether this is more of a development feature or if this is an argument that users will want in their clusters.

Alternatively, if new options are the preferred approach, perhaps with #472, we may want to expose CLI / environment / config arguments in the future for the metallb options only offered through METAL_LOAD_BALANCER query parameters today.

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 24, 2023

k8s-ci-robot requested a review from deitch October 24, 2023 19:12

k8s-ci-robot requested a review from displague October 24, 2023 19:12

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 24, 2023

ctreatma commented Oct 24, 2023

View reviewed changes

internal/lbaas/v1/.gitignore Outdated Show resolved Hide resolved

cprivitere and others added 23 commits October 24, 2023 14:24

feat: Iniitial Create of EMLB

0fb7d72

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

define an Equinix Metal Load Balancer client and initialize it

f178e49

chore: standardize markdown

42faf22

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

chore: tabs to spaces

2fa2f7f

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

chore: formatting remove line

e3af08d

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

feat: parse config for new emlb lbs

248a36c

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

exchange Metal API key for LB OAuth tokenbefore making LB API requests

1aef0da

add map of known locations

f2bbcf0

scaffold out the LBaaS integration

1182dde

add more detailed implementation outline in comments

15c1de1

pass in API key and project ID for EMLB

69fd1a9

fix: MapKeys() not Keys()

9d7d97f

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

feat: support tilt based development

e53c43f

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: get off experimental builder

c08e63c

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

refactor: introduce usesBGP flag for loadbalancers

1c80d87

feat: move example dev files to dev/

24b62b0

fix: ignore dev directory for rebuilds

978e07b

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

feat: create an Equinix Metal Load Balancer for a k8s service

322373c

return actual service status for non-BGP load balancers

716c3ef

set LoadBalancerIP for backwards compatibility

abd52f3

fix: various updates and comments

4456f42

- add the front end port - use the front end port for the load balancer - point out the service annotations don't work Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: ensure LoadBalancer is implemented

b44bae8

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: use front end port

0fe57f0

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

ctreatma and others added 20 commits October 24, 2023 14:25

fix: handle multiple ports for a service

5a5df10

fix: use a consistent load balancer name

7e80f12

fix: commit svc updates

f9ede3e

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: if service already has a load balancer ID, don't create a new one

b5b86cf

feat: add code for deletes

21328f0

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

feat: add interface function so deletes work

937350f

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: delete pools when load balancer is deleted

adcfa00

feat: report status correctly for new LB

590d80b

Co-authored-by: Charles Treatman <ctreatma@users.noreply.github.com> Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

Wire up UpdateService for EMLB

d4507a3

change ID properties to strings

d35cae5

chore: bump go mods and version

540c751

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: support latest k8s code new flags

d7eb273

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: Tiltfile support new cpem controller name

bac1e6c

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: revert k8s and metallb updates

7f2cfe6

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

fix: remove cloud-sa.json from root

69c4c2a

Signed-off-by: Chris Privitere <23177737+cprivitere@users.noreply.github.com>

chore: refactor manager to replace add/update functions with reconcile

36cf224

ctreatma force-pushed the emlb branch from 35bed9c to e81f2c1 Compare October 24, 2023 19:25

ctreatma marked this pull request as ready for review October 24, 2023 19:26

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 24, 2023

k8s-ci-robot requested a review from cprivitere October 24, 2023 19:26

ctreatma added 2 commits October 24, 2023 15:06

chore: address lint failures

2a3dc23

docs: update README with EMLB support and config

8a048dc

displague reviewed Oct 24, 2023

View reviewed changes

displague merged commit 5a5aa8a into kubernetes-sigs:main Oct 26, 2023
5 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for Equinix Metal Load Balancer #470

feat: add support for Equinix Metal Load Balancer #470

ctreatma commented Oct 24, 2023 •

edited

k8s-ci-robot commented Oct 24, 2023

displague Oct 24, 2023

displague Oct 24, 2023 •

edited

displague Oct 24, 2023 •

edited

displague Oct 24, 2023

displague Oct 24, 2023 •

edited

	Equinix Metal Load Balancer (EMLB) is a beta service that is available to a limited number of Equinix Metal customers that provides managed layer 4 load balancers.
	[Equinix Metal Load Balancer](https://github.com/equinix/lbaas-api-docs) (EMLB) is a [service under development](https://feedback.equinixmetal.com/network/p/load-balancer-as-a-service-lbaas), currently in [beta](https://deploy.equinix.com/enroll-beta-lab/), that provides managed layer 4 load balancers.

feat: add support for Equinix Metal Load Balancer #470

feat: add support for Equinix Metal Load Balancer #470

Conversation

ctreatma commented Oct 24, 2023 • edited

k8s-ci-robot commented Oct 24, 2023

displague Oct 24, 2023

Choose a reason for hiding this comment

displague Oct 24, 2023 • edited

Choose a reason for hiding this comment

displague Oct 24, 2023 • edited

Choose a reason for hiding this comment

displague Oct 24, 2023

Choose a reason for hiding this comment

displague Oct 24, 2023 • edited

Choose a reason for hiding this comment

ctreatma commented Oct 24, 2023 •

edited

displague Oct 24, 2023 •

edited

displague Oct 24, 2023 •

edited

displague Oct 24, 2023 •

edited