Add support for loadbalancer interface #11

johnsca · 2021-04-06T20:37:40Z

The current way that the relations are set up for CK, with kubeapi-load-balancer sitting in between the master and workers, is
unnecessarily convoluted, and leads to mistakes due to the relations not being set up properly since they have to be done differently depending on whether or not kubeapi-load-balancer is being used.

This change makes this charm support the loadbalancer interface and depends on the master to forward the LB address via the kube-control relation. This means that there's only one relation between the workers and master, the endpoint address is always handed off in the same way, and the relations are consistent regardless of whether the API LB charm is being used or if another provider (such as a cloud integrator charm) is providing the LB.

Part of lp:1921776

The current way that the relations are set up for CK, with kubeapi-load-balancer sitting in between the master and workers, is unnecessarily convoluted, and leads to mistakes due to the relations not being set up properly since they have to be done differently depending on whether or not kubeapi-load-balancer is being used. This change makes this charm support the `loadbalancer` interface and depends on the master to forward the LB address via the `kube-control` relation. This means that there's only one relation between the workers and master, the endpoint address is always handed off in the same way, and the relations are consistent regardless of whether the API LB charm is being used or if another provider (such as a cloud integrator charm) is providing the LB. Part of [lp:1897818][] [lp:1897818]: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1897818

This adds support for the `loadbalancer` interface so that cloud-native LBs can be provided by the integrator charms. Additionally, it simplifies the confusing way the relations between the masters and workers change depending on whether kubeapi-load-balancer is being used or not by making that use the same `lb-provider` endpoint and always forwarding the API endpoint URLs via the `kube-control` relation. Part of [lp:1897818][] Depends on: * juju-solutions/loadbalancer-interface#13 * juju-solutions/interface-kube-control#33 * charmed-kubernetes/charm-kubernetes-worker#84 * charmed-kubernetes/charm-kubeapi-load-balancer#11 [lp:1897818]: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1897818

Simplify the relations between the master, worker, and API LB charms in favor of the newer `loadbalancer` interface relation. Part of [lp:1897818][] Depends on: * juju-solutions/loadbalancer-interface#13 * juju-solutions/interface-kube-control#33 * charmed-kubernetes/charm-kubernetes-control-plane#153 * charmed-kubernetes/charm-kubernetes-worker#84 * charmed-kubernetes/charm-kubeapi-load-balancer#11 [lp:1897818]: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1897818

Cynerva

I'm not convinced this is a good change. Previously, the network connection between kubeapi-load-balancer <-> kubernetes-worker was modeled with a direct relation. That made it possible to support cross-model relations and network bindings / network spaces.

With this change, we instead proxy the connection info through kubernetes-master. We can't properly support network spaces or cross-model relations when doing that, can we?

Cynerva · 2021-04-07T15:53:58Z

reactive/load_balancer.py

+                        'port': port,
+                    }
+                    for address, port in itertools.product(
+                        request.backends, request.port_mapping.values()


It looks like we're ignoring the port_mapping keys. Are consumers smart enough to handle this? If the consumer requests port-mapping {443: 6443} but the kubeapi-load-balancer charm is configured with port: 12345, what happens?

How is the real port communicated to the consumer?

Hrm, yeah, this is a case where the philosophy diverges, since the loadbalancer interface (and all other LB providers) expect the consumer to dictate what the ingress port should be. This should honor the port on the request and the config option on the kapi-lb charm should be deprecated.

Cynerva · 2021-04-07T15:56:36Z

reactive/load_balancer.py

    else:
-        website.configure(port=hookenv.config('port'))
+        address = hookenv.unit_get('public-address')


public-address is usually not what we want. We want internal services to talk to eachother via internal IPs, e.g. ingress-address on a relation. If we use a public address, then kubeapi-load-balancer must be juju expose'd, or other units will not be able to reach it (on clouds with network security, like AWS). Bare metal deployments also tend to have stricter network requirements than we test with, and I expect this would cause problems with network routing in those environments.

The loadbalancer interface protocol can already support multiple requests with different ones marked as public or not. I was already thinking that it probably makes sense for the k8s-master to request two LBs, one public for the kube config files it writes for the user to download, and one private for internal use.

That still doesn't cover the case where the workers are CMR and on a different network than the master / kapi-lb, but that seems like a pretty strange setup anyway. Even in strict network environments, I would still at least expect the cluster to be in one network and then potentially need some form of "public" ingress for clients in another network.

reactive/load_balancer.py

Cynerva · 2021-04-07T16:07:28Z

reactive/load_balancer.py

+    address = _get_lb_address()
+    website.configure(port=hookenv.config('port'),
+                      private_address=address,
+                      hostname=address)


Without hacluster or loadbalancer-ips configured, this will tell other units to connect to a public-address, which is not ideal.

The correct address to use here is the ingress address of the website relation (per relation id). This was previously handled by omitting the private_address and hostname arguments, in which case, interface-http uses the proper ingress address: https://github.com/juju-solutions/interface-http/blob/632131b1f122daf6fb601fd4c9f1e4dbb1a92e09/provides.py#L31

Cynerva · 2021-04-07T16:12:38Z

reactive/load_balancer.py

+        services = []
+        if apiserver:
+            services.extend(apiserver.services())
+        for request in lb_consumers.all_requests:


We should probably verify the request protocol is http/https, since that's all we can really loadbalance to.

Cynerva · 2021-04-16T20:48:11Z

reactive/load_balancer.py

+        private_address = lb_address
+        public_address = lb_address
+    else:
+        rel_id = lb_consumers.relations[0].id


After relating kubernetes-master:loadbalancer-internal and kubernetes-master:loadbalancer-external to kubapi-load-balancer:lb-consumers:

unit-kubeapi-load-balancer-0: 15:43:43 ERROR unit.kubeapi-load-balancer/0.juju-log lb-consumers:18: Hook error: Traceback (most recent call last): File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main bus.dispatch(restricted=restricted_mode) File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch _invoke(other_handlers) File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke handler.invoke() File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke self._action(*args) File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/reactive/load_balancer.py", line 265, in provide_lb_consumers rel_id = lb_consumers.relations[0].id IndexError: list index out of range

Cynerva

I'm too braindead today to review this properly. I did hit a hook error though, check it out.

reactive/load_balancer.py

This adds support for the `loadbalancer` interface so that cloud-native LBs can be provided by the integrator charms. Additionally, it simplifies the confusing way the relations between the masters and workers change depending on whether kubeapi-load-balancer is being used or not by making that use the same `lb-provider` endpoint and always forwarding the API endpoint URLs via the `kube-control` relation. Part of [lp:1897818][] Depends on: * juju-solutions/loadbalancer-interface#13 * juju-solutions/interface-kube-control#33 * charmed-kubernetes/charm-kubernetes-worker#84 * charmed-kubernetes/charm-kubeapi-load-balancer#11 [lp:1897818]: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1897818

johnsca · 2021-05-13T14:35:14Z

Manual test run result using the modified k8s-master and k8s-worker charms.

This adds support for the `loadbalancer` interface so that cloud-native LBs can be provided by the integrator charms. Additionally, it simplifies the confusing way the relations between the masters and workers change depending on whether kubeapi-load-balancer is being used or not by making that use the same `lb-provider` endpoint and always forwarding the API endpoint URLs via the `kube-control` relation. Part of [lp:1897818][] Depends on: * juju-solutions/loadbalancer-interface#13 * juju-solutions/interface-kube-control#33 * charmed-kubernetes/charm-kubernetes-worker#84 * charmed-kubernetes/charm-kubeapi-load-balancer#11 [lp:1897818]: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1897818

Cynerva

I think the tests could use a bit of cleaning up, but no show stoppers here. Good work.

Cynerva · 2021-05-19T16:21:07Z

tests/data/bundle.yaml

+    channel: edge
+    num_units: 1
+    resources:
+      easyrsa: 5


I would remove resource revs from the bundle. No point specifying those if the charm revisions themselves are unspecified.

Cynerva · 2021-05-19T16:24:45Z

tests/data/bundle.yaml

+    expose: true
+    num_units: 1
+    options:
+      channel: 1.20/stable


This should be at least 1.21/stable, possibly 1.22/edge to match what's in kubernetes-master.

Cynerva · 2021-05-19T16:25:07Z

tests/data/bundle.yaml

+    constraints: cores=4 mem=4G root-disk=16G
+    num_units: 1
+    options:
+      channel: 1.20/stable


Should be 1.21/stable or 1.22/edge?

Cynerva · 2021-05-19T16:33:39Z

tests/data/bundle.yaml

+  kubeapi-load-balancer:
+    charm: {{k8s_lb_charm}}
+    num_units: 1
+    expose: true


kubeapi-load-balancer is missing constraints and to fields.

Cynerva · 2021-05-19T16:35:18Z

tests/integration/test_kubeapi-load-balancer_integration.py

+    await master.add_relation("loadbalancer", "kubeapi-load-balancer")
+    await worker.add_relation("kube-api-endpoint", "kubeapi-load-balancer")
+    await ops_test.model.wait_for_idle(wait_for_active=True, timeout=30 * 60)
+    _check_status_messages(ops_test)


Looks like this test will leave the cluster with the old-style relations. Maybe it should revert it back to the new relations when it's done?

Cynerva · 2021-05-19T16:45:00Z

tests/data/bundle.yaml

+    charm: cs:~containers/kubernetes-master
+    channel: edge
+    constraints: cores=2 mem=4G root-disk=16G
+    expose: true


kubernetes-master typically isn't exposed in deployments that have kubeapi-load-balancer

* Add support for loadbalancer interface This adds support for the `loadbalancer` interface so that cloud-native LBs can be provided by the integrator charms. Additionally, it simplifies the confusing way the relations between the masters and workers change depending on whether kubeapi-load-balancer is being used or not by making that use the same `lb-provider` endpoint and always forwarding the API endpoint URLs via the `kube-control` relation. Part of [lp:1897818][] Depends on: * juju-solutions/loadbalancer-interface#13 * juju-solutions/interface-kube-control#33 * charmed-kubernetes/charm-kubernetes-worker#84 * charmed-kubernetes/charm-kubeapi-load-balancer#11 [lp:1897818]: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1897818 * Fix lint error * Add test for status reporting of incomplete LB relation * Fix lint errors in test * Fix hook error when built with old version of kube-control interface * Split lb-provider endpoint into separate, more explicit ones for internal / external * Fix handling of internal LB response and missing endpoints * Move test to new LB relation pattern and use edge charms * Improve status reporting around the auth-webhook and skip trying to create secrets before apiserver is available * Create a ~/.kube/config for the ubuntu user * Fix NoneType error for old-style relation * Fix unit test * Revert drive-by related to auth setup status reporting; splitting to separate bug

* Support loadbalancer interface between master and API LB Simplify the relations between the master, worker, and API LB charms in favor of the newer `loadbalancer` interface relation. Part of [lp:1897818][] Depends on: * juju-solutions/loadbalancer-interface#13 * juju-solutions/interface-kube-control#33 * charmed-kubernetes/charm-kubernetes-control-plane#153 * charmed-kubernetes/charm-kubernetes-worker#84 * charmed-kubernetes/charm-kubeapi-load-balancer#11 [lp:1897818]: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1897818 * Split k8s-master:lb-provider relation into external / internal * Update core and converged bundle fragments

johnsca mentioned this pull request Apr 6, 2021

Add support for loadbalancer interface charmed-kubernetes/charm-kubernetes-control-plane#153

Merged

johnsca mentioned this pull request Apr 6, 2021

Support loadbalancer interface between master and API LB charmed-kubernetes/bundle#796

Merged

Cynerva reviewed Apr 7, 2021

View reviewed changes

Add support for private LBs and add listeners for requested ports

6ce5277

Cynerva suggested changes Apr 16, 2021

View reviewed changes

johnsca added 2 commits April 22, 2021 10:23

Fix mapping of listen ports (servers) to backends

2e0d357

Reject LB requests for unsupported protocols

013ad11

johnsca requested a review from Cynerva April 22, 2021 14:44

Cynerva suggested changes Apr 22, 2021

View reviewed changes

reactive/load_balancer.py Show resolved Hide resolved

Fix handling of apiserver endpoint

bb0a685

johnsca marked this pull request as draft May 7, 2021 21:27

johnsca added 2 commits May 13, 2021 10:23

Add integration tests for new and old LB relation patterns

73a22ba

Adapt and switch to GH Workflow from master charm

d0d48bd

johnsca marked this pull request as ready for review May 13, 2021 15:33

Use actions-operator on integration tests for vSphere environment

1fb70f3

Cynerva approved these changes May 19, 2021

View reviewed changes

Cynerva merged commit d661b03 into master May 19, 2021

Cynerva deleted the johnsca/lp/1921776/add-loadbalancer-interface branch May 19, 2021 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for loadbalancer interface #11

Add support for loadbalancer interface #11

johnsca commented Apr 6, 2021 •

edited

Loading

Cynerva left a comment

Cynerva Apr 7, 2021

johnsca Apr 7, 2021

Cynerva Apr 7, 2021

johnsca Apr 7, 2021

Cynerva Apr 7, 2021

Cynerva Apr 7, 2021

Cynerva Apr 16, 2021

Cynerva left a comment

johnsca commented May 13, 2021

Cynerva left a comment

Cynerva May 19, 2021

Cynerva May 19, 2021

Cynerva May 19, 2021

Cynerva May 19, 2021

Cynerva May 19, 2021

Cynerva May 19, 2021

Add support for loadbalancer interface #11

Add support for loadbalancer interface #11

Conversation

johnsca commented Apr 6, 2021 • edited Loading

Cynerva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cynerva left a comment

Choose a reason for hiding this comment

johnsca commented May 13, 2021

Cynerva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnsca commented Apr 6, 2021 •

edited

Loading