Kubernetes support

[bartversluijs]

The first "official" Kubernetes support using a Helm chart.
It's "official" in the sense that I'm more confident in this Helm chart than the last one, because of it's features and a better upgrade path in the future (provided that there aren't breaking changes in openBalena itself).

Keep in mind, this Kubernetes Helm chart is made with the added thought that you have knowledge about Kubernetes and Helm

Attention
This version breaks with the old version of the Helm chart, but upgrading to the new one is advised
I've tested the upgrade and it's doable, but you've to have some knowledge about Kubernetes

---

What's new

• There's a README especially for the openBalena Chart
• It now uses the HAProxy Ingress Controller for the VPN as well as the HTTP routing, instead of HAProxy and NGINX. Only 1 Load Balancer will be used now instead of 2.
• More configurable values per service, such as the image, labels, annotations, affinity, resources, nodeSelector, service labels, service annotations and service external IP, plus the storage and storage class
• Ingresses are more configurable
• Cert-manager isn't installed by default, as well as an issuer, which makes it more configurable to create TLS certificates
• It's a standalone Helm Chart instead of an includes 'k8s' script
• The tunnel.<domain> is supported via an added Ingress TLS rule, which is proxied to HAProxy, which adds the TLS to the TCP service

Known issues

• The chart depends on the HAProxy Ingress Chart v0.13, which is still in beta. The v0.13 is necessary because of new features, like better configurations per scope and new TCP services. Because of this, there may be some issues. I've seen that the HAProxy isn't always updated properly while doing multiple things at once. So if some routing fails, please reastart the HAProxy pod first.

---

Breaking changes

• The whole values.yaml is changed. This is because it now uses the same format as many Helm charts.
• When upgrading the Helm chart if you've installed the previous one, Helm will give errors that many things can't be upgraded because you're not allowed to change them (such as statefulset values)

---

Upgrading
Upgrading is simple and hard at the same time. As I mentioned, a simple helm upgrade will not do the trick, because you've to change your values.yaml first, and after you've done that, you'll get errors from Helm itself.

I hoped I could come up with a better upgrade path, but openBalena isn't out of beta yet and the first Helm Chart isn't an official Helm Chart. I did the best I could, but making these breaking changes will improve upgrades later on.

So the upgrade path I chose was as follows:

1. Create a new file called config/kubernetes.yaml
2. Copy the contents of kubernetes/values.template.yaml and paste it in config/kubernetes.yaml
3. Open your current values in config/k8s/settings.yaml
4. Copy every string (like hostname, config.email etc) in the right variable of config/kubernetes.yaml. Every environment variable is added as a comment next to the variable's value in the old settings.yaml
5. Backup all your data (and be absolutely sure you've done this and know how to access this data)
6. Create a new Kubernetes cluster or install everything in a new namespace, but make sure there's no link to the old deployment
7. Install the new Kubernetes Helm Chart with api.replicas set to 0 (because of migrations later on)
8. Copy all the backupped data to the new Helm deployment, or attach the old PVC's to the new deployments / StatefulSets
9. Make sure everything looks good, set api.replicas to 1 or more and upgrade the Helm Chart
10. Everything is good to go!

I've tested this approach with a openBalena cluster with more than 200 devices and all seems to go well!

---

Need help?
If you need help, please open a post on the Balena Forums - openBalena section or react in the Kubernetes for scaling topic