Implement cilium upgrade using Helm

[asauber]

Summary

Implements the cilium upgrade command using a thin wrapper around Helm rather than direct Kubernetes API calls.

A few of the self-contained features are broken down into separate commits.

Motivation

This implementation allows for more consistent documentation and behavior among Cilium installation methods, such as those docs which use only Helm directly.

This implementation includes an alternate implementation for the cilium {clustermesh,hubble} {enable,disable} family of commands. The "legacy" implementation of those commands came from a time when using Helm was insecure (it installed a daemon into the cluster with cluster-admin privileges. Helm is now relatively safe to use, thus we use Helm here as a library to implement all features for Cilium installation, upgrade, and feature enable/disable whcih are currently implemented in the Cilium Helm chart.

Example Usage

For example, we can use this new upgrade command to enable Cluster Mesh on a set of two Cilium clusters with the following procedure.

Create a kind cluster with the following config

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: c1
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
  podSubnet: "10.10.0.0/16"
  serviceSubnet: "10.50.0.0/16"

kind create cluster --config c1.kindconfig.yaml

Test that the kind cluster came up with k get pods -A

Install Cilium using the CLI, making sure to set a Cluster ID and Cluster Name

export CILIUM_CLI_MODE=helm
cilium install --help
# note the suggestion to use --helm-set flags
cilium install --helm-set cluster.id=1 --helm-set cluster.name=cluster1

Create a second cluster using kind with the following config

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: c2
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
  podSubnet: "10.11.0.0/16"
  serviceSubnet: "10.51.0.0/16"

kind create cluster --config c2.kindconfig.yaml

Check that you are authed for Cluster kind-c2

Run k get pods -A and look for kube-apiserver-c2-control-plane

Install Cilium using the CLI, making sure to set a Cluster ID and Cluster Name

export CILIUM_CLI_MODE=helm
cilium install --help
# note the suggestion to use --helm-set flags
cilium install --helm-set cluster.id=2 --helm-set cluster.name=cluster2

Enable clustermesh on Cluster 1 using the new Helm upgrade command

export CILIUM_CLI_MODE=helm
cilium upgrade --help
# note the suggestion to use --helm-set flags
cilium upgrade --context $CLUSTER1 \
    --helm-set clustermesh.useAPIServer=true \
    --helm-set clustermesh.apiserver.service.type=NodePort
# observe that the clustermesh apiserver is running
k get pods -A

Prep environment for the clustermesh enable.

export CLUSTER1=kind-c1 CLUSTER2=kind-c2

Extract the CA cert from Cluster 1 and install it into Cluster 2, then restart Cilium on Cluster 2

kubectl --context $CLUSTER2 delete secret -n kube-system cilium-ca && \
kubectl --context $CLUSTER1 get secrets -n kube-system cilium-ca -oyaml \
    | kubectl --context $CLUSTER2 apply -f -
# you should see the following output
# secret "cilium-ca" deleted
# secret/cilium-ca created
# proceed to restart Cilium on kind-c2
kubectl --context $CLUSTER2 delete pod -l app.kubernetes.io/part-of=cilium -A

Enable clustermesh on Cluster 2 using the new Helm upgrade command

export CILIUM_CLI_MODE=helm
cilium upgrade --help
# note the suggestion to use --helm-set flags
cilium upgrade --context $CLUSTER2 \
    --helm-set clustermesh.useAPIServer=true \
    --helm-set clustermesh.apiserver.service.type=NodePort

Move the secrets expected by the cilium clustermesh connect command (this quirk to be fixed with a helm-mode PR for that command)

k get secrets --context $CLUSTER1 -n kube-system clustermesh-apiserver-remote-cert \
    -oyaml \
    | sed 's/name: .*/name: clustermesh-apiserver-client-cert/' \
    | k apply --context $CLUSTER1 -f -

k get secrets --context $CLUSTER2 -n kube-system clustermesh-apiserver-remote-cert \
    -oyaml \
    | sed 's/name: .*/name: clustermesh-apiserver-client-cert/' \
    | k apply --context $CLUSTER2 -f -

Connect the two clusters using ClusterMesh

cilium clustermesh connect --context $CLUSTER1 --destination-context $CLUSTER2

Run the Multi-Cluster connectivity tests.

cilium connectivity test --context $CLUSTER1 --multi-cluster $CLUSTER2

(This has been included as a CI workflow)

closes #1493

[asauber]

Fix dry run

[michi-covalent]

lgtm once the conflict is resolved

[michi-covalent]

removing github-sec and ci-structure from reviewers. this pr got updated, and now it's only modifying files owned by cli and sig-hubble teams.

[michi-covalent]

EKS (ENI): failed #1514