TLS certificates hot reloading for Hubble and Relay

[kaworu]

Fix #13092

[gandro]

Looks good overall. I know this is WIP, but a very simple unit test which creates files via ioutil.TempDir would be great to have as well.

The current code assumes that the files already exist (which is fine for the hot-reloading use-case). I wonder however if we could have the interface also work somehow where you would only get a WatchedConfig pointer when the files actually exist and the certs/keys have been read successfully. For example, NewWatchedConfig could return a <-chan *WatchedConfig.

[rolinh]

I know this is still WIP but I left a few comments nonetheless. I have some ideas to make this a bit more generic but I'd rather take this offline rather than trying to detail it here.

[kaworu]

Unit tests added.

I think overall we have a good enough coverage of the certloader package. There are a couple of untested scenarios outside of our current use-cases; like non-mutual TLS client/server configuration generation, partial Reload (because k8s update all the files at once through symlink'ing), etc.

[kaworu]

test-me-please

[rolinh]

TLDR: all of this looks very solid to me! Nice work! 🚀

I tested this feature extensively by triggering certs generation via helm upgrade:

$ for name in $(kubectl -n kube-system get secrets -l app.kubernetes.io/managed-by=Helm -o json | jq ".items[].metadata.name"); do echo $name; kubectl -n kube-system get secrets $(echo $name | tr -d \" ) -o json | jq -r '.data["tls.crt"]' | base64 -d | openssl x509 -noout -text | grep -A 1 Serial; done 
"hubble-relay-client-certs"
        Serial Number:
            8c:5f:2d:3f:3e:6f:39:91:85:b9:25:32:45:21:b5:7c
"hubble-relay-server-certs"
        Serial Number:
            9b:15:56:f0:c6:5d:41:b9:df:84:a4:aa:48:29:06:6a
"hubble-server-certs"
        Serial Number:
            ea:38:8f:13:f8:f2:40:1a:a5:66:32:b2:9b:db:4b:25
$ helm upgrade cilium ./cilium --namespace kube-system --set global.nodeinit.enabled=true --set global.kubeProxyReplacement=partial --set global.hostServices.enabled=false --set global.externalIPs.enabled=true --set global.nodePort.enabled=true --set global.hostPort.enabled=true --set global.pullPolicy=IfNotPresent --set config.ipam=kubernetes --set global.hubble.enabled=true --set global.hubble.listenAddress=":4244" --set global.hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" --set global.hubble.relay.enabled=true --set global.hubble.ui.enabled=true --set global.hubble.relay.tls.enabled=true
$ for name in $(kubectl -n kube-system get secrets -l app.kubernetes.io/managed-by=Helm -o json | jq ".items[].metadata.name"); do echo $name; kubectl -n kube-system get secrets $(echo $name | tr -d \" ) -o json | jq -r '.data["tls.crt"]' | base64 -d | openssl x509 -noout -text | grep -A 1 Serial; done
"hubble-relay-client-certs"
        Serial Number:
            05:de:64:f1:21:31:bc:e2:36:4c:2a:7f:3b:c1:de:51
"hubble-relay-server-certs"
        Serial Number:
            7a:48:15:f7:96:4b:69:f4:fe:d1:e2:f1:75:51:eb:78
"hubble-server-certs"
        Serial Number:
            c6:48:98:76:c1:04:df:4b:78:89:9c:a0:5e:72:7f:c5

From the above output, we can see that each cert was regenerated. In the meantime, I ran hubble observe -f against Hubble Relay: no connection loss (which is expected).

I then deleted a cilium pod to force a re-connection from Hubble Relay with the new certs: success!

Next test is to kubectl exec into a cilium pod that was not restarted after a helm upgrade ... (which generates new certs) and ensure that it now uses the new cert:

# openssl s_client -connect localhost:4244 -showcerts 2>/dev/null | openssl x509 -noout -text | grep -A 1 Serial 
        Serial Number:
            c6:48:98:76:c1:04:df:4b:78:89:9c:a0:5e:72:7f:c5

[fristonio]

Great work!!! LGTM 🚀 🚀

[rolinh]

netnext failure is a flake, marking as ready to merge.