HNC: very slow startup after first installation

[adrianludwin]

The builtin cert manager seems to stall the first time we install HNC. This is a poor user experience.

[adrianludwin]

There seem to be two problems here:

• When the Secret is first created, it takes a long time to propagate the updated secret to the Pod. This is expected.
• Leader election seems to be taking a while too, even though we only have one pod and no ability to run a hot standby.

[adrianludwin]

Here are some experiments, from the time kubectl apply -f <manifests> finishes to the time the HNCConfiguration reconciler finishes for the first time.

1. At head:

• Attempt 1: 95s
• Attempt 2: 95s

2. Force HNC to exit after the secret is first written and restart

• Attempt 1: 35s
• Attempt 2: 32s

3. As above, but without leader election

• Attempt 1: 11s
• Attempt 2: 12s

The difference between 2 and 3 (~22s) represents two leader elections, from which we can gather that LE takes about 11s. This suggests that without forcing HNC to exit but disabling LE, we'd reduce our startup time from ~95s to ~84s - not a huge advantage.

However, the second restart of HNC, without LE, takes only ~4s, vs ~15s with LE. This is probably worth doing if only for the high availability aspect (e.g. validators going down for 4s vs 15s).

[adrianludwin]

Here's the patch to make the cert manager exit when the secret is first written:

diff --git a/incubator/hnc/third_party/open-policy-agent/gatekeeper/pkg/webhook/certs.go b/incubator/hnc/third_party/open-policy-agent/gatekeeper/pkg/webhook/certs.go
index 95e40af2..f238c0b1 100644
--- a/incubator/hnc/third_party/open-policy-agent/gatekeeper/pkg/webhook/certs.go
+++ b/incubator/hnc/third_party/open-policy-agent/gatekeeper/pkg/webhook/certs.go
@@ -122,6 +122,7 @@ func (cr *CertRotator) refreshCertIfNeeded() error {
                if err := cr.client.Get(context.Background(), cr.SecretKey, secret); err != nil {
                        return false, errors.Wrap(err, "acquiring secret to update certificates")
                }
+               noSecret := secret.Data == nil
                if secret.Data == nil || !cr.validCACert(secret.Data[caCertName], secret.Data[caKeyName]) {
                        crLog.Info("refreshing CA and server certs")
                        if err := cr.refreshCerts(true, secret); err != nil {
@@ -129,6 +130,10 @@ func (cr *CertRotator) refreshCertIfNeeded() error {
                                return false, nil
                        }
                        crLog.Info("server certs refreshed")
+                       if noSecret {
+                               crLog.Info("Initial secret has just been written; exiting so secret gets projected faster")
+                               os.Exit(0)
+                       }
                        return true, nil
                }
                // make sure our reconciler is initialized on startup (either this or the above refreshCerts() will call this)