Keep the previous certificate in the bundle as well as the new one to prevent periods of service outage

[vassilvk]

@arkaitzj wrote:

During our tests we discovered that every time the certificates rotated we got a lot of TLS bad certificate errors in our cluster.
2022/06/02 13:46:25 http: TLS handshake error from 10.213.162.79:60186: remote error: tls: bad certificate

When we looked into it we discovered that the refresher sets the secret and the caBundle in the webhooks at the same time.
However, that does not account for the time it takes for a pod to get the secret updated and the operator to refresh the certificate.
During that period of time the control-plane is using the new caBundle and the operator the old one, failing all requests and provoking a service outage since none of the resources created at that time will be serviced by kubemod.

The proposed solution keeps the previous certificate in the bundle along with the new one, so that the operator can work well before and after swapping its own certificate.
This means there are always 2 certificates in the caBundle and each certificate will be valid for 2 cron periods, if this is a concern, halving the cron period should work.

[vassilvk]

Closed by #14