The webhook not start because of certFile check when deploy g8r out of cluster

[Poor12]

I have a special scene which g8r is deployed out of cluster and I configure the --kubeconfig option in controller-runtime to make g8r watch the user behavior in the cluster which I would like to.

In this case, cert-controller will generate ca and update the secret which is in the remote cluster. However, the local file, such as tls.crt in certDir will not update. So, because of the certFile check below, the webhook will not start.

cert-controller/pkg/rotator/rotator.go

Lines 700 to 722 in 54af894

	// ensureCertsMounted ensure the cert files exist.
	func (cr *CertRotator) ensureCertsMounted() {
	checkFn := func() (bool, error) {
	certFile := cr.CertDir + "/" + certName
	_, err := os.Stat(certFile)
	if err == nil {
	return true, nil
	}
	return false, nil
	}
	if err := wait.ExponentialBackoff(wait.Backoff{
	Duration: 1 * time.Second,
	Factor: 2,
	Jitter: 1,
	Steps: 10,
	}, checkFn); err != nil {
	crLog.Error(err, "max retries for checking certs existence")
	close(cr.certsNotMounted)
	return
	}
	crLog.Info(fmt.Sprintf("certs are ready in %s", cr.CertDir))
	close(cr.certsMounted)
	}

I wonder if it check the tls.crt in the secret is better. And actually the caBundle which is injected in webhook is based on the secret, not the certFile. Or we should some sync logic if the caFile in the secret is different from the local File. I think the latter is better.

[Poor12]

cc @maxsmythe @adrianludwin

[adrianludwin]

This sounds like a fairly specialized case that cert-controller isn't really designed to handle. Is cert-manager able to handle that usecase? IMO, cert-controller is designed for fairly simple applications - we don't really have the expertise to develop multicluster (or out-of-cluster) features, let alone test them reliably to ensure those features keep working. But others might disagree with me.

…

On Sat, Apr 23, 2022 at 11:11 PM Tiecheng Shen ***@***.***> wrote: cc @maxsmythe <https://github.com/maxsmythe> @adrianludwin <https://github.com/adrianludwin> — Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE43PZBKNF2MQYXQFZ34LWDVGS3URANCNFSM5UFR3QDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[maxsmythe]

+1 @adrianludwin

cert-controller is mainly a way to avoid dependencies on 3rd party certificate management systems for dev/test/release and simple deployments. More complex cases will likely need to disable cert controller and figure out a solution that works for their specific use case.

WRT checking the contents of the secret...

The reason for checking the contents of the folder is because that is where the private key used by the webhook server is read from (it signs the webhook response, which is verified by the public key in the VWH config), so that folder being populated is necessary for the webhook to begin serving.

In this case, it's probably easiest to disable cert-controller and have something else handle generating certs and making sure the contents of the VWH config and webhook server secret folder are in sync (a K8s secret may not be necessary when running off-cluster, especially if there is nothing handling mounting the secret contents to a folder like there would be on K8s).

[Poor12]

Thanks for explanation. I will try to use another way to fix it and be glad to share some best practice where gatekeeper is used in multicluster or off-cluster.