Operator with webhooks fail to deploy on 4.5, works on master

[jpkrohling]

Bug Report

As discussed in the operator-framework mailing list, I have an operator built using the Operator SDK 1.1.0 and webhooks, which is failing to be deployed using OLM 4.5, but works in master.

In order to get the operator deployed on clusters using OLM 4.5, fixes to those issues would need to be backported.

What did you do?

See the CONTRIBUTING.md from the jpkrohling/opentelemetry-operator repository, branch jpkrohling/reproduce-tls-problems-olm-4.5: https://github.com/jpkrohling/opentelemetry-operator/blob/jpkrohling/reproduce-tls-problems-olm-4.5/CONTRIBUTING.md#testing-it-with-olm-on-openshift

What did you expect to see?
The operator gets deployed

What did you see instead? Under which circumstances?
kubectl get events -n operators shows that the operator fails to deploy due to TLS issues. Messages are similar to this:

55s         Warning   Failed                pod/opentelemetry-operator-controller-manager-9bdd9fcc4-ctr8k     Error: cannot find volume "apiservice-cert" to mount into container "kube-rbac-proxy"
55s         Warning   Failed                pod/opentelemetry-operator-controller-manager-9bdd9fcc4-ctr8k     Error: cannot find volume "apiservice-cert" to mount into container "manager"

Environment

• operator-lifecycle-manager version: 4.5 fails, master works

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: minikube

Possible Solution
n/a

[jpkrohling]

cc @awgreene

[jpkrohling]

@awgreene do you know already whether this is going to be fixed for the next patch versions?

[jpkrohling]

@awgreene do you have news about this one?

[njhale]

@jpkrohling sorry for the delayed response here!

After some deliberation with the team, we've come to the conclusion that this is a backwards compatibility and/or docs issue with SDK.

• Webhook mount location is incompatible with older versions of OLM operator-sdk#4439
• Document Compatibility with OLM operator-sdk#4438

It makes sense for OLM to switch to the new location -- while still supporting the old -- in master and for new releases, but any attempt to backport a patch would amount to supporting forwards compatibility with SDK; something that doesn't seem feasible overall. Additionally, low-priority bug patches will not get cherry-pick approval for the foreseeable future (due to high bug volume).

While we wait for SDK's responses to these issues, I think it might be possible for you to add some logic at the operator level that falls back to the old cert locations if the new locations aren't populated.

new locations:

• /tmp/k8s-webhook-server/serving-certs/tls.cert
• /tmp/k8s-webhook-server/serving-certs/tls.key

old locations:

• /apiserver.local.config/certificates/apiserver.crt
• /apiserver.local.config/certificates/apiserver.key