Operator does not start properly on the first attempt after the upgrade

[Crumby]

Operator fails to start up properly first time after performing upgrade, giving tls: bad certificate messages. The cause of this issue is probably several updates to infinispan-operator-controller-manager-service-cert secret which happen during the update (with last update happening after the container starts).

While this issue is doesn't cause much harm in running deployments it messes up with upgrade tests timings resulting in failures. Operator container is marked as ready and CSV status phase marked as Successful which makes the issue hard to reasonably overcome in the test.

OpenShift versions: 4.10, 4.12

Test failure:

panic: Internal error occurred: failed calling webhook "minfinispan.kb.io": failed to call webhook: Post "https://infinispan-operator-controller-manager-service.openshift-operators.svc:443/mutate-infinispan-org-v1-infinispan?timeout=10s": service "infinispan-operator-controller-manager-service" not found [recovered]
	panic: Internal error occurred: failed calling webhook "minfinispan.kb.io": failed to call webhook: Post "https://infinispan-operator-controller-manager-service.openshift-operators.svc:443/mutate-infinispan-org-v1-infinispan?timeout=10s": service "infinispan-operator-controller-manager-service" not found

goroutine 231 [running]:
testing.tRunner.func1.2(0x160e680, 0xc0007c5b90)
	/usr/lib/golang/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc000890780)
	/usr/lib/golang/src/testing/testing.go:1146 +0x4b6
panic(0x160e680, 0xc0007c5b90)
	/usr/lib/golang/src/runtime/panic.go:965 +0x1b9
github.com/infinispan/infinispan-operator/test/e2e/utils.ExpectNoError(...)
	/home/pdrobek/git/infinispan-operator/test/e2e/utils/asserts.go:18
github.com/infinispan/infinispan-operator/test/e2e/upgrade.TestUpgrade(0xc000890780)
	/home/pdrobek/git/infinispan-operator/test/e2e/upgrade/upgrade_test.go:155 +0x14ab
testing.tRunner(0xc000890780, 0x18f4af8)
	/usr/lib/golang/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/lib/golang/src/testing/testing.go:1238 +0x2b3
FAIL	github.com/infinispan/infinispan-operator/test/e2e/upgrade	134.716s

Log:

2022-12-06T10:43:10.140Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
2022-12-06T10:43:10.140Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2022-12-06T10:43:10.141Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2022-12-06T10:43:10.141Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
2022-12-06T10:43:10.141Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2022/12/06 10:43:24 http: TLS handshake error from 10.129.0.99:35094: remote error: tls: bad certificate
2022/12/06 10:43:25 http: TLS handshake error from 10.129.0.99:35108: remote error: tls: bad certificate
2022/12/06 10:43:27 http: TLS handshake error from 10.129.0.99:35110: remote error: tls: bad certificate
2022/12/06 10:43:27 http: TLS handshake error from 10.129.0.99:35112: remote error: tls: bad certificate
2022/12/06 10:43:28 http: TLS handshake error from 10.129.0.99:35126: remote error: tls: bad certificate
2022/12/06 10:43:30 http: TLS handshake error from 10.129.0.99:35132: remote error: tls: bad certificate
2022/12/06 10:43:30 http: TLS handshake error from 10.129.0.99:35142: remote error: tls: bad certificate
2022/12/06 10:43:31 http: TLS handshake error from 10.129.0.99:55806: remote error: tls: bad certificate