Scylla Operator fails to run with sidecars

[samoscybercraig]

Describe the bug

When you have any sidecar containers in the operator pod, such as istio or envoy etc. The Operator will fail to start

To Reproduce
Steps to reproduce the behavior:

1. Add an additional container to the operator spec
2. Apply the operator spec via kubectl
3. Notice as the sidecar pod will starts but operator will show failed/error.

Expected behavior
Scylla Operator should start

Logs

{"L":"INFO","T":"2020-06-09T09:54:56.393Z","M":"Operator started","version":"0.1.6","build_date":"2020-04-30T08:16:18Z","commit":"9858a394f3f771a28f9cfe829a9a85e33ba0d4d3","built_by":"goreleaser","go_version":"go version go1.14.2 linux/amd64","options":{"Name":"scylla-operator-controller-manager-0","Namespace":"default","LogLevel":"info","Image":"","EnableAdmissionWebhook":true}}
{"L":"INFO","T":"2020-06-09T09:54:56.499Z","M":"Registering Components.","_trace_id":"XgST-rVZQjWqNrCKTk8QAA"}
{"L":"FATAL","T":"2020-06-09T09:54:56.551Z","M":"Operator Pod must have exactly 1 container","count":2,"_trace_id":"4NbUvpYIQCCvnh4t_buwCQ","S":"github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log.Logger.log\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log/logger.go:92\ngithub.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log.Logger.Fatal\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/scylladb/go-log/logger.go:81\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.getOperatorImage\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:153\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.newReconciler\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:65\ngithub.com/scylladb/scylla-operator/pkg/controller/cluster.Add\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/cluster/cluster_controller.go:49\ngithub.com/scylladb/scylla-operator/pkg/controller.addToManager\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/register.go:44\ngithub.com/scylladb/scylla-operator/pkg/controller.UseOperatorControllers\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/pkg/controller/register.go:38\nmain.newOperatorCmd.func1\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/cmd/operator.go:56\ngithub.com/scylladb/scylla-operator/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/spf13/cobra/command.go:766\ngithub.com/scylladb/scylla-operator/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/scylladb/scylla-operator/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/cmd/main.go:35\nruntime.main\n\t/home/henrik/go/src/github.com/scylladb/scylla-operator/bin/deps/go/src/runtime/proc.go:203"}

Environment:

• Platform: Minikube and Openshift
• Kubernetes version: v1.17.5
• Scylla version: 4.0.0 Agent 2.0.2
• Scylla-operator version: scylladb/scylla-operator:v0.1.6

Additional context
This is the problematic line

scylla-operator/pkg/controller/cluster/cluster_controller.go

Line 152 in 7c425ca

if len(pod.Spec.Containers) != 1 {

[yanniszark]

Argh, this is an artifact of the logic to get the operator's image in order to populate the operator sidecar (they share the same image).
While I had implemented it this way a while back to be completely hands-off for the user, I realize it's not the best way. Thankfully, I had also added an option to the operator to specify a custom image to inject as the operator sidecar. This is given by the --image cli arg (https://github.com/scylladb/scylla-operator/blob/master/cmd/options/operator.go#L25), which should let you work around this issue for now.

[dahankzter]

We can just remove this check right @yanniszark ? We need to ensure we select the correct image but that can maybe be done differently.

[samoscybercraig]

@yanniszark Unfortunately Istio sets its self up as a incoming webhook, so I am not manually injecting the container into the pod - its done automagically whenever a pod is deployed out and pulls the image and configuration for the sidecar, I could probably bypass or mark it not for webhook but then must configure the sidecar from scratch. My preference for the Scylla Operator would be to either have a flag/env which allows to set "additional" pods in the cluster, by default this would be 0, but for me this would 1, + 1 for scylla then compare against pod.spec