operator installed via OLM cannot set kiali server pod securityContext.readOnlyRootFilesystem to false

[jmazzitelli]

When the operator is installed via OLM, the kiali pod's securityContext.readOnlyRootFilesystem cannot be set to false. Even though the Kiali CR has it set to false, that never makes its way to the pod yaml - it remains true. We have a molecule test that tests this, which is how I noticed it. This test has been showing this issue for a while now - but the molecule tests must be run with OLM for the failure to be seen.

The failure is in the config-values-test molecule test:

TASK [Make sure the security context was overridden and as expected] ***********
fatal: [localhost]: FAILED! => {
    "assertion": "kiali_pod_spec.containers[0].securityContext.readOnlyRootFilesystem == false",

To see the failure:

hack/ci-kind-molecule-tests.sh --kind-exe $(which kind) --client-exe $(which kubectl) -dorp podman --upload-logs false --logs-branch kind -ir "" --rebuild-cluster true --olm-enabled true --operator-installer skip -at config-values-test

[jmazzitelli]

Another way to see this fail without using that ci script:

1. Get a cluster up and install Istio (e.g. hack/k8s-minikube.sh start && hack/k8s-minikube.sh istio)
2. Install OLM: make -e CLUSTER_TYPE=minikube olm-install
3. Install the Kiali operator via OLM: kubectl create -f https://operatorhub.io/install/stable/kiali.yaml
4. Wait for the operator to start: kubectl wait -n operators deploy/kiali-operator --for=condition=Available=True --timeout=5m
5. Allow the operator to do things that are required by the tests (**this is the problem - see followup comment):

for env_name in ALLOW_AD_HOC_KIALI_NAMESPACE ALLOW_AD_HOC_KIALI_IMAGE; do kubectl -n operators patch $(kubectl -n operators get csv -o name | grep kiali) --type=json -p "[{'op':'replace','path':"/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/$(kubectl -n operators get $(kubectl -n operators get csv -o name | grep kiali) -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env[*].name}' | tr ' ' '\n' | cat --number | grep ${env_name} | cut -f 1 | xargs echo -n | cat - <(echo "-1") | bc)/value",'value':"\"true\""}]"; done

6. Run the test: hack/run-molecule-tests.sh --client-exe "$(which kubectl)" -dorp "podman" --cluster-type "minikube" --operator-installer "skip" --minikube-exe "$(which minikube)" --all-tests config-values-test

Results in the log file show:

TASK [Make sure the security context was overridden and as expected] ***********
fatal: [localhost]: FAILED! => {
    "assertion": "kiali_pod_spec.containers[0].securityContext.readOnlyRootFilesystem == false",
    "changed": false,
    "evaluated_to": false,
    "msg": "Assertion failed"
}

NOTE: to run the test locally (outside of any podman container), you can run make -e DORP="podman" -e CLUSTER_TYPE=minikube -e MOLECULE_OPERATOR_INSTALLER=skip -e MOLECULE_SCENARIO=config-values-test molecule-test

If you have a kiali pod, you can get the readOnlyRootFilesystem setting via:
kubectl get pods -n istio-system -l app.kubernetes.io/name=kiali -o jsonpath='{.items..spec.containers[0].securityContext.readOnlyRootFilesystem}{"\n"}'

[jmazzitelli]

The problem is with test setup. In the past we added the env var ALLOW_SECURITY_CONTEXT_OVERRIDE to the operator which defaults to false as a security measure. We need to set this to true in the OLM-installed operator because our test tries to set the security context. Because the operator doesn't allow that if that env var is false, the test fails.

So this command in my replication procedures in the previous comment needs to add that env var:

for env_name in ALLOW_AD_HOC_KIALI_NAMESPACE ALLOW_AD_HOC_KIALI_IMAGE ALLOW_SECURITY_CONTEXT_OVERRIDE; do kubectl -n operators patch $(kubectl -n operators get csv -o name | grep kiali) --type=json -p "[{'op':'replace','path':"/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/$(kubectl -n operators get $(kubectl -n operators get csv -o name | grep kiali) -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env[*].name}' | tr ' ' '\n' | cat --number | grep ${env_name} | cut -f 1 | xargs echo -n | cat - <(echo "-1") | bc)/value",'value':"\"true\""}]"; done

This needs to be added to the hack/ci-kind-molecule-tests.sh test script.