opnfv/functest-kubernetes-security:v1.23 run_tests -t kube_bench_master

[sultetveny]

What happened?

I've faced with the following issue running the kubernetes security test using kube_bench_master.

podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/cluster-admin.pem:/home/opnfv/functest/cluster-admin.pem:Z \
-v ~/opnfv/cluster-admin-key.pem:/home/opnfv/functest/cluster-admin-key.pem:Z \
opnfv/functest-kubernetes-security:v1.23 /bin/bash

# then within the container
run_tests -t kube_bench_master

Test case failed. For more information please check attached file.
functest-kubernetes.debug.log

2023-02-24 14:10:38,039 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+----------------------------------------------------------+
|         ENV VAR         |                          VALUE                           |
+-------------------------+----------------------------------------------------------+
|         CI_LOOP         |                          daily                           |
|          DEBUG          |                           true                           |
|     DEPLOY_SCENARIO     |                 k8s-nosdn-nofeature-noha                 |
|      INSTALLER_TYPE     |                         unknown                          |
|        BUILD_TAG        |                                                          |
|        NODE_NAME        |                                                          |
|       TEST_DB_URL       |     http://testresults.opnfv.org/test/api/v1/results     |
|     TEST_DB_EXT_URL     |                                                          |
|     S3_ENDPOINT_URL     |                                                          |
|        S3_DST_URL       |                                                          |
|       HTTP_DST_URL      |                                                          |
+-------------------------+----------------------------------------------------------+

2023-02-24 14:10:38,049 - xtesting.ci.run_tests - INFO - Loading test case 'kube_bench_master'...
2023-02-24 14:10:38,418 - xtesting.ci.run_tests - INFO - Running test case 'kube_bench_master'...

2023-02-24 14:30:38,520 - xtesting.ci.run_tests - ERROR - 

Please fix the testcase kube_bench_master.
All exceptions should be caught by the testcase instead!

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/xtesting/ci/run_tests.py", line 171, in run_test
    test_case.run(**kwargs)
  File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 212, in run
    self.details["report"] = ast.literal_eval(self.pod_log)
  File "/usr/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 0
    
SyntaxError: unexpected EOF while parsing
2023-02-24 14:30:38,522 - xtesting.ci.run_tests - ERROR - The test case 'kube_bench_master' failed.
2023-02-24 14:30:38,522 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_ERROR

The content of the env file successfully read as you see below the env variables set within the container properly. However their values somehow not respected (e.g., NON_BLOCKING_TAINTS).
The K8S cluster contains 3 controllers, 2 workers, 1 edge node.

bash-5.1# printenv
HOSTNAME=947d26a68598
GCR_REPO=gcr-io.repo.cci.nokia.net
PWD=/
container=podman
HOME=/root
KUBE_TEST_REPO_LIST=/home/opnfv/functest/repositories.yml
DEPLOY_SCENARIO=k8s-nosdn-nofeature-noha
TERM=xterm
SHLVL=1
NON_BLOCKING_TAINTS=is_control,is_edge
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG=true
K8S_GCR_REPO=k8s-gcr-io.repo.cci.nokia.net
_=/bin/printenv
bash-5.1# 

kubectl get events -A --watch

NAMESPACE          LAST SEEN   TYPE      REASON             OBJECT                        MESSAGE
kube-bench-f4dqn   56s         Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.
kube-bench-f4dqn   57s         Normal    SuccessfulCreate   job/kube-bench-master         Created pod: kube-bench-master-t5l89
kube-bench-f4dqn   0s          Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.
kube-bench-f4dqn   0s          Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.

kubectl get pods -n kube-bench-f4dqn -o wide

NAME                      READY   STATUS    RESTARTS   AGE     IP       NODE     NOMINATED NODE   READINESS GATES
kube-bench-master-t5l89   0/1     Pending   0          2m53s   <none>   <none>   <none>           <none>

What did you expect to happen?

I expected the test case executed successfully.

How can we reproduce it (as minimally and precisely as possible)?

See above.

Anything else we need to know?

Please be aware I opened a ticket towards a different organization as I'm unsure who need to handle it.
aquasecurity/kube-bench#1389

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

Openstack.

OS version

# On Linux:
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

$ uname -a
Linux cbis-sut1-control-01 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sultetveny]

/sig security
/sig testing

[BenTheElder]

/kind support
/close

opnfv is not part of Kubernetes, this issue should be filed with https://github.com/opnfv/functest