Running in a kubernetes cluster don't work as expected when check cis-1.4-1.1.8

[mozillazg]

k8s version: 1.14.8
kube-bench version: latest (0.2.3)
run command: wget https://github.com/aquasecurity/kube-bench/blob/master/job-master.yaml && kubectl apply -f job-master.yaml
problem: --profiling=false is exists but this check is not passed

kube-bench/cfg/cis-1.4/master.yaml

Lines 120 to 134 in 77f6651

	- id: 1.1.8
	text: "Ensure that the --profiling argument is set to false (Scored)"
	audit: "/bin/ps -ef | grep $apiserverbin | grep -v grep"
	tests:
	test_items:
	- flag: "--profiling"
	compare:
	op: eq
	value: false
	set: true
	remediation: |
	Edit the API server pod specification file $apiserverconf
	on the master node and set the below parameter.
	--profiling=false
	scored: true

debug in k8s container:

/opt/kube-bench # kube-bench version
v0.2.3
/opt/kube-bench # kube-bench |grep 1.1.8
[FAIL] 1.1.8 Ensure that the --profiling argument is set to false (Scored)
1.1.8 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
/opt/kube-bench # /bin/ps -ef | grep apiserver | grep -v grep
root      3334  3305  2 Feb05 ?        01:10:07 kube-apiserver --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/log/kubernetes/kubernetes.audit --audit-log-maxage=7 --audit-policy-file=/etc/kubernetes/audit-policy.yml --apiserver-count=500 --endpoint-reconciler-type=lease --enable-aggregator-routing=true --runtime-config=admissionregistration.k8s.io/v1beta1 --profiling=false --advertise-address=...

When run kube-bench bin file in master node it check passed, but run in a k8s container it check failed

BTW, other check itemes which check it's flag are failed too(there are needed flag value but check failed like 1.1.8)

[mozillazg]

I have found the reason: the exec.Command will parse grep in audit to /usr/bin/grep which is mount from host, the /usr/bin/grep can't be run in container, so that all checks which use grep command will be failed :

kube-bench/check/check.go

Lines 233 to 235 in 77f6651

	cmd := exec.Command(cs[0], cs[1:]...)
	cmds = append(cmds, cmd)

/opt/kube-bench # ls /usr/bin/grep
/usr/bin/grep

/opt/kube-bench # /usr/bin/grep
/bin/sh: /usr/bin/grep: not found

Maybe we should change the value of PATH in Dockerfile and move /bin to first, or add this issue to README to remind user of don't mount /usr/bin to container when running in a kubernetes cluster.

[lizrice]

The user may want to mount /usr/bin to get access to the kubectl/kubelet binaries for version detection (if they don't want to specify the version explicitly), so we shouldn't change that advice to the user.

I am intrigued about why /usr/bin/grep shows up when you run ls in the container, but you can't execute it. What do you get from ls -l /usr/bin/grep?

I am open to the idea of changing the path to PATH to run the grep from inside the container. I have some recollection of an issue in the past due to different grep versions but if we always ran the version from the container it would give more consistency. Wdyt @robertojrojas ?

[robertojrojas]

@mozillazg good catch! and thanks for reporting this.

@lizrice I think this issue is same as this #501 and, to be consistent, we should fix it in the same manner as #508

@mozillazg would you like to submit a PR similar to #508? :)

[mozillazg]

@robertojrojas Yes, I'll submit a PR later.

IMHO, Maybe the idea of changing the path to PATH is better, in case of there are similar cases in feature or maybe there are other commands(for example: stat) provided by some user host OS(the user host is not under control but the docker image is) that can't be run in container(because of same reason as #501).

[robertojrojas]

@mozillazg I'm not completely comfortable of changing the PATH as it would be difficult to predict what side effects it could bring with existing folks using the software at the moment. The absolute path is safer.

In the case of stat, it will not be fixed completely by either changing the PATH nor providing an absolute path to it. This is because, based on my research, on Ubuntu 18+ stat resides on /usr/bin/stat only, but on RHEL/CentOS 7+, it resides on /bin/stat and /usr/bin/stat
So in one OS the PATH change will work, but on another the full path will.

[mozillazg]

@robertojrojas 👌

[mozillazg]

After testing I found that use absolute path can't fix same problem of xargs and awk (They are in /usr/bin but has been overwrited by mount from host). Maybe we need think again about the idea of change the PATH? :

• Change PATH from default value /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin to /usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin or $PATH:/bin_from_host via Dockerfile. BTW, the dir /usr/local/sbin in alpine is not exist.
• Mount /usr/bin to /usr/local/sbin or /bin_from_host.

Use this way we can both use all builtin commands from container and other needed commands from host(e.g. kubeclt), there is no longer surprise for where is the command come from and why the command is run failed.

[robertojrojas]

@mozillazg we agree with changing the PATH. Would you be able to work on that today? If not, it's not a problem. I can take care of it.