Can kube-bench run pod file system?

[Algoss]

Overview

I am running kube-bench inside a pod in k8s cluster. Components like kube-proxy is running as a pod. Now kube-bench is not able to read kube-proxy file system to read config files to scan this.

How did you run kube-bench?

I am running kube-bench inside a pod in k8s cluster via kube-bench job creation.

What happened?

I1121 06:45:15.595459 19013 util.go:486] Checking for oc I1121 06:45:15.595578 19013 util.go:515] Can't find oc command: exec: "oc": executable file not found in $PATH I1121 06:45:15.595594 19013 kubernetes_version.go:36] Try to get version from Rest API I1121 06:45:15.595637 19013 kubernetes_version.go:161] Loading CA certificate I1121 06:45:15.595657 19013 kubernetes_version.go:115] getWebData srvURL: https://kubernetes.default.svc/version I1121 06:45:15.600934 19013 kubernetes_version.go:100] vd: { "major": "1", "minor": "19", "gitVersion": "v1.19.16", "gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074", "gitTreeState": "clean", "buildDate": "2021-10-27T16:20:18Z", "goVersion": "go1.15.15", "compiler": "gc", "platform": "linux/amd64" } I1121 06:45:15.600992 19013 kubernetes_version.go:105] vrObj: &cmd.VersionResponse{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} I1121 06:45:15.601003 19013 util.go:293] Kubernetes REST API Reported version: &{1 19 v1.19.16} I1121 06:45:15.601030 19013 kubernetes_version.go:36] Try to get version from Rest API I1121 06:45:15.601060 19013 kubernetes_version.go:161] Loading CA certificate I1121 06:45:15.601072 19013 kubernetes_version.go:115] getWebData srvURL: https://kubernetes.default.svc/version I1121 06:45:15.606164 19013 kubernetes_version.go:100] vd: { "major": "1", "minor": "19", "gitVersion": "v1.19.16", "gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074", "gitTreeState": "clean", "buildDate": "2021-10-27T16:20:18Z", "goVersion": "go1.15.15", "compiler": "gc", "platform": "linux/amd64" } I1121 06:45:15.606210 19013 kubernetes_version.go:105] vrObj: &cmd.VersionResponse{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} I1121 06:45:15.606228 19013 util.go:293] Kubernetes REST API Reported version: &{1 19 v1.19.16} I1121 06:45:15.606263 19013 common.go:281] mapToBenchmarkVersion for k8sVersion: "1.19" cisVersion: "cis-1.20" found: true I1121 06:45:15.606270 19013 common.go:347] Mapped Kubernetes version: 1.19 to Benchmark version: cis-1.20 I1121 06:45:15.606277 19013 common.go:350] Kubernetes version: "1.19" to Benchmark version: "cis-1.20" I1121 06:45:15.606283 19013 run.go:40] Checking targets [node] for cis-1.20 I1121 06:45:15.606400 19013 common.go:273] Using config file: cfg/cis-1.20/config.yaml I1121 06:45:15.606414 19013 run.go:75] Running tests from files [cfg/cis-1.20/node.yaml] I1121 06:45:15.606443 19013 common.go:79] Using test file: cfg/cis-1.20/node.yaml I1121 06:45:15.606471 19013 util.go:79] ps - proc: "hyperkube" I1121 06:45:15.626663 19013 util.go:83] [/bin/ps -C hyperkube -o cmd --no-headers]: exit status 1 I1121 06:45:15.626672 19013 util.go:86] ps - returning: "" I1121 06:45:15.626701 19013 util.go:227] reFirstWord.Match() I1121 06:45:15.626707 19013 util.go:257] executable 'hyperkube kubelet' not running I1121 06:45:15.626710 19013 util.go:79] ps - proc: "kubelet" I1121 06:45:15.647047 19013 util.go:86] ps - returning: "/export/apps/kubernetes-kubelet/bin/kubelet --config=/etc/kubernetes-kubelet/kubelet_config.yaml --kubeconfig=/etc/kubernetes-kubelet/kubeconfig --container-runtime=remote --network-plugin=cni --root-dir=/export/content/data/kubelet/kubelet-root --container-runtime-endpoint=unix:///run/containerd/containerd.sock --v=3\n" I1121 06:45:15.647068 19013 util.go:227] reFirstWord.Match(/export/apps/kubernetes-kubelet/bin/kubelet --config=/etc/kubernetes-kubelet/kubelet_config.yaml --kubeconfig=/etc/kubernetes-kubelet/kubeconfig --container-runtime=remote --network-plugin=cni --root-dir=/export/content/data/kubelet/kubelet-root --container-runtime-endpoint=unix:///run/containerd/containerd.sock --v=3) I1121 06:45:15.647096 19013 util.go:115] Component kubelet uses running binary kubelet I1121 06:45:15.647118 19013 util.go:79] ps - proc: "kube-proxy" I1121 06:45:15.665792 19013 util.go:86] ps - returning: "/kubeproxy/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=ltx1-app4710.stg.linkedin.com --v=1\n" I1121 06:45:15.665814 19013 util.go:227] reFirstWord.Match(/kubeproxy/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=ltx1-app4710.stg.linkedin.com --v=1) I1121 06:45:15.665827 19013 util.go:115] Component proxy uses running binary kube-proxy I1121 06:45:15.665860 19013 util.go:200] Component kubelet uses config file '/etc/kubernetes-kubelet/kubelet_config.yaml' I1121 06:45:15.665895 19013 util.go:193] Using default config file name '/etc/kubernetes/addons/kube-proxy-daemonset.yaml' for component proxy I1121 06:45:15.665912 19013 util.go:193] Using default config file name '/etc/kubernetes/config' for component kubernetes I1121 06:45:15.665930 19013 util.go:200] Component kubelet uses service file '/etc/systemd/system/kubelet.service' I1121 06:45:15.665944 19013 util.go:196] Missing service file for proxy I1121 06:45:15.665954 19013 util.go:196] Missing service file for kubernetes I1121 06:45:15.665978 19013 util.go:193] Using default kubeconfig file name '/etc/kubernetes/kubelet.conf' for component kubelet I1121 06:45:15.665994 19013 util.go:193] Using default kubeconfig file name '/etc/kubernetes/proxy.conf' for component proxy I1121 06:45:15.666004 19013 util.go:196] Missing kubeconfig file for kubernetes I1121 06:45:15.666018 19013 util.go:193] Using default ca file name '/etc/kubernetes/pki/ca.crt' for component kubelet I1121 06:45:15.666035 19013 util.go:196] Missing ca file for proxy I1121 06:45:15.666045 19013 util.go:196] Missing ca file for kubernetes I1121 06:45:15.666062 19013 util.go:387] Substituting $kubeletbin with 'kubelet' I1121 06:45:15.666077 19013 util.go:387] Substituting $proxybin with 'kube-proxy' I1121 06:45:15.666084 19013 util.go:387] Substituting $proxyconf with '/etc/kubernetes/addons/kube-proxy-daemonset.yaml' I1121 06:45:15.666091 19013 util.go:387] Substituting $kubernetesconf with '/etc/kubernetes/config' I1121 06:45:15.666097 19013 util.go:387] Substituting $kubeletconf with '/etc/kubernetes-kubelet/kubelet_config.yaml' I1121 06:45:15.666116 19013 util.go:387] Substituting $kubeletsvc with '/etc/systemd/system/kubelet.service' I1121 06:45:15.666132 19013 util.go:387] Substituting $proxysvc with 'proxy' I1121 06:45:15.666139 19013 util.go:387] Substituting $kubernetessvc with 'kubernetes' I1121 06:45:15.666148 19013 util.go:387] Substituting $kubeletkubeconfig with '/etc/kubernetes/kubelet.conf' I1121 06:45:15.666165 19013 util.go:387] Substituting $proxykubeconfig with '/etc/kubernetes/proxy.conf' I1121 06:45:15.666191 19013 util.go:387] Substituting $kuberneteskubeconfig with 'kubernetes' I1121 06:45:15.666197 19013 util.go:387] Substituting $kubeletcafile with '/etc/kubernetes/pki/ca.crt' I1121 06:45:15.666211 19013 util.go:387] Substituting $proxycafile with 'proxy' I1121 06:45:15.666220 19013 util.go:387] Substituting $kubernetescafile with 'kubernetes'

What did you expect to happen:

I expect kube-bench to read correct config files in kube-proxy pod file system.

Environment

k8s version: 1.19

Running processes

[Please include the output from running ps -eaf | grep kube on the affected node. This will allow us to check what Kubernetes processes are running, and how this compares to what kube-bench detected.]

Configuration files

`---

Controls Files.

These are YAML files that hold all the details for running checks.

Uncomment to use different control file paths.

masterControls: ./cfg/master.yaml

nodeControls: ./cfg/node.yaml

master:
components:
- apiserver
- scheduler
- controllermanager
- etcd
- flanneld
# kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark
- kubernetes
- kubelet

kubernetes:
defaultconf: /etc/kubernetes/config

apiserver:
bins:
- "kube-apiserver"
- "hyperkube apiserver"
- "hyperkube kube-apiserver"
- "apiserver"
- "openshift start master api"
- "hypershift openshift-kube-apiserver"
confs:
- /etc/kubernetes/manifests/kube-apiserver.yaml
- /etc/kubernetes/manifests/kube-apiserver.yml
- /etc/kubernetes/manifests/kube-apiserver.manifest
- /var/snap/kube-apiserver/current/args
- /var/snap/microk8s/current/args/kube-apiserver
- /etc/origin/master/master-config.yaml
- /etc/kubernetes/manifests/talos-kube-apiserver.yaml
defaultconf: /etc/kubernetes/manifests/kube-apiserver.yaml

scheduler:
bins:
- "kube-scheduler"
- "hyperkube scheduler"
- "hyperkube kube-scheduler"
- "scheduler"
- "openshift start master controllers"
confs:
- /export/content/lid/apps/scheduler/i001/var/schedulerconfig
- /etc/kubernetes/manifests/kube-scheduler.yaml
- /etc/kubernetes/manifests/kube-scheduler.yml
- /etc/kubernetes/manifests/kube-scheduler.manifest
- /var/snap/kube-scheduler/current/args
- /var/snap/microk8s/current/args/kube-scheduler
- /etc/origin/master/scheduler.json
- /etc/kubernetes/manifests/talos-kube-scheduler.yaml
defaultconf: /etc/kubernetes/manifests/kube-scheduler.yaml
kubeconfig:
- /etc/kubernetes/scheduler.conf
- /var/lib/kube-scheduler/kubeconfig
- /var/lib/kube-scheduler/config.yaml
- /system/secrets/kubernetes/kube-scheduler/kubeconfig
defaultkubeconfig: /etc/kubernetes/scheduler.conf

controllermanager:
bins:
- "kube-controller-manager"
- "kube-controller"
- "hyperkube controller-manager"
- "hyperkube kube-controller-manager"
- "controller-manager"
- "openshift start master controllers"
- "hypershift openshift-controller-manager"
confs:
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-controller-manager.yml
- /etc/kubernetes/manifests/kube-controller-manager.manifest
- /var/snap/kube-controller-manager/current/args
- /var/snap/microk8s/current/args/kube-controller-manager
- /etc/kubernetes/manifests/talos-kube-controller-manager.yaml
defaultconf: /etc/kubernetes/manifests/kube-controller-manager.yaml
kubeconfig:
- /etc/kubernetes/controller-manager.conf
- /var/lib/kube-controller-manager/kubeconfig
- /system/secrets/kubernetes/kube-controller-manager/kubeconfig
defaultkubeconfig: /etc/kubernetes/controller-manager.conf

etcd:
optional: true
bins:
- "etcd"
- "openshift start etcd"
confs:
- /etc/kubernetes/manifests/etcd.yaml
- /etc/kubernetes/manifests/etcd.yml
- /etc/kubernetes/manifests/etcd.manifest
- /etc/etcd/etcd.conf
- /var/snap/etcd/common/etcd.conf.yml
- /var/snap/etcd/common/etcd.conf.yaml
- /var/snap/microk8s/current/args/etcd
- /usr/lib/systemd/system/etcd.service
defaultconf: /etc/kubernetes/manifests/etcd.yaml

flanneld:
optional: true
bins:
- flanneld
defaultconf: /etc/sysconfig/flanneld

kubelet:
optional: true
bins:
- "hyperkube kubelet"
- "kubelet"

node:
components:
- kubelet
- proxy
# kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark
- kubernetes

kubernetes:
defaultconf: "/etc/kubernetes/config"

kubelet:
cafile:
- "/etc/kubernetes-kubelet/identity.cert"
- "/etc/kubernetes/pki/ca.crt"
- "/etc/kubernetes/certs/ca.crt"
- "/etc/kubernetes/cert/ca.pem"
- "/var/snap/microk8s/current/certs/ca.crt"
svc:
# These paths must also be included
# in the 'confs' property below
- "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
- "/etc/systemd/system/kubelet.service"
- "/lib/systemd/system/kubelet.service"
- "/etc/systemd/system/snap.kubelet.daemon.service"
- "/etc/systemd/system/snap.microk8s.daemon-kubelet.service"
- "/etc/systemd/system/atomic-openshift-node.service"
- "/etc/systemd/system/origin-node.service"
bins:
- "hyperkube kubelet"
- "kubelet"
kubeconfig:
- "/etc/kubernetes-kubelet/kubeconfig"
- "/etc/kubernetes/kubelet.conf"
- "/etc/kubernetes/kubelet-kubeconfig.conf"
- "/var/lib/kubelet/kubeconfig"
- "/etc/kubernetes/kubelet-kubeconfig"
- "/etc/kubernetes/kubelet/kubeconfig"
- "/var/snap/microk8s/current/credentials/kubelet.config"
- "/etc/kubernetes/kubeconfig-kubelet"
confs:
- "/etc/kubernetes-kubelet/kubelet_config.yaml"
- "/etc/kubernetes/kubelet-config.yaml"
- "/var/lib/kubelet/config.yaml"
- "/var/lib/kubelet/config.yml"
- "/etc/kubernetes/kubelet/kubelet-config.json"
- "/etc/kubernetes/kubelet/config"
- "/home/kubernetes/kubelet-config.yaml"
- "/home/kubernetes/kubelet-config.yml"
- "/etc/default/kubeletconfig.json"
- "/etc/default/kubelet"
- "/var/lib/kubelet/kubeconfig"
- "/var/snap/kubelet/current/args"
- "/var/snap/microk8s/current/args/kubelet"
## Due to the fact that the kubelet might be configured
## without a kubelet-config file, we use a work-around
## of pointing to the systemd service file (which can also
## hold kubelet configuration).
## Note: The following paths must match the one under 'svc'
- "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
- "/etc/systemd/system/kubelet.service"
- "/lib/systemd/system/kubelet.service"
- "/etc/systemd/system/snap.kubelet.daemon.service"
- "/etc/systemd/system/snap.microk8s.daemon-kubelet.service"
- "/etc/kubernetes/kubelet.yaml"
defaultconf: "/var/lib/kubelet/config.yaml"
defaultsvc: "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
defaultkubeconfig: "/etc/kubernetes/kubelet.conf"
defaultcafile: "/etc/kubernetes/pki/ca.crt"

proxy:
optional: true
bins:
- "kube-proxy"
- "hyperkube proxy"
- "hyperkube kube-proxy"
- "proxy"
- "openshift start network"
confs:
- /var/lib/kube-proxy/config.conf
- /etc/kubernetes/proxy
- /etc/kubernetes/addons/kube-proxy-daemonset.yaml
- /etc/kubernetes/addons/kube-proxy-daemonset.yml
- /var/snap/kube-proxy/current/args
- /var/snap/microk8s/current/args/kube-proxy
kubeconfig:
- "/var/lib/kube-proxy/config.conf"
- "/etc/kubernetes/kubelet-kubeconfig"
- "/etc/kubernetes/kubelet-kubeconfig.conf"
- "/etc/kubernetes/kubelet/config"
- "/var/lib/kubelet/kubeconfig"
- "/var/snap/microk8s/current/credentials/proxy.config"
svc:
- "/lib/systemd/system/kube-proxy.service"
- "/etc/systemd/system/snap.microk8s.daemon-proxy.service"
defaultconf: /etc/kubernetes/addons/kube-proxy-daemonset.yaml
defaultkubeconfig: "/etc/kubernetes/proxy.conf"

etcd:
components:
- etcd

etcd:
bins:
- "etcd"
confs:
- /etc/kubernetes/manifests/etcd.yaml
- /etc/kubernetes/manifests/etcd.yml
- /etc/kubernetes/manifests/etcd.manifest
- /etc/etcd/etcd.conf
- /var/snap/etcd/common/etcd.conf.yml
- /var/snap/etcd/common/etcd.conf.yaml
- /var/snap/microk8s/current/args/etcd
- /usr/lib/systemd/system/etcd.service
defaultconf: /etc/kubernetes/manifests/etcd.yaml

controlplane:
components:
- apiserver

apiserver:
bins:
- "kube-apiserver"
- "hyperkube apiserver"
- "hyperkube kube-apiserver"
- "apiserver"

policies:
components: []

managedservices:
components: []

version_mapping:
"1.15": "cis-1.5"
"1.16": "cis-1.6"
"1.17": "cis-1.6"
"1.18": "cis-1.6"
"1.19": "cis-1.20"
"1.20": "cis-1.20"
"1.21": "cis-1.20"
"1.22": "cis-1.23"
"1.23": "cis-1.23"
"eks-1.0.1": "eks-1.0.1"
"eks-1.1.0": "eks-1.1.0"
"gke-1.0": "gke-1.0"
"gke-1.2.0": "gke-1.2.0"
"ocp-3.10": "rh-0.7"
"ocp-3.11": "rh-0.7"
"ocp-4.0": "rh-1.0"
"aks-1.0": "aks-1.0"
"ack-1.0": "ack-1.0"
"cis-1.6-k3s": "cis-1.6-k3s"

target_mapping:
"cis-1.5":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.6":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.6-k3s":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.20":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.23":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"gke-1.0":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
- "managedservices"
"gke-1.2.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"eks-1.0.1":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"eks-1.1.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"rh-0.7":
- "master"
- "node"
"aks-1.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"ack-1.0":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
- "managedservices"
"rh-1.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "etcd"
"eks-stig-kubernetes-v1r6":
- "node"
- "controlplane"
- "policies"
- "managedservices"`

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]