Openshift 3.10: need apiserver executable but none of the candidates are running

[skam-github]

Hi All,

I am facing a similar issue for the OpenShift Container Platform version 3.10.

Issue 1:
[root@user kube-bench]# ./kube-bench master
need apiserver executable but none of the candidates are running

Issue 2:
[root@user1 kube-bench]# ./kube-bench node

need proxy executable but none of the candidates are running

I tried checking for apiserver and response is:
ps -ef | grep apiserver
root 45678 910112 0 06:01 pts/1 00:00:00 grep --color=auto apiserver

Then I tried getting all the pods namespaces and response is:
oc get pods --all-namespaces
Output
default docker-registry-1-4qq 1/1 Running 0 2d
default docker-registry-2-deploy 0/1 Error 0 2d
default kube-bench-master 0/1 Pending 0 5h
default master 0/1 Pending 0 3h
default registry-console-1-479 1/1 Running 1 2d
default router-1-d7zdg 1/1 Running 0 2d
default router-1-rhg2m 1/1 Running 0 2d
default router-1-vz45m 1/1 Running 0 2d
kube-system kube-storage-controller-doryd-7c8c6d5dc-5fkjg 1/1 Running 0 2d
kube-system master-api-user1.something.local 1/1 Running 1 2d
kube-system master-api-user2.something.local 1/1 Running 0 2d
kube-system master-api-user3.something.local 1/1 Running 2 2d
kube-system master-controllers-user1.something.local 1/1 Running 1 2d
kube-system master-controllers-user2.something.local 1/1 Running 0 2d
kube-system master-controllers-user3.something.local 1/1 Running 2 2d
openshift-node sync-AAAAA 1/1 Running 0 2d
openshift-node sync-BBBBB 1/1 Running 1 2d
openshift-node sync-CCCCC 1/1 Running 0 2d
openshift-sdn sdn-DDDDD 1/1 Running 2 2d
openshift-sdn sdn-EEEEE 1/1 Running 0 2d
openshift-sdn sdn-FFFFF 1/1 Running 0 2d
openshift-web-console webconsole-6ff6ff-fhrhb 1/1 Running 1 2d
openshift-web-console webconsole-6ff6ff-tdd42 1/1 Running 1 2d
openshift-web-console webconsole-6ff6ff-tflz6 1/1 Running 0 2d

Then oc status returns me:
command: oc status
In project default on server https://user1.something.local:8443

https://docker-registry-default.router.default.svc.cluster.local (passthrough) (svc/docker-registry)
dc/docker-registry deploys aaa.aaa.aaaa/openshift3/ose-docker-registry:v3.10.111
deployment #2 failed 2 days ago: config change
deployment #1 deployed 2 days ago - 1 pod

svc/kubernetes - XXX.XX.X.X ports 443->8443, 53->8053, 53->8053

https://registry-console-default.router.default.svc.cluster.local (passthrough) (svc/registry-console)
dc/registry-console deploys aaa.aaa.aaaa/openshift3/registry-console:v3.10
deployment #1 deployed 2 days ago - 1 pod

svc/router - YYY.YY.YY.Y ports 80, 443, 1936
dc/router deploys registry.access.redhat.com/openshift3/ose-haproxy-router:v3.10.111
deployment #1 deployed 2 days ago - 3 pods

pod/master runs aquasec/kube-bench:latest

pod/kube-bench-master runs aquasec/kube-bench:latest

Then checked kubectl up and running
kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b81c8f8", GitCommit:"b81c8f8", GitTreeState:"clean", BuildDate:"2019-02-07T18:49:53Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"linux/amd64"}

Please let me know how to proceed in resolving this issue.

Originally posted by @skam-github in #136 (comment)

[lizrice]

@skam-github for OpenShift at the moment you'll need to explicitly specify --version ocp-3.10 to pick up configuration which includes the executables that OpenShift uses. (We are hoping to auto-detect this at some point.)

[skam-github]

Thanks for your inputs.
With this flag I was able to run "kube-bench" on node but not on master.
For master I get the same error as "need apiserver executable but none of the candidates are running"
I realized in our setup of OCP we are not running any command related to kubenetes like "kube-apiserver", so these will not be found by the "kube-bench".

It will be really helpful if you can guide me in running "kube-bench" for "ocp" master

[lizrice]

@skam-github good news that it's working on the node at least! On the master, what do you see if you run ps -eaf | grep apiserver ?

We think that for OCP this should include hypershift openshift-kube-apiserver.

[skam-github]

@skam-github good news that it's working on the node at least! On the master, what do you see if you run ps -eaf | grep apiserver ?

We think that for OCP this should include hypershift openshift-kube-apiserver.

Hi,

Thanks again for your response.
In my case, ps -ef | grep apiserver results in:
root 45678 910112 0 06:01 pts/1 00:00:00 grep --color=auto apiserver

Thanks in advance, your inputs will help us use kube-bench efficiently

[ttousai]

@skam-github we have a recurring problem with the openshift apiserver component binary name. To assist us please share to help in troubleshooting:

• the output of oc get nodes
• the output of ps -eaf

[bgoareguer]

In my case the API server is run as:
openshift start master api --config=/etc/origin/master/master-config.yaml

[skam-github]

@skam-github we have a recurring problem with the openshift apiserver component binary name. To assist us please share to help in troubleshooting:

• the output of oc get nodes
• the output of ps -eaf

Hi,

Currently I have OpenShift 3.11 up and running.
I installed kube-bench successfully on the master node.
Before running kube-bench, I just checked I dont have "hypershift" in my environment,
Instead I have "oc". So I can use oc to execute something related to openshift in my environment
Then I ran following commands on the master node and output is listed below:

Command 1: ./kube-bench master --version ocp-3.10
Output 1: need apiserver executable but none of the candidates are running

Command 2: oc get nodes
Output 2:
NAME STATUS ROLES AGE VERSION
infranode01.net.local Ready infra 33d v1.11.0+d4cacc0
infranode02.net.local NotReady infra 33d v1.11.0+d4cacc0
infranode03.net.local Ready infra 33d v1.11.0+d4cacc0
master01.net.local Ready master 33d v1.11.0+d4cacc0
master02.net.local Ready master 33d v1.11.0+d4cacc0
master03.net.local Ready master 33d v1.11.0+d4cacc0
worker01.net.local Ready compute 33d v1.11.0+d4cacc0
worker02.net.local Ready compute 33d v1.11.0+d4cacc0
worker03.net.local Ready compute 33d v1.11.0+d4cacc0

Command 3: ps -eaf
Output 3: Please refer to attached file for ouput of this command

kubebench-ps-eaf.txt

Please share your inputs, to help me make progress.
If the team is available for chat/call I can show my environment for details.

[Gigilamalice]

Hi,
I have the same problem

Command 1: ./kube-bench master --version ocp-3.10
Output 1: need apiserver executable but none of the candidates are running

Command 2: oc get nodes
Output 2:
NAME STATUS ROLES AGE VERSION
infranode1.5eda.internal Ready infra 1d v1.11.0+d4cacc0
infranode2.5eda.internal Ready infra 1d v1.11.0+d4cacc0
master1.5eda.internal Ready master 1d v1.11.0+d4cacc0
master2.5eda.internal Ready master 1d v1.11.0+d4cacc0
master3.5eda.internal Ready master 1d v1.11.0+d4cacc0
node1.5eda.internal Ready compute 1d v1.11.0+d4cacc0
node2.5eda.internal Ready compute 1d v1.11.0+d4cacc0
node3.5eda.internal Ready compute 1d v1.11.0+d4cacc0

Thanks for your help.

[bgoareguer]

In Openshift 3.11 the API server is run as follows (seen in the output of your ps command):
openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=2

Here is the content of the ocp-3.10-config.yaml I used in order to get kubebench to work against an Openshift 3.11 cluster:

---
## Controls Files.
# These are YAML files that hold all the details for running checks.
#
## Uncomment to use different control file paths.
# masterControls: ./cfg/master.yaml
# nodeControls: ./cfg/node.yaml
# federatedControls: ./cfg/federated.yaml

master:
  apiserver:
    bins:
      - openshift start master api

  scheduler:
    bins:
      - "openshift start master controllers"
    confs:
      - /etc/origin/master/scheduler.json

  controllermanager:
    bins:
      - "openshift start master controllers"

  etcd:
    bins:
      - openshift start etcd

node:
  proxy:
    bins:
      - openshift start network

[Gigilamalice]

Thanks bgoareguer, it's working for me now.

[ttousai]

Closing this as resolved.

[ykfq]

Its maybe because you were not on master node, or your master node was managed by the kubernetes vendor.