Failed to fetch eligible deployments for namespace whereas the namespace exists

[sudarshntn]

I0221 03:49:36.201590 7 kubemonkey.go:24] Status Update: Generating next schedule at 2018-02-22 00:00:00 +0530 IST
I0221 18:30:00.000194 7 schedule.go:54] Status Update: Generating schedule for terminations
W0221 18:30:00.015033 7 factory.go:44] Failed to fetch eligible deployments for namespace cam-km due to error: User "system:serviceaccount:cam-km:default" cannot list deployments.extensions in the namespace "cam-km". (get deployments.extensions)
I0221 18:30:00.015728 7 schedule.go:47] Status Update: 0 terminations scheduled today
I0221 18:30:00.015775 7 kubemonkey.go:63] Status Update: Waiting to run scheduled terminations.
I0221 18:30:00.015789 7 kubemonkey.go:77] Status Update: All terminations done.
I0221 18:30:00.015892 7 kubemonkey.go:24] Status Update: Generating next schedule at 2018-02-23 00:00:00 +0530 IST
********** Today's schedule **********
No terminations scheduled
********** End of schedule **********

[asobti]

Appears to be a permissions issue. What namespace is kube-monkey running under and what permissions does it have?

[sudarshntn]

The kube-monkey app is also running in the same namespace. We have our own namespace called cam-km. The config details in the configmap is as below:

apiVersion: v1
kind: ConfigMap
metadata:
name: kube-monkey-config-map
namespace: cam-km
data:
config.toml: |
[kubemonkey]
# Terminations are only logged
dry_run = true
# Run scheduling at 8am on weekdays
run_hour = 0
# Don't schedule any pod deaths before 10am
start_hour = 1
# Don't schedule any pod deaths after 4pm
end_hour = 22
# The amount of time in seconds a pod is given to shut down gracefully
GracePeriodSec = 5
# Critical apps live here
blacklisted_namespaces = ["kube-system","kube-public","ibm-system","ibm-cert-store","cam-km-core","default"]
whitelisted_namespaces = ["cam-km"]
# Set tzdata timezone #America/New_York
time_zone = "Asia/Calcutta"

The kube-monkey and other pods are in the same namespace.
kubectl -n cam-km get pods
NAME READY STATUS RESTARTS AGE
camanalytics-4163974132-4h3jx 1/1 Running 0 1d
camanalytics-4163974132-mp8h5 1/1 Running 0 1d
camapigateway-deploy-2118870940-c9n90 1/1 Running 0 1d
camapigateway-deploy-2118870940-zh1h3 1/1 Running 0 1d
camapisvrdeploy-1865230188-p8mz5 1/1 Running 0 1d
camservice-1027919561-0p8t1 1/1 Running 0 1d
camservice-1027919561-1bq15 1/1 Running 0 1d
camui-deploy-1847475197-0rsqb 1/1 Running 0 1d
camui-deploy-1847475197-j3g69 1/1 Running 0 1d
kube-monkey-3209647839-d6x66 1/1 Running 0 23h

[sudarshntn]

Hi Any update or suggestions?

[sudarshntn]

Any suggestion I have now moved it to the kube-system namespace also. How do we set the namespace permission

[sudarshntn]

Got it working by moving the kube-monkey pod from my namespace to kube-system namespace. So the kube-monkey pod and config map needs to be part of the kube-system namespace than being part of our namespace.

[asobti]

Sorry for not responding sooner.

The kube-monkey deployment and config map definitely need to be in the same namespace, but I'm not sure why it didn't work when it was in the same namespace as your app. I assumed the default user in the namespace has all access to objects within the namespace, but maybe that assumption was wrong.

[bharath0208]

@sudarshntn @asobti I am getting the same error now. However my kube-monkey and config are running on kube-system namespace.

I0719 15:14:37.034731 1 config.go:74] Successfully validated configs
I0719 15:14:37.034795 1 main.go:52] Starting kube-monkey with v logging level 5 and local log directory /var/log/kube-monkey
I0719 15:14:37.121400 1 kubemonkey.go:24] Status Update: Generating next schedule at 2019-07-22 08:00:00 -0300 ADT
I0722 11:00:00.000198 1 schedule.go:54] Status Update: Generating schedule for terminations
W0722 11:00:00.078503 1 factory.go:44] Failed to fetch eligible deployments for namespace dev due to error: deployments.extensions is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "deployments" in API group "extensions" in the namespace "dev"

Any suggestions or help in . this regards.

[aluwala]

Am also getting the same error :
Failed to fetch eligible deployments for namespace hygiearesilency due to error: deployments.apps is forbidden: User "system:serviceaccount:hygiearesilency:default" cannot list resource "deployments" in API group "apps" in the namespace "hygiearesilency"

found that creating a cluster role binding will help,but am unable to create at the clusterscope.It is forbidden.
Any suggestions regarding this!