Mistaken auth header name

[Fly-Luck]

Problem

Even if correct ServiceAccount access was established(ServiceAccount + ClusterRoleBinding + ClusterRole or ServiceAccount + RoleBinding + Role) and the pod was referring to the serviceAccountName, the pod still cannot call k8s api inside the cluster(403 Forbidden from k8s api server). e.g.

{"level":"error","msg":"Status: 403 Forbidden, Body: {\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"notebooks.kubeflow.org is forbidden: User \\\"system:anonymous\\\" cannot list resource \\\"notebooks\\\" in API group \\\"kubeflow.org\\\" in the namespace \\\"default\\\"\",\"reason\":\"Forbidden\",\"details\":{\"group\":\"kubeflow.org\",\"kind\":\"notebooks\"},\"code\":403}\n","time":"2019-06-06T02:57:08Z"}

Code to call the k8s api:

		cfg, err := config.InClusterConfig()
		if err != nil {
			panic(err.Error())
		}
                k8sClient = client.NewAPIClient(cfg)
                // the code to list a CustomResource Object
                ......

Temporary Fix

Simple, I just replaced the original header with another header name(Authentication->Authorization), and the api call succeeded:

                MistakenTokenHeader := "Authentication"
                CorrectTokenHeader := "Authorization"
                TokenValuePrefix = "Bearer "
                SATokenPath := "/var/run/secrets/kubernetes.io/serviceaccount/token"
		cfg, err := config.InClusterConfig()
		if err != nil {
			panic(err.Error())
		}
		tokenValue := cfg.DefaultHeader[MistakenTokenHeader]
		// this should never happen, just in case
		if len(tokenValue) == 0 {
			tv, err := ioutil.ReadFile(SATokenPath)
			if err != nil {
				panic(err.Error())
			}
			tokenValue = TokenValuePrefix + string(tv)
		}
		defaultHeader := map[string]string{CorrectTokenHeader : tokenValue}
		cfg.DefaultHeader = defaultHeader
                k8sClient = client.NewAPIClient(cfg)

Question

Is there anything wrong with my client code or is this an issue of kubernetes-client?

[rsreeni]

I am getting the same error. Unfortunately, even the workaround provided above doesn't seem to work for me.

Error Reading abc-svc-config-map ConfigMap Status: 403 Forbidden, Body: {"kind":"Status","apiVersion":"v1", "metadata":{},"status":"Failure","message":"configmaps \"abc-svc-config-map\" is forbidden: User \"system:serviceaccount:test:api-access-sa\" cannot get resource \"configmaps\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"name":"abc-svc-config-map","kind":"configmaps"},"code":403}

am I missing something ? Is there any fix in progress ?

Thanks,

-SR

[rsreeni]

I found issue why the above workaround was not working. The ClusterRoleBinding was not applied to the role in the namespace the pod was running. Now that I have added, the workaround is working and I am able to reach the API Server.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.