websocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden

[rjshree]

Hi,

I'm using Kubernetes client to write a file once the pod turns to running state. Below code creates the file but throwing 403 exception and not able to write the file.
Note: I'm running this code within a pod (say, namespace a) and trying to create file in another namespaced pod (say namespace b).

Code:
config.load_incluster_config()
client_config = Configuration()
batch_client = client.BatchV1Api()
client_config.verify_ssl = False
kube_api = client.CoreV1Api(client.ApiClient(client_config))
script = TEMPLATE_STARTER_SCRIPT.format(instance_name, instance_type, instance_size)
exec_command = ['/bin/sh', '-c', "echo helloworld"]
pod_resp = stream(kube_api.connect_get_namespaced_pod_exec, pod_name, 'jhub-starter',
stderr=True, stdin=True,
stdout=True, tty=False,
_preload_content=False)
LOG.info("preload content")
pod_resp.close()

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 254, in websocket_call
client = WSClient(configuration, get_websocket_url(url), headers)
File "/usr/local/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 77, in init
self.sock.connect(url, header=header)
File "/usr/local/lib/python3.7/site-packages/websocket/_core.py", line 226, in connect
self.handshake_response = handshake(self.sock, *addrs, **options)
File "/usr/local/lib/python3.7/site-packages/websocket/_handshake.py", line 79, in handshake
status, resp = _get_resp_headers(sock)
File "/usr/local/lib/python3.7/site-packages/websocket/_handshake.py", line 160, in _get_resp_headers
raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden

Could anyone please help on this issue? I'm struck!

Thanks
Rajashree

[roycaihw]

/assign @alanjcastonguay

[k8s-ci-robot]

@roycaihw: GitHub didn't allow me to assign the following users: alanjcastonguay.

Note that only kubernetes-client members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @alanjcastonguay

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rjshree]

Hi All,

I'm triggering this delete pvc from one namespace to delete other namespace pvc and I'm getting the same error 403.

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"persistentvolumeclaims "claim-admin-pls-2dstart-2d1" is forbidden: User "system:serviceaccount:edisonai-brahmos:default" cannot delete resource "persistentvolumeclaims" in API group "" in the namespace "jhub-starter"","reason":"Forbidden","details":{"name":"claim-admin-pls-2dstart-2d1","kind":"persistentvolumeclaims"},"code":403}

I tried deleting same namespace' pvc, still could not achieve it. Below is the error:

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"persistentvolumeclaims "data-cassandra-2" is forbidden: User "system:serviceaccount:edisonai-brahmos:default" cannot delete resource "persistentvolumeclaims" in API group "" in the namespace "edisonai-brahmos"","reason":"Forbidden","details":{"name":"data-cassandra-2","kind":"persistentvolumeclaims"},"code":403}

I could not write a file nor delete pvc.
Could anyone please help me on this?

Code:
from kubernetes import client as kubec
from kubernetes import config
from kubernetes.client.rest import ApiException

config.load_incluster_config()

kube_configuration = kubec.Configuration()
namespace = 'jhub-starter'
corev1api_instance = kubec.CoreV1Api(kubec.ApiClient(kube_configuration))
try:
body = kubec.V1DeleteOptions(propagation_policy='Background')
corev1api_instance.delete_namespaced_persistent_volume_claim(name="claim-admin-pls-2dstart-2d1", namespace=namespace, body=body)
except ApiException as ex:
if "NotFound" not in ex.body:
err_msg = "Error in deleting training job: {}".format(ex)
log.info(self.experiment_id, err_msg)
raise ApiException(err_msg)

Thanks in Advance!

[ellieayla]

This doesn’t look like an API client problem. Instead it’s an RBAC config problem. You’re authenticating as the default ServiceAccount in the edisonai-brahmos namespace. What RoleBinding or ClusterRoleBinding cites that ServiceAccount, and what Role or ClusterRole does it cite?

[rjshree]

@alanjcastonguay PVC deletion is working with below RBAC:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
#namespace: edisonai-brahmos
name: pvc-writer
rules:

• apiGroups: [""] # "" indicates the core API group
  resources: ["pvc"]
  verbs: ["get", "post", "watch", "list"]

Still Im not able to write a file inside the running pod, Below is the RBAC for Pod:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: edisonai-brahmos
name: pod-writer
rules:

• apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "post", "watch", "list", "put", "patch"]

======================================

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: pod-writer
namespace: edisonai-brahmos
roleRef:
kind: Role
apiGroup: rbac.authorization.k8s.io
name: pod-writer
subjects:

• kind: ServiceAccount
  name: default
  namespace: edisonai-brahmos

Error:
Reason: Handshake status 403 Forbidden
ERROR:flask.app:Exception on /v1/notebooks [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 254, in websocket_call
client = WSClient(configuration, get_websocket_url(url), headers)
File "/usr/local/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 77, in init
self.sock.connect(url, header=header)
File "/usr/local/lib/python3.7/site-packages/websocket/_core.py", line 226, in connect
self.handshake_response = handshake(self.sock, *addrs, **options)
File "/usr/local/lib/python3.7/site-packages/websocket/_handshake.py", line 79, in handshake
status, resp = _get_resp_headers(sock)
File "/usr/local/lib/python3.7/site-packages/websocket/_handshake.py", line 160, in _get_resp_headers
raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden

Please correct me if I have given it wrong.

Thanks in Advance!!

[rjshree]

Again my pvc deletion is also not working. Is there any solution to delete pvc in different namespace? Could anyone please help me with this?
Previously it was working fine. I tried to add more verb and it is failing.
Since I have to delete the pvc in another namespace I have given clusterrole and clusterrolebinding.(attached below)

Error:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"persistentvolumeclaims "claim-admin-end-2dde-2dgrandfinale" is forbidden: User "system:serviceaccount:edisonai-brahmos:default" cannot delete resource "persistentvolumeclaims" in API group "" in the namespace "jhubtest2"","reason":"Forbidden","details":{"name":"claim-admin-end-2dde-2dgrandfinale","kind":"persistentvolumeclaims"},"code":403}

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
#namespace: edisonai-brahmos
name: pod-writer
rules:

• apiGroups: [""] # "" indicates the core API group
  resources: ["pvc"]
  verbs: ["get", "post", "create", "delete", "update", "watch", "list", "put", "patch"]
  =======================================================
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
  name: pod-writer
  roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: pod-writer
  subjects:
• kind: ServiceAccount
  name: default
  namespace: edisonai-brahmos
  =================================

Thanks in Advance !

[rjshree]

@alanjcastonguay
I have attached the clusterrole and its corresponding clusterrole binding. Please have a look.

Thanks
Alan

[rjshree]

I have to assign cluster-admin clusterrole to the default serviceaccount of my namespace.. which solved the issue.
But is there any other way?

Thanks in Advance!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[roycaihw]

AFAICT RBAC doesn't support short names. You need to specify "persistentvolumeclaims" instead of "pvc". The following role & binding work for me. For requests across namespaces you need to use cluster role and cluster role binding.

$ cat role.yaml 
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-rule
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "exec"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["delete", "deletecollection"]

$ cat bind.yaml 
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: Role
  name: my-rule
  apiGroup: rbac.authorization.k8s.io

# within a pod in the default namespace
root@my-shell:/# kubectl auth can-i get pvc
no
root@my-shell:/# kubectl auth can-i delete pvc
yes
root@my-shell:/# kubectl auth can-i deletecollection pvc
yes
root@my-shell:/# kubectl auth can-i get pod             
yes
root@my-shell:/# kubectl auth can-i list pod
yes
root@my-shell:/# kubectl auth can-i exec pod
yes
root@my-shell:/# kubectl auth can-i delete pod
no

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[ellieayla]

/close