Proxy Mode API for child cluster: how to specific the user credentials

[panpan0000]

What happened:

When following the guide: https://github.com/clusternet/clusternet#visit-managedcluster-with-rbac

It will be a problem about how to pass the user cred to managed cluster.
I suffer from "forbidden: User "system:anonymous" cannot get path "/api"" error .
api-server does not set -anonymous-auth=false explicitly.

What you expected to happen:

Using curl + token to add Child Cluster API, or use kubectl to access it .

How to reproduce it (as minimally and precisely as possible):

Neither proxy/https mode or proxy/direct mode, you will have to find a way to satisfy Auth of **Child Cluster **.
So I tried to specific cert/key in your kube.conf or token in curl header (--header "Authorization: Bearer $TOKEN" )

It will be looked like (curl to access http proxy child cluster )

curl --header "Authorization: Bearer $TOKEN"  \
https://${HUB_CLUSTER_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-6556cb89cc2b/proxy/https/${CHILD_CLUSTER_IP}:6443/api/v1/nodes

At first, I thought the TOKEN should be CHILD-Cluster authorized token . So I picked an admin privilege token of child cluster.

Before sending it to Hub Cluster, I verified the token with directly access the Child cluster without clusterNet. token is good.✅

curl --header "Authorization: Bearer $TOKEN" https://${CHILD_CLUSTER_IP}:6443/api/v1/nodes -k
# the result is good, return the nodes to me 

But with clusterNet proxy ,using the same $TOKEN (Child Cluster token), it failed。❌

curl --header "Authorization: Bearer $TOKEN"  https://${HUB_CLUSTER_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-656cb89cc2b/proxy/https/${CHILD_CLUSTER_IP}:6443/api/v1/nodes -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

I believe it's due to Hub Cluster reject this token . So the first gate(Hub Cluster AuthZ) blocks request.

Ok, I change the $TOKEN to Hub Cluster admin token.
Now I can access Child Cluster's /healthz (this API does not require auth), ✅ like below

curl --header "Authorization: Bearer $TOKEN"  https://${HUB_CLUSTER_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-656cb89cc2b/proxy/https/${CHILD_CLUSTER_IP}:6443/healthz  -k

But it still failed for other API( like /apis/v1/nodes ), ❌ error will be :

 "message": "forbidden: User \"system:anonymous\" cannot get path \"/ping\"",

Same problem , I'm also confused the user config in kube-config.
For Child Cluster configuration in kube.conf:

• The ca cert(certificate-authority-data) should be Hub-Cluster cert
• But the user , I copied client-certificate-data and client-key-data from child cluster kube config file.

Below is my kubectl config, the failure was

kubectl config use-context k8s-21-child
kubectl get no  -v=9

Error:
[] GET https://172.**.**.20:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-6556cb89cc2b/proxy/https/172.**.**.21:6443/api?timeout=32s 403 Forbidden in 16 milliseconds
[] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/api\"","reason":"Forbidden","details":{},"code":403}

[root@my-172-**-**-20 ]# kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://172.**.**.20:6443
  name: k8s-20-parent
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://172.**.**.20:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-6556cb89cc2b/proxy/https/172.**.**.21:6443
  name: k8s-21-child
contexts:
- context:
    cluster: k8s-20-parent
    user: kubernetes-admin20
  name: k8s-20-parent
- context:
    cluster: k8s-21-child
    user: kubernetes-admin20
  name: k8s-21-child
current-context: k8s-21-child
kind: Config
preferences: {}
users:
- name: kubernetes-admin20
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
- name: kubernetes-admin21
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

Anything else we need to know?:

Environment:

• Clusternet version:
  • Clusternet-agent version (user clusternet-agent --version=json): latest . 0.4.0
  • Clusternet-hub version (user clusternet-hub --version=json): 0.4.0
• Kubernetes version (use kubectl version): k8s: 1.19.13(build by kubeadm ). kubectl binary : v1.18.20
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

[panpan0000]

Chinese text explains the same with above message , just to make myself more clear
补充一下中文,
通过主集群的API Aggregation，去访问子集群。 鉴权问题如何解决？
这是一个单纯的kubeadm搭建的k8s，没有特别的设置。
当我们curl被AA代理的子集群API，需要同时满足（1）主集群的认证鉴权 （2）子集群的认证鉴权
（1）其实不需要做，因为 sockets.proxies.clusternet.io 已经在clusterrolebinding (clusternet:system:socketsproxy)里面给anonymous用户赋权了
（2）子集群的鉴权无法避免。但是不管是curl还是kubectl，无法将TOKEN或者证书传递下去。如果我curl -H "Authorization: Bearer $TOKEN", 这个TOKEN会被Hub集群的api-server截获，而不会往下传递给子集群。所以，我没有办法去通过ClusterNet的Proxy API去调用子集群。

我目前唯一的打通方式是在子集群开启无认证的kubectl proxy --address= '$MYIP' --accept-hosts= ' ^*$ ' --port=8080

但是这并不是生产集群的实践。

弱弱请教，有劳 @dixudx

[dixudx]

@panpan0000 Thanks for using Clusternet.

The kubeconfig you are using is not correct. Please follow visting ManagedCluster with RBAC to construct a valid kubeconfig to access child clusters.

Clusternet DOES support visiting all your managed clusters with RBAC.

And Clusternet does not care about those credentials (tokens, keys/certficates) at all, passing them directly to child clusters. That means all the authentication/authorization are handled by child clusters. What the parent cluster does is just allowing forwarding these proxies requests to child clusters. And related RBACs are already declared when you deploy clusternet-hub.

[dixudx]

@panpan0000 I've created a PR to demonstrate curl usage. Please take a look.

[panpan0000]

Below (following your PR https://github.com/clusternet/clusternet/pull/125/files ) now works ✅
TOKEN is Child Cluster admin SA token

curl -H "Accept: application/json" -H "Impersonate-User: clusternet" -H "Impersonate-Extra-Clusternet-Token: $TOKEN"  -H "Authorization: Basic system:anonymous"  https://${HUB_CLUSTER_IP}:6443/apis/proxies.clusternet.io/v1alpha1/sockets/7a93727c-6609-45bc-8c3e-6556cb89cc2b/proxy/https/${CHILD_CLUSTER_IP}:6443/api/v1/nodes  -k

[panpan0000]

Copied . I missed those part of user impersonation

  user:
    username: system:anonymous
    as: clusternet
    as-user-extra:
        clusternet-token:
            - BASE64-DECODED-PLEASE-CHANGE-ME

I found the theory behind it:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation

good job and Thank you @dixudx