GKE MasterAuth clientCertificate has no permissions on Create

[lukeweber]

/kind bug

What happened:
When provisioning a GKE cluster with google-api-go-client, the MasterAuth client certificate is set to Subject: CN=client. When I try to use the MasterAuth certs to connect to the cluster it says client has no permissions.

kubectl get ns
Error from server (Forbidden): namespaces is forbidden: User "client" cannot list namespaces at the cluster scope: Unknown user "client"

If I use gclient to get other credentials, I can modify RBAC to fix this with the following code:

ClusterRoleBinding 
apiVersion: rbac.authorization.k8s.io/v1 
metadata: 
name: client
subjects: 
- kind: User 
name: client
roleRef: 
kind: ClusterRole 
name: "cluster-admin" 
apiGroup: rbac.authorization.k8s.io

What you expected to happen:
I would expect that "client" would have admin permissions on the cluster similar to basic auth which has full admin, or that the cert would simply be an admin cert(Subject: CN=admin) or that subject common name could be configurable on cluster create via the api so users could choose.

How to reproduce it (as minimally and precisely as possible):

• Create a new gke cluster
• Use gcloud container clusters describe
• Try to connect to cluster using masterAuth: clientCertificate and clientKey and do any action on the server.

Anything else we need to know?:
As an example basic auth gives you admin user, but the more secure certs gives you a user with no permissions.

Environment:

• Kubernetes version (use kubectl version): 1.10.4-gke.2
• Cloud provider or hardware configuration: gke
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[ghost]

/sig auth

[mattfysh]

@lukeweber did you end up finding a solution for this? I've just upgraded to 1.11 and noticed this behaviour (which I believe didn't happen back in 1.9)

context here is I'm trying to get an uninterrupted terraform provisioning script, right now after provisioning the GKE cluster I need to stop the terraform script and manually bind client User to the cluster-admin role. My terraform kubernetes provider is configured as such:

provider "kubernetes" {
  host = "https://${google_container_cluster.primary.endpoint}"
  client_certificate = "${base64decode("${google_container_cluster.primary.master_auth.0.client_certificate}")}"
  client_key = "${base64decode("${google_container_cluster.primary.master_auth.0.client_key}")}"
  cluster_ca_certificate = "${base64decode("${google_container_cluster.primary.master_auth.0.cluster_ca_certificate}")}"
}

[aeneasr]

I'm facing the same issue. Why was this closed and how can I work around it?

[lukeweber]

So per recommendation, I did post on the kubernetes engine bug tracker and it became this private issue:

https://issuetracker.google.com/u/1/issues/111101728, feel free to reference it.

In a nutshell, the client cert has CN=client encoded and client user doesn't have any permissions. If you use masterAuth username/password (basic auth), then you can apply the yaml.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: client-binding
subjects:
- kind: User
  name: client
roleRef:
  kind: ClusterRole
  name: "cluster-admin"
  apiGroup: rbac.authorization.k8s.io

Which will give the user on the cert admin permissions. Additionally, to remove basic auth you can set the username="" in the api, but this will cause a reboot which will take 5 more minutes to do a master switch.

[Stelminator]

This was hard to find: https://www.terraform.io/docs/providers/google/d/datasource_client_config.html

I'm not sure if that solves your problem, but, this was the bit I needed for authorization w/o client cert or master auth. I'm leaving it here in case it helps someone else.

data "google_client_config" "default" {}

data "google_container_cluster" "my_cluster" {
  name   = "my-cluster"
  zone   = "us-east1-a"
}

provider "kubernetes" {
  load_config_file = false

  host = "https://${data.google_container_cluster.my_cluster.endpoint}"
  token = "${data.google_client_config.default.access_token}"
  cluster_ca_certificate = "${base64decode(data.google_container_cluster.my_cluster.master_auth.0.cluster_ca_certificate)}"
}

crosslink to other somewhat relevant issue: Shippable/support#4667

[duckie]

Hi

Thanks for the catch, I was going nuts with it. Not sure if the token workaround should be used when giving the creds to other people.