kubernetes default "admin" user isn't authorized to do anything useful

[azazel75]

Issue description

In NixOS 18.03, with a default installation of an "all-in-one"
kubernetes cluster, the configured and default "admin" user isn't able
to do anything useful when kubectl is used by connecting it to the
https port. This is probably caused by the configuration of
kube-apiserver to use RBAC but without associating the admin user to
none of the administrative groups.

Steps to reproduce

In a fresh installation of NixOS 18.03 add the following to
configuration.nix:

  services.kubernetes.roles = [ "master" "node"];

Then switch the configuration with:

# nixos-rebuild switch

When the new configuration is installed, open a terminal and type:

# kubectl -s https://localhost --insecure-skip-tls-verify get nodes
Please enter Username: admin
Please enter Password: **********
Error from server (Forbidden): nodes is forbidden: User "admin" cannot list nodes at the cluster scope

#

Use the default configured password for user admin:
"kubernetes". Compare that result with the use of the unsecured http
port 8080:

# kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
giskard.lan   Ready     <none>    1m        v1.9.1

#

The solution to this issue is to associate the admin with the
"system:masters" group defined by the kube-apiserver:

# kubectl describe clusterrolebinding cluster-admin
Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind   Name            Namespace
  ----   ----            ---------
  Group  system:masters  

#

That is achieved by adding that group name to the user file created by
the configuration:

# echo 'kubernetes,admin,0,"system:masters"' > /tmp/users

Then add this line to configuration.nix:

  services.kubernetes.apiserver.basicAuthFile = /tmp/users;

Then install the configuration and re-run the first kubectl line:

# nixos-rebuild switch
building Nix...
building the system configuration...
...
starting the following units: kube-apiserver.service

# kubectl -s https://localhost --insecure-skip-tls-verify get nodes
Please enter Username: admin
Please enter Password: **********
NAME          STATUS    ROLES     AGE       VERSION
giskard.lan   Ready     <none>    25m       v1.9.1

#

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

• system: "x86_64-linux"
• host os: Linux 4.15.18, NixOS, 18.03.132336.ef74cafd3e5 (Impala)
• multi-user?: no
• sandbox: yes
• version: nix-env (Nix) 2.0.1
• channels(root): "nixos-18.03.132336.ef74cafd3e5"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[FRidh]

cc @offlinehacker

[arianvp]

Maybe this is related, but last time I tried setting up K8s, the kube-dns service wouldn't come up as it couldn't connect to the apiserver. I feel it was maybe some permission problem as well

[azazel75]

Eh, you should open another issue and to try to clarify it a bit.

[azazel75]

This is now superseded by the current state of Kubernetes