v1.27.0 kube-apiserver restart failed after edit yaml file （add --token-auth-file）

[chaseSpace]

What happened?

I did:

• kubeadm init (single node)
• edit /etc/kubernetes/manifests/kube-apiserver.yaml

edit part：

spec:
  containers:
  - command:
    - kube-apiserver
    - --token-auth-file=/etc/kubernetes/k8s_account_tokens.csv   <------------- new line added
    - --advertise-address=192.168.31.2

Then kube-apiserver pod Exited! it reported no such file... for token-auth-file added：

➜  practice crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD
4389df402233a       48f6f02f2e904       2 seconds ago       Exited              kube-apiserver            4                   2d7f04230740f       kube-apiserver-k8s-master
2f3fcac27979d       62a4b43588914       6 minutes ago       Running             kube-scheduler            1                   1a9bfdeacb631       kube-scheduler-k8s-master
4261cad5bc6f3       2fdc9124e4ab3       6 minutes ago       Running             kube-controller-manager   1                   95b15dd3eec53       kube-controller-manager-k8s-master
6387487eb96a3       b2d7e01cd611a       6 minutes ago       Running             kube-proxy                0                   c4537525bb235       kube-proxy-k5wst
b149955b883da       62a4b43588914       7 minutes ago       Exited              kube-scheduler            0                   1a9bfdeacb631       kube-scheduler-k8s-master
af95214f6a93b       2fdc9124e4ab3       7 minutes ago       Exited              kube-controller-manager   0                   95b15dd3eec53       kube-controller-manager-k8s-master
9ec75b0d0ecbe       fce326961ae2d       7 minutes ago       Running             etcd                      0                   822a8fef1c695       etcd-k8s-master

➜  practice crictl logs 4389df402233a            
I1118 21:47:07.791379       1 server.go:551] external host was not specified, using 192.168.31.2
I1118 21:47:07.792647       1 server.go:165] Version: v1.27.0
I1118 21:47:07.792663       1 server.go:167] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
E1118 21:47:08.124085       1 run.go:74] "command failed" err="open /etc/kubernetes/k8s_account_tokens.csv: no such file or directory"

# but it exactly exists!
➜  practice cat /etc/kubernetes/k8s_account_tokens.csv                                        
nlZtQeHoS8k0Pvbe,user3,3
nxdt123445k0P21d,user4,4
➜  practice ls -l /etc/kubernetes/k8s_account_tokens.csv 
-rwxrwxrwx. 1 root root 50 Nov 19 03:06 /etc/kubernetes/k8s_account_tokens.csv

What did you expect to happen?

kube-apiserver pod restart normally, not Exited status!

How can we reproduce it (as minimally and precisely as possible)?

Just do as i described above.

Anything else we need to know?

No

Kubernetes version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:10:18Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1

Cloud provider

OS version

➜ practice cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

➜ practice uname -a
Linux k8s-master 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

kubeadm

Container runtime (CRI) and version (if applicable)

containerd containerd.io 1.6.24 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523

Related plugins (CNI, CSI, ...) and versions (if applicable)

no installed

[chaseSpace]

/sig Auth

[neolit123]

1.25 is not in support.
https://kubernetes.io/releases/

is the problem reproducible with newer k8s?

[neolit123]

/sig api-machinery

[chaseSpace]

I reproduced it on v1.27.0. @neolit123

[chaseSpace]

Problem solved.

I noticed /etc/kubernetes/manifests/kube-apiserver.yaml contains volumns part as below:

volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
...

  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs

Then I realized kube-apiserver's args with file must read path only from one of volumeMounts.mountPath[*], so i updated my args:

spec:
  containers:
  - command:
    - kube-apiserver
    - --token-auth-file=/etc/kubernetes/pki/k8s_account_tokens.csv   <------------- read from /etc/kubernetes/pki
    - --advertise-address=192.168.31.2

It works.

PS: below are some experiences i got from when i troubleshooting:

• use crictl tool to check k8s core component pod status, like watch crictl ps, crictl ps -a, crictl logs <container-id>
• these pods will restart automatically when you delete it with crictl rm <container-id>
  • and them will restart periodly if pod gets to Exited status, about ten seconds
• use journalctl -u kubelet -f --lines=10 |grep apiserver to check kubelet log

[neolit123]

i was about to post that, yes you must mount a dir or a file in the static pod container.

for support in the future please use #kubeadm on the k8s slack or other support channels.
not github.
https://github.com/kubernetes/kubernetes/blob/master/SUPPORT.md

[alexzielenski]

/triage accepted