Experiencing Intermittent 401 Unauthorized Errors from Kube API Server #1240
Description
Describe the bug
The Kubernetes JavaScript client library is employed in our Node.js application. Recently, experiencing intermittent 401 Unauthorized errors from the Kube API Server.
Error trace:
body: {
kind: 'Status',
apiVersion: 'v1',
metadata: {},
status: 'Failure',
message: 'Unauthorized',
reason: 'Unauthorized',
code: 401
}
Within the Node.js application, the logic involves listing the pods. Most of the time there were no errors observed, but sometimes this 401 error is thrown by the Kubernetes client.
This issue began to be noticed following the latest Kubernetes upgrade, which went from version 1.22.1 to 1.24.4.
Initially, the suspicion was directed towards the Kubernetes service account token. This suspicion arose because, starting from Kubernetes version 1.24, the token is no longer mounted as a secret by default; instead, it is mounted inside the container and refreshed by the Kubelet every hour. In contrast, in the 1.22 version, this token was stored as a secret.
we print the error stack trace in the app. By decoding the token that is passed as headers to the kube server, it is found that the token was generated a few seconds back and this happens intermittently when a new token is used after the token rotation every 1 hour.
However, upon further analysis, it became evident that this was not the root cause. This is because, for the most part, everything operates smoothly, and the occurrence of the 401 error appears to be sporadic and random.
Need help to find the concrete root cause of why this issue is happening.
Client Version
e.g. 0.16.3
Server Version
e.g. 1.24.4
Example Code
sample code snippet, the error is thrown from line 5
1 let kubeConfig = new k8s.KubeConfig();
2 kubeConfig.loadFromDefault();
3 let kubeApi = kubeConfig.makeApiClient(k8s.CoreV1Api)
4 let labelSelector = 'app=' + appName;
5 let res = await kubeApi.listPodForAllNamespaces(false, null, 'status.phase=Running', labelSelector);
Environment (please complete the following information):
- OS: Linux
- NodeJS Version 12.22.12
- Cloud runtime : NA