daprd sidecar startup error: "/.kube/config: no such file or directory"

[alecor191]

In what area(s)?

/area runtime

What version of Dapr?

1.13.0 but also observed in 1.12 and 1.11.

Expected Behavior

Dapr sidecars start up successfully when a K8S pod is created

Actual Behavior

Dapr sidecar fails with

Failed to init component kubernetes (secretstores.kubernetes/v1):
[INIT_COMPONENT_FAILURE]: initialization error occurred for kubernetes (secretstores.kubernetes/v1):
stat /.kube/config: no such file or directory

Fatal error from runtime: process component kubernetes error: [INIT_COMPONENT_FAILURE]:
initialization error occurred for kubernetes (secretstores.kubernetes/v1):
[INIT_COMPONENT_FAILURE]: initialization error occurred for kubernetes (secretstores.kubernetes/v1):
stat /.kube/config: no such file or directory

K8S tries to restart the daprd container several times, but they all fail.

The only way to get out of it is to delete the whole pod: K8S will recreate the pod and daprd can be started successfully

Steps to Reproduce the Problem

We've been observing this for months now: Every couple of days, one of the pods in our AKS clusters fails to start up, because the daprd sidecar crashes (with the error above).
As soon as we delete the pod and let K8S create a new one, daprd starts up successfully.

Frequency: We have multiple clusters, each running a dozen or more pods with Dapr sidecar. Once every couple of days one of them fails to start up. By now, it has happened to all AKS clusters and all our services. I.e. the issue is not specific to one of our services or a specific cluster/node.

[darraghjones]

I'm having the same intermittent issue. I raised it in the dapr discord and received this advice:

The built in kubernetes secret store is always enabled (unless you use the annotation dapr.io/disable-builtin-k8s-secret-store ). Without this secret store you can not load Dapr components and reference Kubernetes secrets as values. If you always hardcode values / secrets - or use a different secret store instead (Azure KeyVault, Hashicorp Vault etc) and don't need Kubernetes secrets - then feel free to disable.

As for why the kubeconfig isn't mounted into your sidecar.. I do not know. Inspect the helmchart you used to deploy Dapr and the options you specified. Also check your cluster permissions.

[JoshVanL]

Indeed @darraghjones what is written there is correct. It is suspicious to me as to why this is happening intermittently and seemingly at random. Containers in Kubernetes Pods will only execute once all associated volumes are ready and successfully mounted into the target container(s). Service account tokens should always be written to the underlying tmpfs volume before mounting into the Dapr container. If confident that the service account has been enabled for mounting to Pods, this happening randomly would suggest to me that there may be some kind of race condition in the underlying Kubernetes provider (AKS).

[madala0611]

I am also facing similar kind of issue in AKS cronjob->job->pod->containers(application and daprd).
After enabling the secure computing mode (seccopm) in azure kubernetes application pod, the daprd sidecar container unable to create it and we are having the below error in daprd container logs:

time="2024-04-26T02:49:33.995282529Z" level=info msg="Closing HTTP server [::1]:3500…" app_id=newcsp-scheduler-notifyinactiveusers instance=newcsp-scheduler-notifyinactiveusers-1-z962r scope=dapr.runtime.http type=log ver=1.12.0
time="2024-04-26T02:49:33.995567845Z" level=fatal msg="Fatal error from runtime: process component kubernetes error: [INIT_COMPONENT_FAILURE]: initialization error occurred for kubernetes (secretstores.kubernetes/v1): [INIT_COMPONENT_FAILURE]: initialization error occurred for kubernetes (secretstores.kubernetes/v1): stat /home/nonroot/.kube/config: no such file or directory" app_id=newcsp-scheduler-notifyinactiveusers instance=newcsp-scheduler-notifyinactiveusers-1-z962r scope=dapr.runtime type=log ver=1.12.0

Appreciate if you have any solution for above issue at the earliest.

[alicejgibbons]

A note here that this was seen intermittently in EKS as well, so not AKS specific.