[prometheus-kube-stack] prometheus-node-exporter pod is not running through helm install

[vallabhav]

Describe the bug
Helm install of the chart "prometheus-community/kube-prometheus-stack" in windows 10 machine is not installing "prometheus-node-exporter" pod and it is always in "CrashLoopBackOff" mode.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Which chart: kube-prometheus-stack

Which version of the chart: kube-prometheus-stack-11.0.4
prometheus-community/kube-prometheus-stack

What happened:
When I install through helm install, all the pods are running but "prometheus-node-exporter" is failed with "CrashLoopBackOff" status and the following is the error.

The log shows the following:

failed to try resolving symlinks in path "/var/log/pods/monitoring_prometheus-prometheus-node-exporter-5xxvz_591e5b9d-1f11-4418-9d44-2918fbfe66f2/node-exporter/4.log": lstat /var/log/pods/monitoring_prometheus-prometheus-node-exporter-5xxvz_591e5b9d-1f11-4418-9d44-2918fbfe66f2/node-exporter/4.log: no such file or directory

What you expected to happen:
The prometheus-node-exporter pod should run as other pods.

How to reproduce it (as minimally and precisely as possible):
Try run helm install

I tried the following command.

helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

I had windows 10 machine with latest docker desktop installed.

<~--
This could be something like:

values.yaml (only put values which differ from the defaults)

key: value

helm install my-release prometheus-community/name-of-chart --version version --values values.yaml

-->

Anything else we need to know:

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[kristijorgji]

/reopen

This issue is still present for me today in addition to other issue reporter here
#687

All versions are reported there as well

[maiorano84]

Also can confirm this issue on Windows 10.

I was able to get around it by setting hostRootFsMount to false, but I don't know what the far-reaching consequences might be:

helm install prometheus prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount=false --set grafana.adminPassword=secret

[mrvisser]

In more recent versions of the helm chart, the workaround property is now:

nodeExporter.hostRootfs=false

[markrendle]

With the latest version of the chart, in Docker Desktop for Windows, I still had to use

--set prometheus-node-exporter.hostRootFsMount=false

to get this working.

Full command:

helm install prom prometheus-community/kube-prometheus-stack --namespace telemetry --set prometheus-node-exporter.hostRootFsMount=false

[mrvisser]

With the latest version of the chart, in Docker Desktop for Windows, I still had to use

Ah, ok. I just double checked and my special case is that I'm installing the prometheus chart, which requires value nodeExporter.hostRootfs. But if installing the node exporter directly, you'd want to use prometheus-node-exporter.hostRootFsMount.

[coman3]

Just to help the next person that stumbles onto this issue with Docker Desktop for windows like i did, the latest version seems to work by setting prometheus-node-exporter.hostRootFsMount.enabled=false, all of the above failed to work on version 43.0.0

[jas-atwal]

I am encountering this exact same issue, however I am using DockerDesktop on MacOS - 13.6.7 (22G720) - Ventura.

Does anyone know of any workarounds/resolutions please?

[jas-atwal]

Managed to get around this issue by running the following command to patch the node-exporter deamon-set

kubectl -n patch ds --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'