[kube-prometheus-stack] node-exporter crashes at startup

[davgia]

Describe the bug
Node exporter pod underlying container fails to startup. This is the error reported by the deamon:

Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount

This error is related to #325 and it appears to be caused by this pull request. I've create this issue as suggested by @monotek in #80 (comment) (it appears that the original reporter of this issue has not created a new issue).

Moreover, I've tried to change kube-prometheus-stack chart manually to revert the commit above and I am able to deploy the chart without any error.

I am using docker-desktop (v2.5.0.1) on Windows 10 1909 and I have enabled Kubernetes cluster from it.

I'm not sure if this error is caused by docker-desktop, in any case, if it does not happen in other type of clusters, I would suggest to make the mount rootfs mountPropagation configurable via the chart values (and maybe enable it by default if you think it should not be optional) to prevent this error.

Another (better) solution would be to mount / as shared, but I've not found a way to do it.

Version of Helm and Kubernetes:

Helm Version:

version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

Kubernetes Version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Which chart: kube-prometheus-stack

Which version of the chart: 12.7.0

What happened: node-exporter pod fails to startup

What you expected to happen: node-exporter should not fail

How to reproduce it:

No additional change in values.yaml file to reproduce.

The helm command that you execute and failing/misfunctioning:

helm install prom-stack prometheus-community/kube-prometheus-stack --version v12.7.0 -n monitoring

[radhus]

I have the same issue, running a k3s v1.18 cluster on Linux (Alpine) if that helps rule anything out.

[Gowlar]

I have the this issue also on Docker Desktop for Mac 3.0.0

[mvaldes14]

Same issue on Docker for Windows.

Client: Docker Engine - Community
 Version:           20.10.0

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)

helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

kubectl.exe version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

[jonyjalfon94]

Same issue in docker for windows ( Using Hyper-V backend ) with Kubernetes v1.19.3 and Helm v3.4.1.

[msergei]

The same issue with Docker Desktop for Mac v.3.

[iemkamran]

Same issue using Docker Desktop for windows 10 on WSL2

[dshah2012]

Facing the same issue using Docker Desktop for windows 10.

DockerEngine Version: 20.10.2
Helm version: v3.3.1
Kubernetes version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[3wolf]

+1

[aagmangour]

+1
Same issue on docker desktop for Mac. v 2.5.0.1
Kubernete version 1.19.3

[ktzsolt]

+1 for Docker Desktop for Windows 10 (Pro, version 2004) on WSL2

[nunoguerreirorosa]

+1

[moatorres]

+1

[v1lev]

+1

[kingpeti]

+1

[golosegor]

+1

[prasetiyohadi]

+1

[monotek]

Please stop the "+1".
It just creates noise.

Alternatively you just could add a thumbs up via the smiley function.

[kristijorgji]

Same issue for me, this + another one
#687
In above issue are all my versions of toolings and steps to reproduce

Prometheus is exporting nothing, kube-state-metrics if i port forward has some metrics, but querying prometheus is totally empty, node exporter broken

[anktsrkr]

@monotek what is the alternate to make it work?

[ortizjavier]

I had the same issue and as a workaround, I removed the mountPropagation to use the default one.
It seems that everything is better but I'm not sure to understand what this change means.

kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

[davgia]

@monotek I think this issue is not solved. It was opened for kube-prometheus-stack. The pull request by @radhus
changed only the prometheus chart that it's not a dependency of the kube-prometheus-stack chart.

The changes made to the prometheus chart must be replicated in the prometheus-node-exporter chart too.

Let me know if I'm correct, I will create a pull request myself if needed.

[monotek]

I think this was automatically closed because the pr #737 was merged and this issue was mentioned...
Reopening...

[Azbesciak]

Same to me as @dfsdevops , to say more because of this my other services are unable to export ports.
I am operating on Windows,

docker v20.10.5
kubernetes v1.19.7

I fetched kube-prometeus, then

kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/

and after 2-3 mins my services stopped proxying. Their restart does not help, I need to delete everything applied in manifest and setup

[aldenjenkins]

I, too, am having this issue on diskless iPXE booted alpine nodes:

Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount

The kube-prometheus-stack chart needs to account for this nodeExporter.hostRootfs variable as well.

[aldenjenkins]

aha in the kube-prometheus-stack template, but it is a different name, and does not exist in the kube-prometheus-stack/values.yaml file for easy reference.

The value you need to use for the kub-prometheus-stack values is:

prometheus-node-exporter:
  hostRootFsMount: false

[ozbillwang]

you don't have to copy the whole values.yaml

with below way, you can fix the problem easily

$ cat values.yaml

prometheus-node-exporter:
  hostRootFsMount: false
  
$ helm list
$ helm upgrade --install my-prometheus prometheus-community/kube-prometheus-stack -f values.yaml

my-prometheus is my prometheus chart name, you can replace with yours

[bluetechy]

I had to do the following to get it to work in my values.yaml file because I am running kube-prometheus-stackas a dependency in my helm chart:

kube-prometheus-stack:
    prometheus-node-exporter:
        hostRootFsMount: false

[OlamideOl1]

had to do the following to get it to work with my helm install command for Docker-Desktop for Mac:

--set nodeExporter.hostRootfs=false

[jamesbright]

I had the same issue and as a workaround, I removed the mountPropagation to use the default one. It seems that everything is better but I'm not sure to understand what this change means.

kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

Thanks a lot, this worked.

[vg185118]

had to do the following to get it to work with my helm install command for Docker-Desktop for Mac:

--set nodeExporter.hostRootfs=false

helm install {Name} prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount.enabled=false

[doublespaces]

For those using the kube-prometheus-stack where prometheus-node-exporter is a dependency, you need to add this change to the prometheus-node-exporter values file and not the kube-prometheus-stack values file. Change the enabled value to false:

# Mount the node's root file system (/) at /host/root in the container
hostRootFsMount:
  enabled: false
  # Defines how new mounts in existing mounts on the node or in the container
  # are propagated to the container or node, respectively. Possible values are
  # None, HostToContainer, and Bidirectional. If this field is omitted, then
  # None is used. More information on:
  # https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
  mountPropagation: HostToContainer

[filwu8]

PS C:\Users\do> kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'
Error from server (NotFound): daemonsets.apps "monitoring-prometheus-node-exporter" not found

[puitiza]

For those using the kube-prometheus-stack where prometheus-node-exporter is a dependency, you need to add this change to the prometheus-node-exporter values file and not the kube-prometheus-stack values file. Change the enabled value to false:

# Mount the node's root file system (/) at /host/root in the container
hostRootFsMount:
  enabled: false
  # Defines how new mounts in existing mounts on the node or in the container
  # are propagated to the container or node, respectively. Possible values are
  # None, HostToContainer, and Bidirectional. If this field is omitted, then
  # None is used. More information on:
  # https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
  mountPropagation: HostToContainer

@doublespaces thanks your comment I could add those values.
I'm more visual personal so If someone ask where is it ? here , look below.

[mahesh2121]

@puitiza thanks for solution it works for me just updating the values.

[helbing]

The prometheus-node-exporter is version 4 now, you can use the values.yaml as follows to make it work.

prometheus-node-exporter:
  hostRootFsMount:
    enabled: false

[GitJaehyeonLee]

I found another way.

Because My error was another folder, But the solution was the same.

if your error is
Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount

OS [Shell Command]:
mount --make-rshared /

else if your error is
Error: failed to start container "node-exporter": Error response from daemon: path /sys is mounted on /sys but it is not a shared or slave mount

OS [Shell Command]:
mount --make-rshared /sys

But I haven't been able to confirm that it works even after rebooting.
If that doesn't work, you can consider fstab.

[yybht155]

我找到了另一种方法。

因为我的错误是另一个文件夹，但解决方案是相同的。

如果你的错误是 Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount

操作系统[外壳命令]： mount --make-rshared /

否则如果你的错误是 Error: failed to start container "node-exporter": Error response from daemon: path /sys is mounted on /sys but it is not a shared or slave mount

操作系统[外壳命令]： mount --make-rshared /sys

但我无法确认它即使在重新启动后也能工作。 如果这不起作用，您可以考虑 fstab。

In a k3s cluster on wls2 this method solved the problem for me!

[patrykpanter]

OS [Shell Command]:
mount --make-rshared /

Worked for me in a proxmox container (Ubuntu 18.04.6).
Thanks.

[AxelTahmid]

For those using the kube-prometheus-stack where prometheus-node-exporter is a dependency, you need to add this change to the prometheus-node-exporter values file and not the kube-prometheus-stack values file. Change the enabled value to false:

# Mount the node's root file system (/) at /host/root in the container
hostRootFsMount:
  enabled: false
  # Defines how new mounts in existing mounts on the node or in the container
  # are propagated to the container or node, respectively. Possible values are
  # None, HostToContainer, and Bidirectional. If this field is omitted, then
  # None is used. More information on:
  # https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
  mountPropagation: HostToContainer

Thanks. this worked for me, added it my command

helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
    --namespace monitoring --create-namespace \
    --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
    --set prometheus-node-exporter.hostRootFsMount.enabled=false \
    --set prometheus-node-exporter.hostRootFsMount.mountPropagation='HostToContainer' \