node_filesystem_readonly showing wrong information

[schewara]

For an unknown reason the node_filesystem_readonly showing various mounts falsely as read-only when running as container.

As the mixin alerts also depend on the correct readonly status, this also results in broken alerting.

The output on the host machine, which correctly shows the correct state

# cat /proc/mounts|grep '/dev/sda3 /'
/dev/sda3 / ext4 rw,relatime,errors=remount-ro 0 0

# mount|grep '/dev/sda3'
/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro)

For an unknown reason, the mount in question is listed as readonly.

# docker exec -it nodeexporter cat /rootfs/proc/mounts |grep '/dev/sda3 /rootfs'
/dev/sda3 /rootfs ext4 ro,relatime,errors=remount-ro 0 0

My assumption would be that these files should be identical and it's totally not clear to me how the content in the mounts file differs when it is just bind-mounted into the container.

/rootfs/proc/mounts itself links to self/mounts but the links are pointing to different directories in and outside the container and the directory from the host is not existing inside of the container.

# ls -l /proc/ |grep self
lrwxrwxrwx  1 root             root                          11 Jun 21 10:14 mounts -> self/mounts
lrwxrwxrwx  1 root             root                           8 Jul  6 05:41 net -> self/net
lrwxrwxrwx  1 root             root                           0 Jun 20 06:46 self -> 20115
lrwxrwxrwx  1 root             root                           0 Jun 20 06:46 thread-self -> 20115/task/20115

# docker exec -it nodeexporter ls -l /rootfs/proc/ |grep self
lrwxrwxrwx    1 root     root            11 Jun 21 10:14 mounts -> self/mounts
lrwxrwxrwx    1 root     root             8 Jul  6 05:41 net -> self/net
lrwxrwxrwx    1 root     root             0 Jun 20 06:46 self -> 20241
lrwxrwxrwx    1 root     root             0 Jun 20 06:46 thread-self -> 20241/task/20241

# docker exec -it nodeexporter ls -l /rootfs/proc/20115
ls: /rootfs/proc/20115: No such file or directory

Maybe someone with deeper knowledge of Docker has an Idea what is happening here in the background and a way to have node_exporter provide the correct metric values.

Host operating system: output of uname -a

4.19.0-20-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

quay.io/prometheus/node-exporter:v1.3.1

node_exporter command line flags

services:
  nodeexporter:
    container_name: nodeexporter
    hostname: nodeexporter
    image: quay.io/prometheus/node-exporter:v1.3.1
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro,rslave
      - /run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro
      - /opt/my/textfile.metrics:/textfiles:ro
    command:
      - --path.procfs=/rootfs/proc
      - --path.rootfs=/rootfs
      - --path.sysfs=/rootfs/sys
      - --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)
      - --collector.systemd
      - --collector.textfile.directory=/textfiles
      - --log.format=json

I tested it with explicit proc and sys mounts and also with and without the --path.procfs and --path.sysfs options, but it had no effect

node_exporter log output

N/A

Are you running node_exporter in Docker?

yes

What did you expect to see?

node_filesystem_readonly{device="/dev/sda3",fstype="ext4",mountpoint="/"} 0

What did you see instead?

node_filesystem_readonly{device="/dev/sda3",fstype="ext4",mountpoint="/"} 1

[discordianfish]

Well you bind-mount rootfs as read only by specifying - /:/rootfs:ro,rslave. To be fair though.. mounting it as read-only probably makes sense and yield this problem so we might consider some solution. Can't think of anything though.. @SuperQ thoughts?