Metric was collected before with the same name and label values

[gnanasalten]

Host operating system: output of uname -a

Linux dc2cpoenrvmd534 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed May 18 16:02:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970)
build user: root@6e7732a7b81b
build date: 20221129-18:59:09
go version: go1.19.3
platform: linux/amd64

node_exporter command line flags

/usr/local/bin/node_exporter --collector.systemd --collector.sockstat --collector.filefd --collector.textfile.directory=/var/lib/node_exporter/

node_exporter log output

Sep 15 02:57:37 xxxxxxxxx node_exporter: ts=2023-09-15T02:57:37.684Z caller=stdlib.go:105 level=error msg="error gathering metrics: 17 error(s) occurred:\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/boot" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/boot" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var/log" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var/log/audit" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/boot" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/home" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/opt" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/tmp" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var/tmp" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/dev/shm" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/home" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var/log" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var/tmp" > untyped:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_fstab_mount_status" { label:<name:"filesystem" value:"/var/log/audit" > untyped:<value:1 > } was collected before with the same name and label values"

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Scrape from prometheus

What did you expect to see?

No error

What did you see instead?

error

[discordianfish]

Can you provide your /etc/fstab and /proc/mounts?

[gnanasalten]

Can you provide your /etc/fstab and /proc/mounts?
/etc/fstab

LABEL=img-rootfs / ext4 rw,relatime 0 1
LABEL=img-boot /boot ext4 rw,relatime 0 1
LABEL=fs_var /var ext4 rw,relatime 0 2
LABEL=fs_var_tmp /var/tmp ext4 rw,nosuid,nodev,noexec,relatime 0 2
LABEL=fs_var_log /var/log ext4 rw,relatime 0 3
LABEL=var_log_aud /var/log/audit ext4 rw,relatime 0 4
LABEL=fs_home /home ext4 rw,nodev,relatime 0 2
LABEL=fs_opt /opt ext4 rw,nodev,relatime 0 2
LABEL=fs_tmp /tmp ext4 rw,nodev,nosuid,noexec,relatime 0 2
tmpfs /dev/shm tmpfs nodev,nosuid,noexec 0 0

/proc/mounts

sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=3976556k,nr_inodes=994139,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/mapper/ubuntu_vg-lv_root / ext4 rw,seclabel,relatime,data=ordered 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12112 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0
/dev/mapper/ubuntu_vg-lv_home /home ext4 rw,seclabel,nodev,relatime,data=ordered 0 0
/dev/vda1 /boot ext4 rw,seclabel,relatime,data=ordered 0 0
/dev/mapper/ubuntu_vg-lv_opt /opt ext4 rw,seclabel,nodev,relatime,data=ordered 0 0
/dev/mapper/ubuntu_vg-lv_var /var ext4 rw,seclabel,relatime,data=ordered 0 0
/dev/mapper/ubuntu_vg-lv_var_tmp /var/tmp ext4 rw,seclabel,nosuid,nodev,noexec,relatime,data=ordered 0 0
/dev/mapper/ubuntu_vg-lv_var_log /var/log ext4 rw,seclabel,relatime,data=ordered 0 0
/dev/mapper/ubuntu_vg-lv_var_log_audit /var/log/audit ext4 rw,seclabel,relatime,data=ordered 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
/dev/mapper/ubuntu_vg-lv_tmp /tmp ext4 rw,seclabel,nodev,relatime,data=ordered 0 0
tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=800892k,mode=700,uid=1000,gid=1000 0 0

[gnanasalten]

@discordianfish can you please help on this ?

[dongjiang1989]

@gnanasalten Can you priovide your textfile in /var/lib/node_exporter/ ?

Maybe like:
https://github.com/prometheus-community/node-exporter-textfile-collector-scripts/blob/master/fstab-check.sh

[gnanasalten]

node_fstab_mount_status{filesystem="/"} 1
node_fstab_mount_status{filesystem="/boot/efi"} 1
node_fstab_mount_status{filesystem="/home"} 1
node_fstab_mount_status{filesystem="/opt"} 1
node_fstab_mount_status{filesystem="/tmp"} 1
node_fstab_mount_status{filesystem="/var"} 1
node_fstab_mount_status{filesystem="/var/log"} 1
node_fstab_mount_status{filesystem="/var/log/audit"} 1
node_fstab_mount_status{filesystem="/var/tmp"} 1
node_fstab_mount_status{filesystem="/dev/shm"} 1
node_syslog_err_count 0
node_syslog_bad_block_count 0

[dongjiang1989]

Different versions of the fstab collection plugin maybe used.

@gnanasalten Can you priovide your textfile script?

if you use fstab-check.sh script, mountpoint will appear in the tag.

node_fstab_mount_status{mountpoint='xxx'} 1

[SuperSandro2000]

node_exporter shouldn't fail that loudly when two mountpoints have the same path. That is a totally valid to do on linux.

[discordianfish]

@SuperSandro2000 It should not but I don't know if this is what is going on here

[jerviscui]

A similar problem happened to me.
I am using wsl2 and starting systemd.

node_exporter starts and issues the following error:

ts=2024-04-19T05:09:48.730Z caller=stdlib.go:105 level=error msg="error gathering metrics: 6 error(s) occurred:
* [from Gatherer #2] collected metric \"node_filesystem_device_error\" { label:{name:\"device\"  value:\"none\"}  label:{name:\"fstype\"  value:\"tmpfs\"}  label:{name:\"mountpoint\"  value:\"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1/run/user\"}  gauge:{value:1}} was collected before with the same name and label values
* [from Gatherer #2] collected metric \"node_filesystem_device_error\" { label:{name:\"device\"  value:\"none\"}  label:{name:\"fstype\"  value:\"tmpfs\"}  label:{name:\"mountpoint\"  value:\"/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1/run/user\"}  gauge:{value:1}} was collected before with the same name and label values
* [from Gatherer #2] collected metric \"node_filesystem_device_error\" { label:{name:\"device\"  value:\"none\"}  label:{name:\"fstype\"  value:\"tmpfs\"}  label:{name:\"mountpoint\"  value:\"/parent-distro/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1/run/user\"}  gauge:{value:1}} was collected before with the same name and label values
* [from Gatherer #2] collected metric \"node_filesystem_readonly\" { label:{name:\"device\"  value:\"none\"}  label:{name:\"fstype\"  value:\"tmpfs\"}  label:{name:\"mountpoint\"  value:\"/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1/run/user\"}  gauge:{value:0}} was collected before with the same name and label values
* [from Gatherer #2] collected metric \"node_filesystem_readonly\" { label:{name:\"device\"  value:\"none\"}  label:{name:\"fstype\"  value:\"tmpfs\"}  label:{name:\"mountpoint\"  value:\"/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1/run/user\"}  gauge:{value:0}} was collected before with the same name and label values
* [from Gatherer #2] collected metric \"node_filesystem_readonly\" { label:{name:\"device\"  value:\"none\"}  label:{name:\"fstype\"  value:\"tmpfs\"}  label:{name:\"mountpoint\"  value:\"/parent-distro/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1/run/user\"}  gauge:{value:0}} was collected before with the same name and label values"

[jerviscui]

And no such directory exists on my system

/run/desktop/...
/parent-distro/...
/mnt/host/...

[discordianfish]

Hrm label:{name:\"device\" value:\"none\"} looks suspicious. Is there anything else in the log that would point to issue tretrieving the device? @SuperQ any ideas?