error gathering metrics: collected metric \"apt_upgrades_pending\" ... was collected before with the same name and label values

[arno01]

Host operating system:

Linux fra-pg-01 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version:

$ /usr/bin/prometheus-node-exporter --version
node_exporter, version 0.17.0+ds (branch: debian/sid, revision: 0.17.0+ds-2)
  build user:       pkg-go-maintainers@lists.alioth.debian.org
  build date:       20181214-14:09:27
  go version:       go1.10.4

node_exporter command line flags

$ cat /etc/default/prometheus-node-exporter
# Set the command-line arguments to pass to the server.
# Due to shell escaping, to pass backslashes for regexes, you need to double
# them (\\d for \d). If running under systemd, you need to double them again
# (\\\\d to mean \d), and escape newlines too.
ARGS="--collector.diskstats.ignored-devices=^(ram|loop|fd)\\\\d+$ \
      --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run|var/lib/docker/overlay2/[a-z0-9]{64}/merged|var/lib/docker/aufs/mnt/[a-z0-9]{64}|var/lib/docker/devicemapper/mnt/[a-z0-9]{64}|var/lib/docker/containers/[a-z0-9]{64}/shm|var/lib/docker/containers/[a-z0-9]{64}/mounts/shm|var/lib/docker/plugins/[a-z0-9]{64})($|/) \
      --collector.netdev.ignored-devices=^lo$ \
      --collector.textfile.directory=/var/lib/prometheus/node-exporter \
      --collector.ntp \
      --no-collector.hwmon"

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

Didn't do anything particular, just looking at the logs.

What did you expect to see?

No error.

What did you see instead?

The following error:

$ systemctl status prometheus-node-exporter --no-pager -l
● prometheus-node-exporter.service - Prometheus exporter for machine metrics
   Loaded: loaded (/lib/systemd/system/prometheus-node-exporter.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-05-02 15:32:02 UTC; 3 months 27 days ago
     Docs: https://github.com/prometheus/node_exporter
 Main PID: 2235 (prometheus-node)
    Tasks: 74
   Memory: 25.7M
      CPU: 21h 7min 57.439s
   CGroup: /system.slice/prometheus-node-exporter.service
           └─2235 /usr/bin/prometheus-node-exporter --collector.diskstats.ignored-devices=^(ram|loop|fd)\d+$ --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run|var/lib/docker/overlay2/[a-z0-9]{64}/merged|var/lib/docker/aufs/mnt/[a-z0-9]{64}|var/lib/docker/devicemapper/mnt/[a-z0-9]{64}|var/lib/docker/containers/[a-z0-9]{64}/shm|var/lib/docker/containers/[a-z0-9]{64}/mounts/shm|var/lib/docker/plugins/[a-z0-9]{64})($|/) --collector.netdev.ignored-devices=^lo$ --collector.textfile.directory=/var/lib/prometheus/node-exporter --collector.ntp --no-collector.hwmon

Aug 29 12:05:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:05:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:05:36 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:05:36Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:05:39 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:05:39Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:06:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:06:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:07:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:07:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:08:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:08:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:09:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:09:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:10:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:10:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:11:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:11:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:12:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:12:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"

[pgier]

Check in your textfile directory, it sounds like you have a duplicate "apt_upgrades_pending" metric either in two different files or in the same file.

[arno01]

You are right, what should I do with that file -- is it generated automatically?

redacted:/var/lib/prometheus/node-exporter$ cat apt.prom
# HELP apt_upgrades_pending Apt package pending updates by origin.
# TYPE apt_upgrades_pending gauge
apt_upgrades_pending{origin="PostgreSQLfor",arch="Debian/Ubuntu"} 3
apt_upgrades_pending{origin="PostgreSQLfor",arch="Debian/Ubuntu"} 6
apt_upgrades_pending{origin="Ubuntu:16.04/xenial-updates",arch="all"} 29
apt_upgrades_pending{origin="Ubuntu:16.04/xenial-updates",arch="amd64"} 97
apt_upgrades_pending{origin="Ubuntu:16.04/xenial-updates,Ubuntu:16.04/xenial-security",arch="all"} 26
apt_upgrades_pending{origin="Ubuntu:16.04/xenial-updates,Ubuntu:16.04/xenial-security",arch="amd64"} 111
# HELP node_reboot_required Node reboot is required for software updates.
# TYPE node_reboot_required gauge
node_reboot_required 0

[arno01]

Ok, I figured that Ubuntu Developers set-up a prometheus-node-exporter-apt.timer systemd job which runs prometheus-node-exporter-apt.service every 15 minutes, which in turn runs /usr/share/prometheus-node-exporter/apt.sh script

So I guess the whole point would be to fix the /usr/share/prometheus-node-exporter/apt.sh script so it does not show the duplicates.

/usr/bin/apt-get --just-print upgrade \
  | /usr/bin/awk -F'[()]' \
      '/^Inst/ { sub("^[^ ]+ ", "", $2); sub("\\[", " ", $2);
                 sub(" ", "", $2); sub("\\]", "", $2); print $2 }' \
  | /usr/bin/sort \
  | /usr/bin/uniq -c \
  | awk '{ gsub(/\\\\/, "\\\\", $2); gsub(/\"/, "\\\"", $2);
           gsub(/\[/, "", $3); gsub(/\]/, "", $3);
           print "apt_upgrades_pending{origin=\"" $2 "\",arch=\"" $3 "\"} " $1}'

It outputs:

apt_upgrades_pending{origin="PostgreSQLfor",arch="Debian/Ubuntu"} 2
apt_upgrades_pending{origin="PostgreSQLfor",arch="Debian/Ubuntu"} 4
apt_upgrades_pending{origin="Ubuntu:18.04/bionic-updates",arch="all"} 19
apt_upgrades_pending{origin="Ubuntu:18.04/bionic-updates",arch="amd64"} 54
apt_upgrades_pending{origin="Ubuntu:18.04/bionic-updates,Ubuntu:18.04/bionic-security",arch="all"} 13
apt_upgrades_pending{origin="Ubuntu:18.04/bionic-updates,Ubuntu:18.04/bionic-security",arch="amd64"} 42
apt_upgrades_pending{origin="universe-updates/201907312020bionic:bionic",arch="all"} 2
apt_upgrades_pending{origin="universe-updates/201907312020bionic:bionic",arch="amd64"} 3

The raw data that the script processes (I took only the PostgreSQLfor parts which relate to the duplication):

$ /usr/bin/apt-get --just-print upgrade |grep 'PostgreSQL for'
Inst postgresql-common [199.pgdg18.04+1] (204.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [all]) []
Inst postgresql-client-common [199.pgdg18.04+1] (204.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [all]) []
Inst libpq5 [11.2-1.pgdg18.04+1] (11.5-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64])
Inst postgresql-contrib-9.6 [9.6.12-1.pgdg18.04+1] (9.6.15-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64]) []
Inst postgresql-client-9.6 [9.6.12-1.pgdg18.04+1] (9.6.15-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64]) []
Inst postgresql-9.6 [9.6.12-1.pgdg18.04+1] (9.6.15-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64])
Conf postgresql-common (204.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [all])
Conf postgresql-client-common (204.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [all])
Conf libpq5 (11.5-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64])
Conf postgresql-contrib-9.6 (9.6.15-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64])
Conf postgresql-client-9.6 (9.6.15-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64])
Conf postgresql-9.6 (9.6.15-1.pgdg18.04+1 PostgreSQL for Debian/Ubuntu repository:bionic-pgdg [amd64])

[arno01]

for master:
created an issue prometheus-community/node-exporter-textfile-collector-scripts#7
PR prometheus-community/node-exporter-textfile-collector-scripts#8