Node Exporter 1.5.0 causing "Error Information Log Entries" increase in NVMe SMART data

[got3nks]

node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)
  build user:       root@6e7732a7b81b
  build date:       20221129-18:59:09
  go version:       go1.19.3
  platform:         linux/amd64

Each iteration of Node Exporter is causing the Error Information Log Entries to increase by two in the NVMe's SMART data.

# smartctl -a /dev/nvme0n1
Critical Warning:                   0x00
Temperature:                        53 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
...
Power On Hours:                     4,252
Unsafe Shutdowns:                   9
Media and Data Integrity Errors:    0
Error Information Log Entries:      1,548,554
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
...

I tried to experiment with the parameters a bit --collector.diskstats.device-exclude=nvme0 / --no-collector.nvme but I'm still noticing an increase of 2 at every iteration. As a confirmation, after stopping the Node Exporter process the Error Information Log Entries value stops growing.

# /usr/local/bin/node_exporter --log.level=debug
ts=2023-03-26T01:00:47.757Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)"
ts=2023-03-26T01:00:47.757Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)"
ts=2023-03-26T01:00:47.757Z caller=node_exporter.go:183 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
ts=2023-03-26T01:00:47.757Z caller=node_exporter.go:186 level=debug msg="Go MAXPROCS" procs=1
ts=2023-03-26T01:00:47.757Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2023-03-26T01:00:47.757Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2023-03-26T01:00:47.757Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=arp
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=bcache
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=bonding
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=btrfs
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=conntrack
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=cpu
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=cpufreq
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=diskstats
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=dmi
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=edac
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=entropy
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=fibrechannel
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=filefd
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=filesystem
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=hwmon
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=infiniband
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=ipvs
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=loadavg
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=mdadm
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=meminfo
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=netclass
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=netdev
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=netstat
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=nfs
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=nfsd
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=nvme
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=os
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=powersupplyclass
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=pressure
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=rapl
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=schedstat
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=selinux
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=sockstat
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=softnet
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=stat
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=tapestats
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=textfile
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=thermal_zone
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=time
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=timex
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=udp_queues
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=uname
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=vmstat
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=xfs
ts=2023-03-26T01:00:47.758Z caller=node_exporter.go:117 level=info collector=zfs
ts=2023-03-26T01:00:47.758Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100
ts=2023-03-26T01:00:47.758Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100
ts=2023-03-26T01:00:49.158Z caller=node_exporter.go:78 level=debug msg="collect query:" filters="unsupported value type"
ts=2023-03-26T01:00:49.159Z caller=collector.go:173 level=debug msg="collector succeeded" name=softnet duration_seconds=7.3325e-05
ts=2023-03-26T01:00:49.159Z caller=infiniband_linux.go:119 level=debug collector=infiniband msg="infiniband statistics not found, skipping"
ts=2023-03-26T01:00:49.159Z caller=collector.go:167 level=debug msg="collector returned no data" name=infiniband duration_seconds=4.0584e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.159Z caller=collector.go:173 level=debug msg="collector succeeded" name=stat duration_seconds=0.00023361
ts=2023-03-26T01:00:49.159Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=8.064e-06
ts=2023-03-26T01:00:49.159Z caller=collector.go:173 level=debug msg="collector succeeded" name=dmi duration_seconds=1.3722e-05
ts=2023-03-26T01:00:49.159Z caller=collector.go:173 level=debug msg="collector succeeded" name=schedstat duration_seconds=9.9374e-05
ts=2023-03-26T01:00:49.159Z caller=bonding_linux.go:64 level=debug collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net
ts=2023-03-26T01:00:49.159Z caller=collector.go:167 level=debug msg="collector returned no data" name=bonding duration_seconds=4.3633e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.159Z caller=nfs_linux.go:103 level=debug collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory"
ts=2023-03-26T01:00:49.159Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfs duration_seconds=3.6933e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.160Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpu duration_seconds=0.000531816
ts=2023-03-26T01:00:49.160Z caller=collector.go:173 level=debug msg="collector succeeded" name=rapl duration_seconds=0.000214019
ts=2023-03-26T01:00:49.160Z caller=collector.go:173 level=debug msg="collector succeeded" name=edac duration_seconds=1.8176e-05
ts=2023-03-26T01:00:49.160Z caller=nfsd_linux.go:69 level=debug collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory"
ts=2023-03-26T01:00:49.160Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfsd duration_seconds=2.1169e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.160Z caller=collector.go:173 level=debug msg="collector succeeded" name=vmstat duration_seconds=0.000133747
ts=2023-03-26T01:00:49.160Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs
ts=2023-03-26T01:00:49.160Z caller=zfs.go:73 level=debug collector=zfs err="ZFS / ZFS statistics are not available"
ts=2023-03-26T01:00:49.160Z caller=collector.go:167 level=debug msg="collector returned no data" name=zfs duration_seconds=2.2639e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.160Z caller=tapestats_linux.go:129 level=debug collector=tapestats msg="scsi_tape stats not found, skipping"
ts=2023-03-26T01:00:49.160Z caller=collector.go:167 level=debug msg="collector returned no data" name=tapestats duration_seconds=1.2659e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.161Z caller=collector.go:173 level=debug msg="collector succeeded" name=nvme duration_seconds=5.7868e-05
ts=2023-03-26T01:00:49.161Z caller=collector.go:173 level=debug msg="collector succeeded" name=mdadm duration_seconds=2.0072e-05
ts=2023-03-26T01:00:49.161Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=0 load=2.06
ts=2023-03-26T01:00:49.161Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=1 load=2.11
ts=2023-03-26T01:00:49.161Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=2 load=2.2
ts=2023-03-26T01:00:49.161Z caller=collector.go:173 level=debug msg="collector succeeded" name=loadavg duration_seconds=3.9813e-05
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/pstore
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/firmware/efi/efivars
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/bpf
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc/sys/fs/binfmt_misc
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/tracing
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/fuse/connections
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/run/credentials/systemd-sysusers.service
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=squashfs
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=squashfs
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=squashfs
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=squashfs
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=squashfs
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=squashfs
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc/sys/fs/binfmt_misc
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=nsfs
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/2af3311293529ff4a255aebc19bae9c373a3869d422d2913826d6951229f65c9/merged
ts=2023-03-26T01:00:49.161Z caller=filesystem_linux.go:59 level=debug collector=filesystem msg="Ignoring fs" type=nsfs
ts=2023-03-26T01:00:49.161Z caller=collector.go:173 level=debug msg="collector succeeded" name=filesystem duration_seconds=0.000412723
ts=2023-03-26T01:00:49.161Z caller=collector.go:173 level=debug msg="collector succeeded" name=powersupplyclass duration_seconds=0.000263269
ts=2023-03-26T01:00:49.161Z caller=collector.go:173 level=debug msg="collector succeeded" name=btrfs duration_seconds=9.1371e-05
ts=2023-03-26T01:00:49.162Z caller=collector.go:173 level=debug msg="collector succeeded" name=udp_queues duration_seconds=0.000128837
ts=2023-03-26T01:00:49.162Z caller=collector.go:173 level=debug msg="collector succeeded" name=sockstat duration_seconds=5.6732e-05
ts=2023-03-26T01:00:49.162Z caller=collector.go:173 level=debug msg="collector succeeded" name=selinux duration_seconds=1.8721e-05
ts=2023-03-26T01:00:49.162Z caller=meminfo.go:54 level=debug collector=meminfo msg="Set node_mem" memInfo="unsupported value type"
ts=2023-03-26T01:00:49.162Z caller=collector.go:173 level=debug msg="collector succeeded" name=meminfo duration_seconds=0.000133845
ts=2023-03-26T01:00:49.162Z caller=collector.go:173 level=debug msg="collector succeeded" name=xfs duration_seconds=1.0969e-05
ts=2023-03-26T01:00:49.162Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=cpu
ts=2023-03-26T01:00:49.162Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=io
ts=2023-03-26T01:00:49.162Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=memory
ts=2023-03-26T01:00:49.162Z caller=collector.go:173 level=debug msg="collector succeeded" name=pressure duration_seconds=7.917e-05
ts=2023-03-26T01:00:49.163Z caller=collector.go:173 level=debug msg="collector succeeded" name=netstat duration_seconds=0.000467337
ts=2023-03-26T01:00:49.163Z caller=fibrechannel_linux.go:113 level=debug collector=fibrechannel msg="fibrechannel statistics not found, skipping"
ts=2023-03-26T01:00:49.163Z caller=collector.go:167 level=debug msg="collector returned no data" name=fibrechannel duration_seconds=1.2075e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.164Z caller=conntrack_linux.go:157 level=debug collector=conntrack msg="conntrack probably not loaded"
ts=2023-03-26T01:00:49.164Z caller=collector.go:167 level=debug msg="collector returned no data" name=conntrack duration_seconds=3.4392e-05 err="collector returned no data"
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=timex duration_seconds=8.236e-06
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=arp duration_seconds=2.8395e-05
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=filefd duration_seconds=1.5349e-05
ts=2023-03-26T01:00:49.164Z caller=os_release.go:146 level=debug collector=os msg="file modification time has changed" file=//etc/os-release old_value=0001-01-01T00:00:00Z new_value=2022-07-28T17:29:51+02:00
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=os duration_seconds=5.743e-05
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=diskstats duration_seconds=0.000164119
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=textfile duration_seconds=4.504e-06
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=bcache duration_seconds=6.019e-06
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=entropy duration_seconds=2.5907e-05
ts=2023-03-26T01:00:49.164Z caller=time.go:73 level=debug collector=time msg="Return time" now=1.6797924491645694e+09
ts=2023-03-26T01:00:49.164Z caller=time.go:75 level=debug collector=time msg="Zone offset" offset=7200 time_zone=CEST
ts=2023-03-26T01:00:49.164Z caller=time_linux.go:38 level=debug collector=time msg="in Update" clocksources="[{0 [tsc hpet acpi_pm] tsc}]"
ts=2023-03-26T01:00:49.164Z caller=collector.go:173 level=debug msg="collector succeeded" name=time duration_seconds=6.055e-05
ts=2023-03-26T01:00:49.166Z caller=ipvs_linux.go:146 level=debug collector=ipvs msg="ipvs collector metrics are not available for this system"
ts=2023-03-26T01:00:49.166Z caller=collector.go:167 level=debug msg="collector returned no data" name=ipvs duration_seconds=0.003813326 err="collector returned no data"
ts=2023-03-26T01:00:49.167Z caller=collector.go:173 level=debug msg="collector succeeded" name=netdev duration_seconds=0.004513598
ts=2023-03-26T01:00:49.167Z caller=collector.go:173 level=debug msg="collector succeeded" name=netclass duration_seconds=0.004997598
ts=2023-03-26T01:00:49.168Z caller=collector.go:173 level=debug msg="collector succeeded" name=thermal_zone duration_seconds=0.004920199
ts=2023-03-26T01:00:49.171Z caller=collector.go:173 level=debug msg="collector succeeded" name=hwmon duration_seconds=0.009453131
ts=2023-03-26T01:00:49.187Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpufreq duration_seconds=0.02797438

[SuperQ]

This sounds like a hardware bug, I am unable to reproduce this on a system with NVMe.

• What are the errors in the log?
• What model system?
• What model SSD?

[got3nks]

Yes I confirm this does not happen on every hardware but it's hardware specific. The hardware is based on Intel NUC 11. Another Intel NUC 11 with a different NVMe doesn't have any issue.

But I'm looking for a way of stopping this from happening.

# uname -r
5.15.0-58-generic
# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

# smartctl -a /dev/nvme0n1
=== START OF INFORMATION SECTION ===
Model Number:                       KINGSTON SKC3000D4096G
Serial Number:                      50026B7282B2BA6C
Firmware Version:                   EIFK31.6
PCI Vendor/Subsystem ID:            0x2646
IEEE OUI Identifier:                0x0026b7
Total NVM Capacity:                 4,096,805,658,624 [4.09 TB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          4,096,805,658,624 [4.09 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            0026b7 282b2ba6c5
Local Time is:                      Sun Mar 26 12:04:28 2023 CEST
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005d):     Comp DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x08):         Telmtry_Lg
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     84 Celsius
Critical Comp. Temp. Threshold:     89 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.80W       -        -    0  0  0  0        0       0
 1 +     7.10W       -        -    1  1  1  1        0       0
 2 +     5.20W       -        -    2  2  2  2        0       0
 3 -   0.0620W       -        -    3  3  3  3     2500    7500
 4 -   0.0620W       -        -    4  4  4  4     2500    7500

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

# nvme error-log /dev/nvme0n1
Error Log Entries for device:nvme0n1 entries:63
.................
 Entry[ 0]
.................
error_count	: 1561452
sqid		: 0
cmdid		: 0x4004
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 1]
.................
error_count	: 1561451
sqid		: 0
cmdid		: 0x3007
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 2]
.................
error_count	: 1561450
sqid		: 0
cmdid		: 0x4003
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 3]
.................
error_count	: 1561449
sqid		: 0
cmdid		: 0x4002
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 4]
.................
error_count	: 1561448
sqid		: 0
cmdid		: 0xf01c
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 5]
.................
error_count	: 1561447
sqid		: 0
cmdid		: 0xe01d
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 6]
.................
error_count	: 1561446
sqid		: 0
cmdid		: 0x6019
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 7]
.................
error_count	: 1561445
sqid		: 0
cmdid		: 0x6018
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 8]
.................
error_count	: 1561444
sqid		: 0
cmdid		: 0x2006
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[ 9]
.................
error_count	: 1561443
sqid		: 0
cmdid		: 0x2005
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[10]
.................
error_count	: 1561442
sqid		: 0
cmdid		: 0x3001
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[11]
.................
error_count	: 1561441
sqid		: 0
cmdid		: 0x3000
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[12]
.................
error_count	: 1561440
sqid		: 0
cmdid		: 0xf014
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[13]
.................
error_count	: 1561439
sqid		: 0
cmdid		: 0xe017
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[14]
.................
error_count	: 1561438
sqid		: 0
cmdid		: 0x1006
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[15]
.................
error_count	: 1561437
sqid		: 0
cmdid		: 0x1005
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[16]
.................
error_count	: 1561436
sqid		: 0
cmdid		: 0x3009
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[17]
.................
error_count	: 1561435
sqid		: 0
cmdid		: 0x3008
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[18]
.................
error_count	: 1561434
sqid		: 0
cmdid		: 0x2001
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[19]
.................
error_count	: 1561433
sqid		: 0
cmdid		: 0x2000
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[20]
.................
error_count	: 1561432
sqid		: 0
cmdid		: 0x100a
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[21]
.................
error_count	: 1561431
sqid		: 0
cmdid		: 0x1009
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[22]
.................
error_count	: 1561430
sqid		: 0
cmdid		: 0xc017
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[23]
.................
error_count	: 1561429
sqid		: 0
cmdid		: 0xc016
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[24]
.................
error_count	: 1561428
sqid		: 0
cmdid		: 0x2
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[25]
.................
error_count	: 1561427
sqid		: 0
cmdid		: 0x1
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[26]
.................
error_count	: 1561426
sqid		: 0
cmdid		: 0xf000
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[27]
.................
error_count	: 1561425
sqid		: 0
cmdid		: 0xe003
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[28]
.................
error_count	: 1561424
sqid		: 0
cmdid		: 0xd002
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[29]
.................
error_count	: 1561423
sqid		: 0
cmdid		: 0xd001
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[30]
.................
error_count	: 1561422
sqid		: 0
cmdid		: 0x9
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[31]
.................
error_count	: 1561421
sqid		: 0
cmdid		: 0x8
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[32]
.................
error_count	: 1561420
sqid		: 0
cmdid		: 0xc000
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[33]
.................
error_count	: 1561419
sqid		: 0
cmdid		: 0xb003
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[34]
.................
error_count	: 1561418
sqid		: 0
cmdid		: 0xe00a
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[35]
.................
error_count	: 1561417
sqid		: 0
cmdid		: 0xe009
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[36]
.................
error_count	: 1561416
sqid		: 0
cmdid		: 0xf005
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[37]
.................
error_count	: 1561415
sqid		: 0
cmdid		: 0xf004
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[38]
.................
error_count	: 1561414
sqid		: 0
cmdid		: 0x4018
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[39]
.................
error_count	: 1561413
sqid		: 0
cmdid		: 0x301b
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[40]
.................
error_count	: 1561412
sqid		: 0
cmdid		: 0x301a
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[41]
.................
error_count	: 1561411
sqid		: 0
cmdid		: 0x3019
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[42]
.................
error_count	: 1561410
sqid		: 0
cmdid		: 0x3018
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[43]
.................
error_count	: 1561409
sqid		: 0
cmdid		: 0x201b
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[44]
.................
error_count	: 1561408
sqid		: 0
cmdid		: 0xb01c
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[45]
.................
error_count	: 1561407
sqid		: 0
cmdid		: 0xa01d
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[46]
.................
error_count	: 1561406
sqid		: 0
cmdid		: 0x701d
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[47]
.................
error_count	: 1561405
sqid		: 0
cmdid		: 0x701c
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[48]
.................
error_count	: 1561404
sqid		: 0
cmdid		: 0x8001
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[49]
.................
error_count	: 1561403
sqid		: 0
cmdid		: 0xc00b
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[50]
.................
error_count	: 1561402
sqid		: 0
cmdid		: 0x1019
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[51]
.................
error_count	: 1561401
sqid		: 0
cmdid		: 0x1018
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[52]
.................
error_count	: 1561400
sqid		: 0
cmdid		: 0xb014
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[53]
.................
error_count	: 1561399
sqid		: 0
cmdid		: 0xa017
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[54]
.................
error_count	: 1561398
sqid		: 0
cmdid		: 0xf01a
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[55]
.................
error_count	: 1561397
sqid		: 0
cmdid		: 0xf019
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[56]
.................
error_count	: 1561396
sqid		: 0
cmdid		: 0xb00b
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[57]
.................
error_count	: 1561395
sqid		: 0
cmdid		: 0xb00a
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[58]
.................
error_count	: 1561394
sqid		: 0
cmdid		: 0xa008
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[59]
.................
error_count	: 1561393
sqid		: 0
cmdid		: 0x900b
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[60]
.................
error_count	: 1561392
sqid		: 0
cmdid		: 0x401c
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[61]
.................
error_count	: 1561391
sqid		: 0
cmdid		: 0x301d
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................
 Entry[62]
.................
error_count	: 1561390
sqid		: 0
cmdid		: 0x8009
status_field	: 0x2002(INVALID_FIELD: A reserved coded value or an unsupported value in a defined field)
phase_tag	: 0
parm_err_loc	: 0x102c
lba		: 0
nsid		: 0
vs		: 0
trtype		: The transport type is not indicated or the error is not transport related.
cs		: 0
trtype_spec_info: 0
.................

[SuperQ]

Sorry, this is likely a hardware bug. There's not much we can do here.

[got3nks]

Isn't there any way I can disable the NVMe data exporting by node_exporter?

[got3nks]

--no-collector.hwmon stopped the increase of Error Information Log Entries. Any suggestions to debug it further?

[SuperQ]

The hwmon collector access files via /sys/class/hwmon. There is a kernel module loaded that maps something to your NVMe device.

ls -l /sys/class/hwmon/

[got3nks]

Yes the nvme is mapped in hwmon1. Can I disable it somehow from the collector without disabling the entire hwmon collector?

Which files are being read by node_exporter? So I can try all of them and see which one is triggering the NVMe SMART error.

lrwxrwxrwx 1 root root 0 Mar 24 12:01 hwmon0 -> ../../devices/virtual/thermal/thermal_zone0/hwmon0
lrwxrwxrwx 1 root root 0 Mar 24 12:01 hwmon1 -> ../../devices/pci0000:00/0000:00:06.0/0000:01:00.0/nvme/nvme0/hwmon1
lrwxrwxrwx 1 root root 0 Mar 24 12:01 hwmon2 -> ../../devices/virtual/thermal/thermal_zone2/hwmon2
lrwxrwxrwx 1 root root 0 Mar 24 12:01 hwmon3 -> ../../devices/platform/coretemp.0/hwmon/hwmon3
lrwxrwxrwx 1 root root 0 Mar 24 12:01 hwmon4 -> ../../devices/platform/USBC000:00/power_supply/ucsi-source-psy-USBC000:001/hwmon4
lrwxrwxrwx 1 root root 0 Mar 24 12:01 hwmon5 -> ../../devices/platform/USBC000:00/power_supply/ucsi-source-psy-USBC000:002/hwmon5

[SuperQ]

No, there is not. Also I don't see any quick way to disable hwmon for nvme. It seems to be built-in to the nvme module itself.

You could try linux-image-generic-hwe-22.04 to see if a newer kernel solves the NVMe bug. But otherwise there's not much you can do other than replace your NVMe with one that is not buggy.

[got3nks]

Found what files are causing the I/O errors:

cat: /sys/class/hwmon/hwmon1/temp3_max: Input/output error
cat: /sys/class/hwmon/hwmon1/temp3_min: Input/output error

I'll probably have to edit the hwmon collector source code to exclude these and build it.

[SuperQ]

We could consider adding device include/exclude flags to the hwmon collector. Take a look at arp_linux.go for an example of how to do this. Would accept a PR for this.

[got3nks]

As a quick fix I edited the hwmon_linux.go file:

func collectSensorData(dir string, data map[string]map[string]string) error {
	sensorFiles, dirError := os.ReadDir(dir)
	if dirError != nil {
		return dirError
	}
	for _, file := range sensorFiles {
		filename := file.Name()
		ok, sensorType, sensorNum, sensorProperty := explodeSensorFilename(filename)
		if !ok {
			continue
		}

		for _, t := range hwmonSensorTypes {
			if t == sensorType {
				// skip /sys/class/hwmon/hwmon1/temp3_min and /sys/class/hwmon/hwmon1/temp3_max
				if dir == "/sys/class/hwmon/hwmon1" && 
				   sensorType == "temp" && 
				   sensorNum == 3 && 
				   (sensorProperty == "min" || sensorProperty == "max") {
					continue
				}

				addValueFile(data, sensorType+strconv.Itoa(sensorNum), sensorProperty, filepath.Join(dir, file.Name()))
				break
			}
		}
	}
	return nil
}