Smart percentage_used alert should be clear that nvme device is reaching its end of life

[przemeklal]

It's easy to misunderstand the percentage_used nvme alert. The hw-health charm config has a pretty good description:

  Set thresholds for nvme percentage used check. Defaults to 80% warning threshold
  and 90% critical threshold. Percentage Used contains a vendor specific estimate
  of the percentage of NVM subsystem life used based on the actual usage and the
  manufacturer’s prediction of NVM life. A value of 100 indicates that the estimated
  endurance of the NVM in the NVM subsystem has been consumed, but may not indicate
  an NVM subsystem failure.

The alert should suggest that the issue is related to filesystem getting full, rather than the NVME being close to death. E.g. Intel Optane drivers start to throttle and report read-only mode after hitting 105% percentage_used, which means they've exceeded their expected lifetime.

Something like: "nvme drive is close to reaching its estimated lifetime" would help.

[Pjack]

We don't have alert for nvme in hw-observer now. Do you submit it in wrong place?

[przemeklal]

This is more of a feature request. I know this alert doesn't exist (yet) and I wanted to capture this requirement for when it is reimplemented in hw-observer.

[Pjack]

Got it . This is a missing part in hardware-observer.
We will check if the related metrics available in grafana-agent (node-exporter) or not. If yes, we will add alert there, if not, we could implement it in hardware-observer. thanks for the reminder.

[jneo8]

Example to get the metrics:

• https://github.com/prometheus-community/node-exporter-textfile-collector-scripts/blob/master/nvme_metrics.py
• Add nvme_metrics.sh text collector example prometheus/node_exporter#1309

Currently it's a missing part in node exporter.
We can consider to implement it on hardware-observer or use textfile-collector(I remember it's a missing part on grafana-agent charm and I prefer to use exporter).

[jneo8]

Need to check if https://github.com/prometheus-community/smartctl_exporter has this.