Qos latency seeing spike

[summertony15]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please let us know in a comment

Problem

When using "qos_latency" metric in harvest, there is a peak latency from 0 to over 3s showing in the dashboard, and the spike last for around three minutes.

This spike happens around three times a day.
From the AIQUM, we didn't see this peak value in the performance dashboard.
And we opened a case, and from the PA dashboard, there didn't have a spike value either,
wondering could be the bug for the harvest metric.

Another cluster that is using an older version of harvest is using metric "qos_volume_latency", and also ran into the same issue, with the peak latency to over 3000s.

This spike also lasts for around three minutes.

Another example here is the volume "VM_ISCSI_DS1" latency dashboard in AIQUM and harvest for 24 hours,
In AIQUM:

In Harvest with "qos_latency" metric:

Configuration

No response

Poller

cluster1

Version

23.11.0

Poller logs

No response

OS and platform

CentOS7.9.2009

ONTAP or StorageGRID version

9.11.1

Additional Context

No response

References

No response

[summertony15]

open a new ticket related to qos_latency spike value #2597

[rahulguptajss]

verified in main and long running setup.