You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please let us know in a comment
Problem
When using "qos_latency" metric in harvest, there is a peak latency from 0 to over 3s showing in the dashboard, and the spike last for around three minutes.
This spike happens around three times a day.
From the AIQUM, we didn't see this peak value in the performance dashboard.
And we opened a case, and from the PA dashboard, there didn't have a spike value either,
wondering could be the bug for the harvest metric.
Another cluster that is using an older version of harvest is using metric "qos_volume_latency", and also ran into the same issue, with the peak latency to over 3000s.
This spike also lasts for around three minutes.
Another example here is the volume "VM_ISCSI_DS1" latency dashboard in AIQUM and harvest for 24 hours,
In AIQUM:
In Harvest with "qos_latency" metric:
Configuration
No response
Poller
cluster1
Version
23.11.0
Poller logs
No response
OS and platform
CentOS7.9.2009
ONTAP or StorageGRID version
9.11.1
Additional Context
No response
References
No response
The text was updated successfully, but these errors were encountered:
A note for the community
Problem
When using "qos_latency" metric in harvest, there is a peak latency from 0 to over 3s showing in the dashboard, and the spike last for around three minutes.
![截圖 2023-12-11 16 03 34](https://private-user-images.githubusercontent.com/33056261/289464975-d4a1cc25-b8d2-49d5-b483-4968f3739462.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyMTY5OTUsIm5iZiI6MTcyNTIxNjY5NSwicGF0aCI6Ii8zMzA1NjI2MS8yODk0NjQ5NzUtZDRhMWNjMjUtYjhkMi00OWQ1LWI0ODMtNDk2OGYzNzM5NDYyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDE4NTEzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFhZjE0MGE1Zjc4NGFmZDA2ZmNiMjI4MTQyYTg3N2Y5MzBiOTExYzVjZTk2NTdkOGIyNzI3YzkwZmFjY2MxNGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.rKI97jYyvP5864UA1cYBpe694XRwgBV50AO8DvqMbqM)
This spike happens around three times a day.
From the AIQUM, we didn't see this peak value in the performance dashboard.
And we opened a case, and from the PA dashboard, there didn't have a spike value either,
wondering could be the bug for the harvest metric.
Another cluster that is using an older version of harvest is using metric "qos_volume_latency", and also ran into the same issue, with the peak latency to over 3000s.
![截圖 2023-12-11 16 25 47](https://private-user-images.githubusercontent.com/33056261/289470298-4080b1c6-4751-432e-886c-46bf294ccbd0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyMTY5OTUsIm5iZiI6MTcyNTIxNjY5NSwicGF0aCI6Ii8zMzA1NjI2MS8yODk0NzAyOTgtNDA4MGIxYzYtNDc1MS00MzJlLTg4NmMtNDZiZjI5NGNjYmQwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDE4NTEzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQwY2I0YjM5NDZjZDE3MzVjN2ZjYTdhZWUxYjdiM2VhMzhhM2QwODM1OTAzMmQxMWYzY2U3ODMzZTI0ZWYwN2EmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-SvxMsZTkVWJ5XPlNMgjDmJ1LBFeMiH0CFETsdaFzvg)
This spike also lasts for around three minutes.
Another example here is the volume "VM_ISCSI_DS1" latency dashboard in AIQUM and harvest for 24 hours,
![截圖 2023-12-11 16 28 10](https://private-user-images.githubusercontent.com/33056261/289470775-7a898f42-e296-4649-aaf9-d003e42484d7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyMTY5OTUsIm5iZiI6MTcyNTIxNjY5NSwicGF0aCI6Ii8zMzA1NjI2MS8yODk0NzA3NzUtN2E4OThmNDItZTI5Ni00NjQ5LWFhZjktZDAwM2U0MjQ4NGQ3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDE4NTEzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ3ZDcyMGQ5ZTAzYmUxNTU3OTU0ZDMyYTBjZTgwMWI4ZmM1MjlhNzAwOGExNmRkZjVjODY0NzkwZTE1NjJmOTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.LTb5Zb-AODbQXDqSN3jhXFA3bScYgSvWPOhBgBzvYjA)
![截圖 2023-12-11 16 30 41](https://private-user-images.githubusercontent.com/33056261/289471423-cdc5cf09-3695-4b05-a806-c166a68613e3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyMTY5OTUsIm5iZiI6MTcyNTIxNjY5NSwicGF0aCI6Ii8zMzA1NjI2MS8yODk0NzE0MjMtY2RjNWNmMDktMzY5NS00YjA1LWE4MDYtYzE2NmE2ODYxM2UzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDE4NTEzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTlmYWE4ZjdlOTVjNzg2MGNhYzllYzFmZjE2ZThkYzdiNDY0ZmM5Y2RiMDRiN2Q0ODQ2NjZlMTQ5M2Q2NWZhZWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.PSo3OdA5CZ8ru_RG2LFk0D-u8hd2X9c3F0GkF6jefGA)
In AIQUM:
In Harvest with "qos_latency" metric:
Configuration
No response
Poller
cluster1
Version
23.11.0
Poller logs
No response
OS and platform
CentOS7.9.2009
ONTAP or StorageGRID version
9.11.1
Additional Context
No response
References
No response
The text was updated successfully, but these errors were encountered: