mgr/dashboard: Compare values of MTU alert by device #45583
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aaSharma14
reviewed
Mar 24, 2022
p-se
force-pushed
the
monitoring-alert-mtu-group-by-devices
branch
from
a73f52d
to
7599770
Compare
|
jenkins test dashboard |
Fixes: https://tracker.ceph.com/issues/55004 Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
Fixes: https://tracker.ceph.com/issues/55004 Signed-off-by: Aashish Sharma <aasharma@redhat.com>
p-se
force-pushed
the
monitoring-alert-mtu-group-by-devices
branch
from
f06c7d0
to
49d6068
Compare
|
jenkins test dashboard |
aaSharma14
approved these changes
Mar 30, 2022
nizamial09
approved these changes
Mar 31, 2022
|
Does this PR need to go through Teuthology? |
epuertat
approved these changes
Apr 1, 2022
There was a problem hiding this comment.
LGTM! Thanks a lot @p-se !
We discussed at today's stand-up whether to improve the MTU alert in a follow-up PR and we agreed on:
- as per this PR the MTU alert will work as long as all hosts are using the same MTU setting for the same interface name across the cluster (i.e.:
eth0in all hosts has MTU 1500,eth1has MTU 9000, etc). - to make this work with the network address instead of with the interface name, we'd need to provide new metrics via mgr/prometheus (since node-exporter doesn't provide this info yet).
- based on upstream users and downstream customer experience so far, we think that it is fair to assume that Ceph users are generally configuring the same networks (cluster or public) on network interfaces whose names are the same.
- for those users don't following that rule, the impact of this alert would be minimal, as it could be easily muted from the Dashboard as soon as it would be first triggered misleadingly.
Comment on lines
+708
to
+718
| node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == | ||
| scalar( | ||
| max by (device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != | ||
| quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) | ||
| ) | ||
| or | ||
| node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == | ||
| scalar( | ||
| min by (device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != | ||
| quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) | ||
| ) |
There was a problem hiding this comment.
🤯 this is well beyond my PromQL literacy 🙈 well, I'll trust the unit tests, which seem to make sense!
There was a problem hiding this comment.
Took me a while to come up with it as well. As @aaSharma14 contributed the tests, found an issue and suggested the fix, some credit goes to him as well. So, thanks @aaSharma14 for the fruitful cooperation :)
14 tasks
This was referenced Apr 7, 2022
14 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: https://tracker.ceph.com/issues/55004
Signed-off-by: Patrick Seidensal pseidensal@suse.com
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windows