Add an efficient, user homedirectory size prometheus reporter #2621
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Merging this PR will trigger the following deployment actions. Support and Staging deployments
Production deployments
|
|
Experimental grafana dashboard on LEAP cluster: 
Lookin good. As it churns through this, it is being CPU limited at the 0.05 we have given it, which seems alright with me. |
pnasrat
suggested changes
Jun 6, 2023
yuvipanda
added a commit
to yuvipanda/grafana-dashboards
that referenced
this pull request
Jun 6, 2023
Monitoring each user's home directory size is an often requested feature for z2jh, and kinda important to prevent disk space related outages when using shared clusters. A single user can easily accidentally take up the space of an entire NFS share, and without an easy way to monitor this, can be hard to let them know before the disk is full. Admins could manually run `du` on an NFS server now and then, but this is time consuming and error prone. This also can hammer the server hard - millions of IO operations are performed as pretty much each file must be touched to calculate directory sizes. https://github.com/yuvipanda/prometheus-dirsize-exporter does this efficiently, primarily by allowing you to throttle how much IO operations per second it can make. This prevents the NFS server from being unsuable due to IO saturation, instead trading off latency of how up to date these stats are. As we don't need upto the minute stats here, this is acceptable. Test deployment at 2i2c-org/infrastructure#2621
|
@pnasrat the graph panel doesn't exist yet, I just made a PR to add it: jupyterhub/grafana-dashboards#80. I'll add some docs too. |
|
ah great. basically lgtm but will wait for your updates |
|
@pnasrat given the situation with utoronto and q planning, I won't be able to write docs this week. Are you ok with me merging this now so it can start collecting data, and I'll work on docs before end of the month? |
|
@yuvipanda As an async team documentation for new services including operational procedures (eg how to turn off) is necessary to ensure the team can both use and troubleshoot new services If you want to merge without docs, please add a specific feature flag enabled and test in a single staging hub cluster to prove it out rather than turn on everywhere |
|
Alright, I'll just mark this as draft and let this be until I have time to come back to it. |
This deploys https://github.com/yuvipanda/prometheus-dirsize-exporter, an *efficient* per-user homedirectory stats (size, no. of files, last modified date, etc) collector. It is capped at performing no more than 250 IO operations per second, to not overwhelm NFS servers. Metrics are refreshed every 2h after completion, although on large servers (like LEAP), they can take many many hours to complete with just 250 IO operations per second. This is perfectly fine though, as we do not need 'up to date' information. Trading off metric latency for minimal resource usage is pretty good here. Ref 2i2c-org#764
We don't have anyone using that!
|
I think the right place for documenting this is upstream, so I've finally opened jupyterhub/grafana-dashboards#84 to setup an actual docs site for the grafana dashboard. Once that's merged, I'll add a page documenting this panel. |
|
I've added documentation about this graph to jupyterhub/grafana-dashboards#84, so I think this is ready for review again. |
pnasrat
approved these changes
Jul 7, 2023
|
Thanks @pnasrat, and will do once I confirm data is coming in! |
|
🎉🎉🎉🎉 Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/5488684979 |
yuvipanda
added a commit
to yuvipanda/grafana-dashboards
that referenced
this pull request
Jul 25, 2024
Monitoring each user's home directory size is an often requested feature for z2jh, and kinda important to prevent disk space related outages when using shared clusters. A single user can easily accidentally take up the space of an entire NFS share, and without an easy way to monitor this, can be hard to let them know before the disk is full. Admins could manually run `du` on an NFS server now and then, but this is time consuming and error prone. This also can hammer the server hard - millions of IO operations are performed as pretty much each file must be touched to calculate directory sizes. https://github.com/yuvipanda/prometheus-dirsize-exporter does this efficiently, primarily by allowing you to throttle how much IO operations per second it can make. This prevents the NFS server from being unsuable due to IO saturation, instead trading off latency of how up to date these stats are. As we don't need upto the minute stats here, this is acceptable. Test deployment at 2i2c-org/infrastructure#2621
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This deploys https://github.com/yuvipanda/prometheus-dirsize-exporter, an efficient per-user homedirectory stats (size, no. of files, last modified date, etc) collector. It is capped at performing no more than 250 IO operations per second, to not overwhelm NFS servers. Metrics are refreshed every 2h after completion, although on large servers (like LEAP), they can take many many hours to complete with just 250 IO operations per second. This is perfectly fine though, as we do not need 'up to date' information. Trading off metric latency for minimal resource usage is pretty good here.
Ref #764