-
Notifications
You must be signed in to change notification settings - Fork 364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Ray Observability] Disk usage in Dashboard #1152
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. Curious why this is opt-in instead of enabled by default (question for the Ray side I guess)
I found ray-project/ray#24416 and ray-project/ray#14676.
CC: @rkooo567 @DmitriGekhtman, could you please provide some insight on whether enabling disk usage for Kubernetes Pods by default would be advisable or not? Thanks! |
Also cc @scottsun94 |
IIUC, the problem with enabling it by default is: when multiple ray pods share the same k8s node, all these ray pods show the disk usage of that node. People might think that they are independent which might lead to confusion. I will still vote for enabling it by default:
|
Yeah I also think it is better enabling it, but we should have some tooltip or UI improvement to make sure people understand the problem. What @scottsun94 said is correct (disk usage in k8s is not set per pod, so the information could be misleading).
I don't think we have such best practices IIRC (pr does kuberay use daemonset to achieve this? @kevin85421 ). |
Thank @scottsun94 @rkooo567 for the reply.
Open an issue to track the progress ray-project/ray#36362.
Yes, this is the best practice, but a lot of users do not follow it, so the issue is still there.
No, KubeRay does not use DaemonSet. |
[Ray Observability] Disk usage in Dashboard
Why are these changes needed?
Related slack thread: https://ray-distributed.slack.com/archives/C01DLHZHRBJ/p1686170239143299
Without this PR, the values in the column
![Screen Shot 2023-06-08 at 1 30 04 PM](https://private-user-images.githubusercontent.com/20109646/244562200-def4800e-e449-4e64-9d67-2550119a905d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU0NTc4NDgsIm5iZiI6MTcyNTQ1NzU0OCwicGF0aCI6Ii8yMDEwOTY0Ni8yNDQ1NjIyMDAtZGVmNDgwMGUtZTQ0OS00ZTY0LTlkNjctMjU1MDExOWE5MDVkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA0VDEzNDU0OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBmMWI5MTQ0YzYwZGY4YTc1NTlhMTc4NjViMWIyOGY4YTliNmEwZWQ5ZTg3OWM1YjAzZDM1YjQxMDRiYjQ5YjAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.t6sBBoJk5uMje8af9ziP7NUiGIiz193YvMQTV4Cf3lo)
Disk(root)
are0.0000B/1.0000B
.With this PR, everything works fine.
![Screen Shot 2023-06-08 at 5 43 37 PM](https://private-user-images.githubusercontent.com/20109646/244561925-eaed5952-e2f5-4faa-a78b-cfcc6233598a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU0NTc4NDgsIm5iZiI6MTcyNTQ1NzU0OCwicGF0aCI6Ii8yMDEwOTY0Ni8yNDQ1NjE5MjUtZWFlZDU5NTItZTJmNS00ZmFhLWE3OGItY2ZjYzYyMzM1OThhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA0VDEzNDU0OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEzOTkyZDhmODRlYmQ0ODQ3MDVkMDc0OWQ5MTZhY2NkYWNmNGM2OTY4MjY1YjU0NzQxNWYyYTRiMTM0Yjk2NTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.pXPeykSq-ULra833VBbzrhRDc3Tj46oS0dwlfSTSog4)
Related issue number
Checks