[Ray Observability] Disk usage in Dashboard

[kevin85421]

Why are these changes needed?

Related slack thread: https://ray-distributed.slack.com/archives/C01DLHZHRBJ/p1686170239143299

• Without this PR, the values in the column Disk(root) are 0.0000B/1.0000B.
  

• With this PR, everything works fine.
  

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[architkulkarni]

Looks good to me. Curious why this is opt-in instead of enabled by default (question for the Ray side I guess)

[kevin85421]

@architkulkarni

I found ray-project/ray#24416 and ray-project/ray#14676.

For example, when running two Ray pods on one physical K8s node, the limit shown for each is the disk limit of the K8s node and the limit of the K8s node gets counted twice in the "totals" section.

CC: @rkooo567 @DmitriGekhtman, could you please provide some insight on whether enabling disk usage for Kubernetes Pods by default would be advisable or not? Thanks!

[kevin85421]

#1152 (comment)

Also cc @scottsun94

[scottsun94]

IIUC, the problem with enabling it by default is: when multiple ray pods share the same k8s node, all these ray pods show the disk usage of that node. People might think that they are independent which might lead to confusion.

I will still vote for enabling it by default:

• I believe the best practice is to have each ray pod on a separate k8s node? (then this issue won't exist)
• not showing it makes it not useful at all
• we can probably add some tooltips or other UI treatments for disk usage column to explain this?

[rkooo567]

Yeah I also think it is better enabling it, but we should have some tooltip or UI improvement to make sure people understand the problem. What @scottsun94 said is correct (disk usage in k8s is not set per pod, so the information could be misleading).

I believe the best practice is to have each ray pod on a separate k8s node? (then this issue won't exist)

I don't think we have such best practices IIRC (pr does kuberay use daemonset to achieve this? @kevin85421 ).

[kevin85421]

Thank @scottsun94 @rkooo567 for the reply.

we can probably add some tooltips or other UI treatments for disk usage column to explain this?

Open an issue to track the progress ray-project/ray#36362.

I believe the best practice is to have each ray pod on a separate k8s node?

Yes, this is the best practice, but a lot of users do not follow it, so the issue is still there.

I don't think we have such best practices IIRC (pr does kuberay use daemonset to achieve this? @kevin85421 ).

No, KubeRay does not use DaemonSet.