[dashboard][kubernetes] Show container's memory info on K8s, not the physical host's.

[DmitriGekhtman]

Why are these changes needed?

Shows container's memory info in the dashboard when using Kubernetes, not the physical host's.
K8s dashboard situation looks a little better:

(1 head, 2 min_workers all with 512Mi memory limit)

Related issue number

Addresses memory subproblem of
#11172

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[wuisawesome]

Why do we need the if statement? Shouldn't ray.utils.get_system_memory() already do all these checks?

If we have a good reason for deviating from how the core measures memory usage, we should carefully document it, especially now that we're auto-reporting memory resources within ray.available_resources().

Btw the issue also mentions CPU usage. Should this PR also include that?

[DmitriGekhtman]

If we have a good reason for deviating from how the core measures memory usage, we should carefully document it, especially now that we're auto-reporting memory resources within ray.available_resources().

I think psutil's definition of percent is different, but sure I'll get rid of the if statement.
I think with this change the percent reported for non-docker set-ups will be slightly lower.

I decided not to deal with CPU right now, as it seems slightly more subtle -- unless you (or anyone else) have a quick fix idea!

[wuisawesome]

ray.utils.get_num_cpus()? Or is the issue that you want to measure usage instead of the scheduler allocations?

[DmitriGekhtman]

ray.utils.get_num_cpus()? Or is the issue that you want to measure usage instead of the scheduler allocations?

When I run on minikube with 8 cpus allocated to minikube, attach to the head pod, and run ray.utils.get_num_cpus(), I get 8 instead of the 1 cpu request/limit I've assigned the pod.

[DmitriGekhtman]

We do want accurate usage and percent for CPU too.

[wuisawesome]

Isn't that a pretty big issue since it means Ray will incorrectly set num_cpus? Do we have an issue tracking that?

[DmitriGekhtman]

Isn't that a pretty big issue since it means Ray will incorrectly set num_cpus? Do we have an issue tracking that?

KubernetesNodeProvider reads from the pod spec: https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/kubernetes/config.py#L93

[DmitriGekhtman]

ray.utils.get_num_cpus()? Or is the issue that you want to measure usage instead of the scheduler allocations?

When I run on minikube with 8 cpus allocated to minikube, attach to the head pod, and run ray.utils.get_num_cpus(), I get 8 instead of the 1 cpu request/limit I've assigned the pod.

After looking into it more carefully -- it reads 8 because it does indeed use all 8 CPUs available to it. But it uses at most x CPU's worth of cycles per unit time where x is the limit specified in the pod spec.

[DmitriGekhtman]

I think the change in this PR should be pretty uncontroversial.

CPUs I've started looking at -- I'd like to isolate those in a different PR.

Got rid of the if, deferred CPU to another PR.

[wuisawesome]

Dealing with CPUs in a separate PR sounds good to me