[core] Add metrics for disk and network I/O #23546
Conversation
| @@ -348,19 +361,25 @@ def _get_load_avg(self): | |||
| per_cpu_load = tuple((round(x / self._cpu_counts[0], 2) for x in load)) | |||
| return load, per_cpu_load | |||
|
|
|||
| @staticmethod | |||
| def _compute_speed_from_hist(hist): | |||
| while len(hist) > 7: | |||
There was a problem hiding this comment.
Not sure, I just copied what was here already...
| ) | ||
| network_speed_stats = self._compute_speed_from_hist(self._network_stats_hist) | ||
|
|
||
| disk_stats = self._get_disk_stats() |
There was a problem hiding this comment.
there is only _get_disk_io function?
There was a problem hiding this comment.
Oops, thanks... clearly I did not actually run this :)
There was a problem hiding this comment.
Yeah let me look into that...
| @@ -651,10 +660,20 @@ bool PullManager::TryPinObject(const ObjectID &object_id) { | |||
| if (ref != nullptr) { | |||
| pinned_objects_size_ += ref->GetSize(); | |||
| pinned_objects_[object_id] = std::move(ref); | |||
| return false; | |||
There was a problem hiding this comment.
looks this changes the semantics of TryPinObject, is it intentional?
There was a problem hiding this comment.
Yeah, actually the previous semantics were flipped :)
stephanie-wang
force-pushed
the
spill-metrics
branch
from
3e21861
to
bfdec73
Compare
| ) | ||
| network_speed_stats = self._compute_speed_from_hist(self._network_stats_hist) | ||
|
|
||
| disk_stats = self._get_disk_io() |
There was a problem hiding this comment.
nit: rename to _get_disk_io_stats()?
| @@ -165,6 +189,9 @@ def __init__(self, dashboard_agent): | |||
| self._hostname = socket.gethostname() | |||
| self._workers = set() | |||
| self._network_stats_hist = [(0, (0.0, 0.0))] # time, (sent, recv) | |||
| self._disk_stats_hist = [ | |||
There was a problem hiding this comment.
nit: _disk_io_stats_hist since there are other disk related stats there as well like disk_usage
| auto it = object_pull_requests_.find(obj_id); | ||
| RAY_CHECK(it != object_pull_requests_.end()); |
There was a problem hiding this comment.
You can use map_find_or_die() which gives you better check failure message.
| @@ -640,10 +649,20 @@ bool PullManager::TryPinObject(const ObjectID &object_id) { | |||
| if (ref != nullptr) { | |||
There was a problem hiding this comment.
It's also useful to record num_failed_pins_
| if (it->second.activate_time_ms > 0) { | ||
| ray::stats::STATS_pull_manager_object_request_time_ms.Record( | ||
| absl::GetCurrentTimeNanos() / 1e3 - it->second.activate_time_ms, | ||
| "MemoryAvailableToPin"); |
stephanie-wang
added
the
@author-action-required
stephanie-wang
added
tests-ok
Why are these changes needed?
Adds some metrics useful for object-intensive workloads:
Checks
scripts/format.shto lint the changes in this PR.