[data] Ray Data jobs detail table #40756
Conversation
Zandew
force-pushed
the
data-head-api
branch
2 times, most recently
from
2e3356a
to
bcdd57d
Compare
| MAX = ( | ||
| "max", | ||
| "max_over_time(sum({}) by (dataset)[" + f"{MAX_TIME_WINDOW}:{SAMPLE_RATE}])", | ||
| ) |
There was a problem hiding this comment.
Is there a simpler way to get the max of a metric?
There was a problem hiding this comment.
What do the brackets ([]) do?
There was a problem hiding this comment.
I think something like [1h:1s] in this case means that we're querying the last 1 hour at 1 second intervals.
dashboard/modules/data/data_head.py
Outdated
| for metric, queries in DATASET_METRICS.items(): | ||
| for query in queries: | ||
| result = await self._query_prometheus(query.value[1].format(metric)) | ||
| for res in result["data"]["result"]: | ||
| dataset, value = res["metric"]["dataset"], res["value"][1] | ||
| if dataset in datasets: | ||
| datasets[dataset][metric][query.value[0]] = value |
There was a problem hiding this comment.
We could store these values back in the _StatsActor for completed datasets so we don't have to query prometheus for datasets that are done executing?
There was a problem hiding this comment.
yeah, let's do this in a future PR (feel free to leave a TODO comment or open an issue)
|
Can we detect if people use Ray data in a ray job? If so, I'd like to move this table to the top above the "Task/actor overview" and hide this table when people don't use Ray Data |
Yeah we can just hide the table if there aren't any datasets. |
Great! Can you update the screenshot? |
Zandew
requested review from
ericl,
scv119,
c21,
amogkam,
scottjlee,
bveeramani,
raulchen and
stephanie-wang
as code owners
| MAX = ( | ||
| "max", | ||
| "max_over_time(sum({}) by (dataset)[" + f"{MAX_TIME_WINDOW}:{SAMPLE_RATE}])", | ||
| ) |
There was a problem hiding this comment.
What do the brackets ([]) do?
|
LGTM. Thanks! |
Signed-off-by: Andrew Xue <andewzxue@gmail.com>
Signed-off-by: Andrew Xue <andewzxue@gmail.com>
Signed-off-by: Andrew Xue <andewzxue@gmail.com>
Signed-off-by: Andrew Xue <andewzxue@gmail.com>
Signed-off-by: Andrew Xue <andewzxue@gmail.com>
dashboard/modules/data/data_head.py
Outdated
| for metric, queries in DATASET_METRICS.items(): | ||
| for query in queries: | ||
| result = await self._query_prometheus(query.value[1].format(metric)) | ||
| for res in result["data"]["result"]: | ||
| dataset, value = res["metric"]["dataset"], res["value"][1] | ||
| if dataset in datasets: | ||
| datasets[dataset][metric][query.value[0]] = value |
There was a problem hiding this comment.
yeah, let's do this in a future PR (feel free to leave a TODO comment or open an issue)
Signed-off-by: Andrew Xue <andewzxue@gmail.com>
|
Hi @alanwguo - do you wanna do a final review on dashboard side? Thanks. |
Creates a table under the jobs page to display dataset-level metrics. The `_StatsActor` now stores dataset metadata like `state`, `progress` and `start/end_time` for each executed dataset, that is directly queried by the new `data_head` dashboard api. This api also makes requests to the prometheus server to get other metrics that are displayed. Signed-off-by: Andrew Xue <andewzxue@gmail.com>
Why are these changes needed?
Creates a table under the jobs page to display dataset-level metrics.
The
_StatsActornow stores dataset metadata likestate,progressandstart/end_timefor each executed dataset, that is directly queried by the newdata_headdashboard api. This api also makes requests to the prometheus server to get other metrics that are displayed.cluster env:
jobs-data-overviewto test on workspaces.Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.