[core][dashboard] summarize_tasks is slow and blocks event loop

[rynewang]

What happened + What you expected to happen

When you have many tasks in cluster, and you query localhost:8265/api/v0/tasks/summarize?timeout=30 you can get the result in ~2s, but meanwhile the event loop is blocked.

Versions / Dependencies

master

Reproduction script

Create many tasks:

import ray
import time
ray.init()


objs = []

# Evil exec
for i in range(100000):
    code = f"""
@ray.remote
def func_{i}():
    return 1

objs.append(func_{i}.remote())
"""
    exec(code)


print(len(objs))
print(sum(ray.get(objs)))

time.sleep(10000)

then make traffic:

watch "curl localhost:8265/api/v0/tasks/summarize?timeout=30 | wc"

then you can see the lag in dashboard. Now doing flame graph on dashboard:

Here is a svg file: dash_flamegraph

Issue Severity

Medium: It is a significant difficulty but I can work around it.