[Core] Nested remote function slow start up time #45926
Labels
bug
Something that is supposed to be working; but isn't
core
Issues that should be addressed in Ray Core
P2
Important issue, but not time-critical
performance
What happened + What you expected to happen
Symptom
Ray has significant overhead when running nested remote function for first time.
In the chart above, blue, green and purple bar stands for the time used by the actual functions, orange and red is the time spent by Ray in between. For the first batch of tasks, Ray takes a lot of time to start each nested remote function. For later tasks, such overhead is more rare.
Analysis
Since there are more tasks than CPUs, the top layer function consumes all CPUs, and Ray have to start new worker process for second and third layer functions.
The logs show that most of the time is spent between the worker starting a new worker process and the new worker process initializing
CoreWorker
and connect itself to raylet. However, Ray spend much less time creating worker processes for the first layer functions.Versions / Dependencies
Ray 2.9.3
Python 3.8.0
OS: Ubuntu 18.04.1 LTS
Reproduction script
Issue Severity
Low: It annoys or frustrates me.
The text was updated successfully, but these errors were encountered: