[xray] Implement timeline and profiling API. #2306

robertnishihara · 2018-06-26T23:12:39Z

This PR adds more detailed profiling information to the timeline for xray and allows you to view the timeline visualization for xray. It also improves the profiling API (for xray).

Example usage. To get a json dump of the profiling information that can be loaded in chrome://tracing, simply do

# Optionally pass in filename= to specify a file to write to.
ray.global_state.chrome_tracing_dump()

The dump will include profiling information for tasks (including the prior breakdown into deserializing arguments, executing the function, getting the results) as well as ray.put, ray.get, ray.wait, task submission, and various other little things. Profiling information will also be included from the driver, and this PR allows for the possibility of creating profiling events from any process that has a GCS client.

To add a custom span to the timeline, simply do

with ray.profile('event name'):
    # Do something.

Then this will appear in the timeline. This can be done within a task or on the driver. An example timeline is attached.

Not implemented in this PR

Timeline integration with web UI.

TODO in this PR maybe:

cc @alanamarzoev

AmplabJenkins · 2018-06-27T00:18:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6298/
Test PASSed.

AmplabJenkins · 2018-06-27T00:39:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6299/
Test PASSed.

AmplabJenkins · 2018-06-27T01:09:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6302/
Test PASSed.

AmplabJenkins · 2018-06-27T09:33:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6317/
Test PASSed.

AmplabJenkins · 2018-06-27T20:53:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6330/
Test PASSed.

AmplabJenkins · 2018-06-29T03:47:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6380/
Test PASSed.

AmplabJenkins · 2018-06-29T04:38:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6381/
Test PASSed.

AmplabJenkins · 2018-06-29T05:13:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6385/
Test PASSed.

AmplabJenkins · 2018-07-03T00:38:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6435/
Test PASSed.

AmplabJenkins · 2018-07-03T01:14:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6436/
Test PASSed.

AmplabJenkins · 2018-07-03T20:38:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6447/
Test FAILed.

…ded.

AmplabJenkins · 2018-07-04T00:34:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6449/
Test FAILed.

AmplabJenkins · 2018-07-04T01:04:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6450/
Test FAILed.

AmplabJenkins · 2018-07-04T01:37:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6451/
Test PASSed.

robertnishihara · 2018-07-04T22:18:43Z

This PR decreases performance for

%time l = ray.get([f.remote() for _ in range(10000)])

from about 1.3s to about 1.6s.

I think a natural solution to this is to do more batching (in particular to flush from workers on a timer instead of after every task). However, that runs into some of the thread-safety issues brought up in #2248, so I may put that on hold for now.

AmplabJenkins · 2018-07-04T22:22:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6464/
Test FAILed.

robertnishihara · 2018-07-04T22:23:59Z

jenkins, retest this please

pcmoritz · 2018-07-04T22:47:23Z

python/ray/experimental/state.py

+    def _profile_table(self, component_id):
+        """Get the profile events for a given component.
+
+        TODO(rkn): This method should support limiting the number of log events


TODOs should not be part of the doc string (user documentation)

pcmoritz · 2018-07-04T22:47:49Z

python/ray/experimental/state.py

+        web browser and load the dumped file. Make sure to enable "Flow events"
+        in the "View Options" menu.
+
+        TODO(rkn): Support including the task specification data in the


TODOs should not be part of the docstring

AmplabJenkins · 2018-07-04T23:24:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6466/
Test PASSed.

AmplabJenkins · 2018-07-05T00:05:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6467/
Test PASSed.

pcmoritz · 2018-07-05T05:47:32Z

src/local_scheduler/local_scheduler_client.h

+/// Store some profile events in the GCS.
+///
+/// \param conn The connection information.
+/// \param event_type The type of the event.


This needs to be updated

pcmoritz · 2018-07-05T05:49:37Z

src/ray/raylet/CMakeLists.txt

        DEPENDS ${FBS_DEPENDS}
        COMMENT "Running flatc compiler on ${NODE_MANAGER_FBS_SRC}"
        VERBATIM)

+#add_custom_target(gen_node_manager_fbs)


remove this

AmplabJenkins · 2018-07-05T07:01:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6481/
Test PASSed.

raulchen · 2018-07-09T07:45:10Z

hi @robertnishihara, it looks like that worker.events is accessed from 2 threads, but it's not protected by any lock. I think this could cause problems in some cases.

For example, if events.append is called between these 2 statements, the data will be lost

ray/python/ray/worker.py

Lines 2733 to 2737 in 4ef9d15

    
               worker.local_scheduler_client.push_profile_events( 
        
                   component_type, ray.ObjectID(worker.worker_id), 
        
                   worker.node_ip_address, worker.events) 
        
           worker.events = []

.

Is my understanding correct? If so, I can file a PR to fix this.

Another quick question, is there any reason why only raylet uses background thread?

raulchen · 2018-07-09T11:20:49Z

Just found that the background thread is only used for drivers. Why do workers not use it?

robertnishihara · 2018-07-09T18:18:51Z

@raulchen you're right, I think it should be protected by a lock.

It's only used in the raylet because I don't think we should add new functionality or change the behavior for the legacy code path.

It's currently only used on the driver, but I think it actually makes sense to use in workers as well. Basically, each worker/driver could just have a background thread that calls _flush_profile_events once a second or so. Then we could remove this line

ray/python/ray/worker.py

Line 1006 in 4ef9d15

flush_profile_data()

from the raylet code path.

raulchen · 2018-07-10T07:45:37Z

python/ray/worker.py

+    # Note(rkn): This is run on a background thread in the driver. It uses the
+    # local scheduler client. This should be ok because it doesn't read from
+    # the local scheduler client and we have the GIL here. However, if either
+    # of those things changes, then we could run into issues.


After doing some research in depth, I believe this is still problematic. The reason is because the get_task function isn't protected by by the GIL See

ray/src/local_scheduler/lib/python/local_scheduler_extension.cc

Line 77 in 450b11f

Py_BEGIN_ALLOW_THREADS

.

Here's the more detailed explanation of how things can go wrong:

the main thread executes the get_task function and switches out when it's in the middle of write_message. (the request is half-sent now)

then the flush-profile-data thread also calls into write_message via push_profile_events, and also write data to the connection.

then the server will receive a corrupted request.

robertnishihara mentioned this pull request Jun 29, 2018

[Issue 2222] Provide better API method for dumping the task timeline … #2309

Closed

robertnishihara added 17 commits July 2, 2018 16:27

Add profile table and store profiling information there.

2aeefed

Code for dumping timeline.

86f4116

Improve color scheme.

c1f21df

Push timeline events on driver only for raylet.

78885b1

Improvements to profiling and timeline visualization

7a76d97

Some linting

1228516

Small fix.

38c4928

Linting

008fd06

Propagate node IP address through profiling events.

748ed71

Fix test.

c631a83

object_id.hex() should return byte string in python 2.

dc73eff

Include gcs.fbs in node_manager.fbs.

c04c31e

Remove flatbuffer definition duplication.

7b9135b

Decode to unicode in Python 3 and bytes in Python 2.

eaa1110

Minor

6ec75d6

Submit profile events in a batch. Revert some CMake changes.

ad50fd9

Fix

1840bce

robertnishihara added 2 commits July 3, 2018 12:09

Workaround test failure.

8e6fde7

Fix linting

52f435b

robertnishihara added 3 commits July 3, 2018 16:23

Linting

53a3b7d

Don't return anything from chrome_tracing_dump when filename is provi…

bd1f79b

…ded.

Remove some redundancy from profile table.

66bba88

Linting

daf40f4

pcmoritz reviewed Jul 4, 2018

View reviewed changes

Move TODOs out of docstring.

1ea0b9c

pcmoritz reviewed Jul 5, 2018

View reviewed changes

Minor

be72d8a

pcmoritz approved these changes Jul 5, 2018

View reviewed changes

pcmoritz merged commit b90e551 into ray-project:master Jul 5, 2018

pcmoritz deleted the timeline branch July 5, 2018 06:23

robertnishihara mentioned this pull request Jul 5, 2018

move import_thread to a separate file #2349

Merged

raulchen reviewed Jul 10, 2018

View reviewed changes

raulchen mentioned this pull request Jul 10, 2018

Make Ray API thread safe #2385

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[xray] Implement timeline and profiling API. #2306

[xray] Implement timeline and profiling API. #2306

robertnishihara commented Jun 26, 2018 •

edited

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 29, 2018

AmplabJenkins commented Jun 29, 2018

AmplabJenkins commented Jun 29, 2018

AmplabJenkins commented Jul 3, 2018

AmplabJenkins commented Jul 3, 2018

AmplabJenkins commented Jul 3, 2018

AmplabJenkins commented Jul 4, 2018

AmplabJenkins commented Jul 4, 2018

AmplabJenkins commented Jul 4, 2018

robertnishihara commented Jul 4, 2018

AmplabJenkins commented Jul 4, 2018

robertnishihara commented Jul 4, 2018

pcmoritz Jul 4, 2018

pcmoritz Jul 4, 2018

AmplabJenkins commented Jul 4, 2018

AmplabJenkins commented Jul 5, 2018

pcmoritz Jul 5, 2018

pcmoritz Jul 5, 2018

AmplabJenkins commented Jul 5, 2018

raulchen commented Jul 9, 2018

raulchen commented Jul 9, 2018

robertnishihara commented Jul 9, 2018

raulchen Jul 10, 2018

[xray] Implement timeline and profiling API. #2306

[xray] Implement timeline and profiling API. #2306

Conversation

robertnishihara commented Jun 26, 2018 • edited

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 27, 2018

AmplabJenkins commented Jun 29, 2018

AmplabJenkins commented Jun 29, 2018

AmplabJenkins commented Jun 29, 2018

AmplabJenkins commented Jul 3, 2018

AmplabJenkins commented Jul 3, 2018

AmplabJenkins commented Jul 3, 2018

AmplabJenkins commented Jul 4, 2018

AmplabJenkins commented Jul 4, 2018

AmplabJenkins commented Jul 4, 2018

robertnishihara commented Jul 4, 2018

AmplabJenkins commented Jul 4, 2018

robertnishihara commented Jul 4, 2018

pcmoritz Jul 4, 2018

Choose a reason for hiding this comment

pcmoritz Jul 4, 2018

Choose a reason for hiding this comment

AmplabJenkins commented Jul 4, 2018

AmplabJenkins commented Jul 5, 2018

pcmoritz Jul 5, 2018

Choose a reason for hiding this comment

pcmoritz Jul 5, 2018

Choose a reason for hiding this comment

AmplabJenkins commented Jul 5, 2018

raulchen commented Jul 9, 2018

raulchen commented Jul 9, 2018

robertnishihara commented Jul 9, 2018

raulchen Jul 10, 2018

Choose a reason for hiding this comment

robertnishihara commented Jun 26, 2018 •

edited