Add a document for profiling #1299

Yicheng-Lu-llll · 2023-08-07T22:28:57Z

Why are these changes needed?

See #982. Users have difficulty getting py-spy to work using Kuberay. This PR introduces documentation to address the issue.

Related issue number

Closes #982

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

kevin85421 · 2023-08-09T00:16:27Z

ray-operator/config/samples/ray-cluster.profiling.yaml

+
+    @ray.remote
+    def long_running_task():
+      while True:


Could you add this to the doc?

ray-operator/config/samples/ray-cluster.profiling.yaml

kevin85421 · 2023-08-09T00:18:07Z

cc @rkooo567 @scottsun94 would you mind reviewing this PR? Thanks!

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

scottsun94 · 2023-08-09T23:43:25Z

Can I preview the doc somewhere? Or I have to build it locally to see it?
Btw, is there a way to see the master doc? Does this https://ray-project.github.io/kuberay/ only show the latest version's doc?

scottsun94 · 2023-08-09T23:48:34Z

We should add a profiling page (which should cover all kinds of topics related to profiling) in Ray doc too and cross-reference this kubeRay doc or even just duplicate this. cc: @rkooo567

Yicheng-Lu-llll · 2023-08-10T03:12:41Z

Can I preview the doc somewhere? Or I have to build it locally to see it?

Hi @scottsun94, you can preview the document here: https://github.com/ray-project/kuberay/blob/4abf746a7f80f1075b1ffba30751eda9c2a03089/docs/guidance/profiling.md.

scottsun94

Sorry for the late reply. Left my comments

scottsun94 · 2023-08-14T07:44:38Z

docs/guidance/profiling.md

@@ -0,0 +1,65 @@
+# Profiling with KubeRay
+
+[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.


Suggested change

[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.

## Stack trace and CPU profiling

[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.

Can we add a h2 title here so that:

make it clear that profiling =! pyspy

we can add more profiling-related info down the road than just py-spy, such as memory profiling, gpu profiling, etc.

scottsun94 · 2023-08-14T07:45:44Z

docs/guidance/profiling.md

+
+This document describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray dashboard.
+
+### **Theory**


Suggested change

### **Theory**

### **Prerequisite**

scottsun94 · 2023-08-14T07:47:57Z

docs/guidance/profiling.md

+This document describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray dashboard.
+
+### **Theory**
+py-spy requires the `SYS_PTRACE` capability to read process memory. However, Kubernetes omits this capability by default. To enable profiling, add the following to the `template.spec.containers` for both the head and workers.


Suggested change

py-spy requires the `SYS_PTRACE` capability to read process memory. However, Kubernetes omits this capability by default. To enable profiling, add the following to the `template.spec.containers` for both the head and workers.

py-spy requires the `SYS_PTRACE` capability to read process memory. However, Kubernetes omits this capability by default. To enable profiling, add the following to the `template.spec.containers` for both the head and worker pods.

scottsun94 · 2023-08-14T07:48:23Z

docs/guidance/profiling.md

+    **Notes:**
+    - If you're running your own examples and encounter the error `Failed to write flamegraph: I/O error: No stack counts found` when viewing CPU Flame Graph, it might be due to the process being idle. Notably, using the `sleep` function can lead to this state. In such situations, py-spy filters out the idle stack traces. Refer to this [issue](https://github.com/benfred/py-spy/issues/321#issuecomment-731848950) for more information.
+
+6. **Profile using the Ray dashboard**:


Suggested change

6. **Profile using the Ray dashboard**:

6. **Profile using Ray Dashboard**:

scottsun94 · 2023-08-14T07:49:19Z

docs/guidance/profiling.md

+
+[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.
+
+This document describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray dashboard.


Suggested change

This document describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray dashboard.

This section describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray Dashboard.

scottsun94 · 2023-08-14T07:50:05Z

docs/guidance/profiling.md

+
+    # (Head Pod) Run a sample job in the Pod
+    # `long_running_task` includes a `while True` loop to ensure the task remains actively running indefinitely. 
+    # This allows you ample time to view the Stack Trace and CPU Flame Graph via the Ray dashboard.


Suggested change

# This allows you ample time to view the Stack Trace and CPU Flame Graph via the Ray dashboard.

# This allows you ample time to view the Stack Trace and CPU Flame Graph via Ray Dashboard.

scottsun94 · 2023-08-14T07:50:51Z

ray-operator/config/samples/ray-cluster.profiling.yaml

+          ports:
+          - containerPort: 6379
+            name: gcs-server
+          - containerPort: 8265 # Ray dashboard


Suggested change

- containerPort: 8265 # Ray dashboard

- containerPort: 8265 # Ray Dashboard

scottsun94 · 2023-08-14T07:51:02Z

ray-operator/config/samples/ray-cluster.profiling.yaml

+              cpu: 500m
+              memory: 2Gi
+          # `py-spy` is a sampling profiler that requires `SYS_PTRACE` to read process memory effectively.
+          # Once enabled, you can profile Ray worker processes through the Ray dashboard.


Suggested change

# Once enabled, you can profile Ray worker processes through the Ray dashboard.

# Once enabled, you can profile Ray worker processes through Ray Dashboard.

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

Yicheng-Lu-llll · 2023-08-15T06:27:45Z

Hi @scottsun94, thank you for the review! I've updated the PR following your suggestions. I have also reran the instructions from the doc, everything works as expected.

scottsun94 · 2023-08-15T07:11:23Z

One more question: do we plan to expose this page in KubeRay doc https://ray-project.github.io/kuberay/ ?

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

Yicheng-Lu-llll · 2023-08-15T18:06:15Z

Hi @scottsun94, I have added profiling.md to https://ray-project.github.io/kuberay/.

You can preview it by following the instructions here: https://github.com/ray-project/kuberay/blob/master/docs/development/development.md#deploying-documentation-locally.

Here is a screenshot from my local setup:

scottsun94

LGTM. Thanks!

kevin85421

Thanks, @Yicheng-Lu-llll and @scottsun94!

Add a document for profiling

profiling

88f9ea9

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

Yicheng-Lu-llll force-pushed the pyspy branch from 0153d7e to 88f9ea9 Compare August 8, 2023 01:26

Yicheng-Lu-llll marked this pull request as ready for review August 8, 2023 03:25

kevin85421 reviewed Aug 9, 2023

View reviewed changes

kevin85421 requested review from rkooo567 and scottsun94 August 9, 2023 00:17

Yicheng-Lu-llll added 2 commits August 9, 2023 22:57

improve doc

a0fa532

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

nit

4abf746

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

scottsun94 reviewed Aug 14, 2023

View reviewed changes

Yicheng-Lu-llll and others added 4 commits August 15, 2023 05:45

Address suggestions from scottsun94

3462a5d

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

nit

ddd4be7

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

nit

b101ec9

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

Merge branch 'ray-project:master' into pyspy

89e6777

add profiling.md to https://ray-project.github.io/kuberay/

d93af34

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

scottsun94 approved these changes Aug 15, 2023

View reviewed changes

kevin85421 approved these changes Aug 15, 2023

View reviewed changes

kevin85421 merged commit 87407ac into ray-project:master Aug 15, 2023
20 checks passed

blublinsky pushed a commit to blublinsky/kuberay that referenced this pull request Aug 22, 2023

Add a document for profiling (ray-project#1299)

b9104d8

Add a document for profiling

lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023

Add a document for profiling (ray-project#1299)

7a5a3ab

Add a document for profiling

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a document for profiling #1299

Add a document for profiling #1299

Yicheng-Lu-llll commented Aug 7, 2023 •

edited

kevin85421 Aug 9, 2023

kevin85421 commented Aug 9, 2023

scottsun94 commented Aug 9, 2023

scottsun94 commented Aug 9, 2023

Yicheng-Lu-llll commented Aug 10, 2023

scottsun94 left a comment

scottsun94 Aug 14, 2023

scottsun94 Aug 14, 2023

scottsun94 Aug 14, 2023

scottsun94 Aug 14, 2023

scottsun94 Aug 14, 2023 •

edited

scottsun94 Aug 14, 2023

scottsun94 Aug 14, 2023

scottsun94 Aug 14, 2023 •

edited

Yicheng-Lu-llll commented Aug 15, 2023

scottsun94 commented Aug 15, 2023

Yicheng-Lu-llll commented Aug 15, 2023

scottsun94 left a comment

kevin85421 left a comment

		@@ -0,0 +1,65 @@
		# Profiling with KubeRay

		[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.

	[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.
	## Stack trace and CPU profiling
	[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.


		This document describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray dashboard.

		### Theory

	py-spy requires the `SYS_PTRACE` capability to read process memory. However, Kubernetes omits this capability by default. To enable profiling, add the following to the `template.spec.containers` for both the head and workers.
	py-spy requires the `SYS_PTRACE` capability to read process memory. However, Kubernetes omits this capability by default. To enable profiling, add the following to the `template.spec.containers` for both the head and worker pods.

	6. Profile using the Ray dashboard:
	6. Profile using Ray Dashboard:

	This document describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray dashboard.
	This section describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph via Ray Dashboard.

	# This allows you ample time to view the Stack Trace and CPU Flame Graph via the Ray dashboard.
	# This allows you ample time to view the Stack Trace and CPU Flame Graph via Ray Dashboard.

	- containerPort: 8265 # Ray dashboard
	- containerPort: 8265 # Ray Dashboard

	# Once enabled, you can profile Ray worker processes through the Ray dashboard.
	# Once enabled, you can profile Ray worker processes through Ray Dashboard.

Add a document for profiling #1299

Add a document for profiling #1299

Conversation

Yicheng-Lu-llll commented Aug 7, 2023 • edited

Why are these changes needed?

Related issue number

Checks

kevin85421 Aug 9, 2023

Choose a reason for hiding this comment

kevin85421 commented Aug 9, 2023

scottsun94 commented Aug 9, 2023

scottsun94 commented Aug 9, 2023

Yicheng-Lu-llll commented Aug 10, 2023

scottsun94 left a comment

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023 • edited

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023

Choose a reason for hiding this comment

scottsun94 Aug 14, 2023 • edited

Choose a reason for hiding this comment

Yicheng-Lu-llll commented Aug 15, 2023

scottsun94 commented Aug 15, 2023

Yicheng-Lu-llll commented Aug 15, 2023

scottsun94 left a comment

Choose a reason for hiding this comment

kevin85421 left a comment

Choose a reason for hiding this comment

Yicheng-Lu-llll commented Aug 7, 2023 •

edited

scottsun94 Aug 14, 2023 •

edited

scottsun94 Aug 14, 2023 •

edited