polish observability (o11y) docs #39069

scottsun94 · 2023-08-29T18:11:56Z

fix [Dashboard] Document that people need to specify --dashboard-host when starting ray within a docker container #35271
resolve https://ray-distributed.slack.com/archives/C01DLHZHRBJ/p1688624808426999
add profiling page under user guides as a central location for people who consider profiling their workloads.
- fix the incorrect statement that gpu profiling is not supported.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

architkulkarni

doc/source/cluster/configure-manage-dashboard.md change looks good to me, giving codeowner approval

rkooo567

Generally LGTM

doc/source/ray-observability/user-guides/debug-apps/debug-hangs.rst

doc/source/cluster/configure-manage-dashboard.md

doc/source/ray-observability/key-concepts.rst

rkooo567 · 2023-08-30T13:02:06Z

doc/source/ray-observability/user-guides/profiling.md

+    - memray
+- GPU profiling
+    - PyTorch Profiler
+- Ray Task / Actor profiling


Suggested change

- Ray Task / Actor profiling

- Ray Task / Actor timeline tracing

I'm not sure if it's tracing. When talking about tracing, to me, it's usually about profiling the time a request takes to go through different services.

I think profiling is too general. Maybe call it timeline?

Removed this title. Only kept the "Ray Task/Actor timeline"

doc/source/ray-observability/user-guides/profiling.md

rkooo567 · 2023-08-30T13:05:50Z

doc/source/ray-observability/user-guides/profiling.md

+
+## GPU profiling
+GPU and GRAM profiling for your GPU workloads like distributed training. This helps you analyze performance and debug memory issues. 
+- PyTorch profiler is supported out of box when used with Ray Train


Add a link?

what link? Ray Train doesn't have doc for it.

Hmm so we don't have any example for this?

Neither Ray Train nor Ray Data has doc about it yet. We should ask them to add it later.

doc/source/ray-observability/key-concepts.rst

kevin85421

LGTM (changes about KubeRay)

rkooo567 · 2023-09-06T07:54:33Z

doc/source/ray-observability/user-guides/profiling.md

+    - memray
+- GPU profiling
+    - PyTorch Profiler
+- Ray Task / Actor profiling


I think profiling is too general. Maybe call it timeline?

angelinalg

Just some suggestions. Nice job!

doc/source/cluster/configure-manage-dashboard.md

doc/source/ray-observability/key-concepts.rst

doc/source/ray-observability/user-guides/debug-apps/debug-hangs.rst

doc/source/ray-observability/user-guides/debug-apps/optimize-performance.rst

doc/source/ray-observability/user-guides/profiling.md

angelinalg · 2023-09-06T22:11:30Z

doc/source/ray-observability/user-guides/profiling.md

+
+(profiling-memory)=
+## Memory profiling
+Memory profiling for Driver and Worker processes. This helps you analyze memory allocations in applications, trace memory leaks, and debug high/low memory or out of memory issues.


Memory profiling for Driver and Worker processes. is not a complete sentence.

Edited it. WDYT?

angelinalg · 2023-09-06T22:12:40Z

doc/source/ray-observability/user-guides/profiling.md

+
+(profiling-timeline)=
+## Ray Task / Actor timeline
+Profiling the execution time of Ray Tasks and Actors with Ray Timeline. This helps you analyze performance, identify the stragglers, and understand the distribution of workloads.


Profiling the execution time of Ray Tasks and Actors with Ray Timeline. is not a complete sentence.

Edited it. WDYT?

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

…erformance.rst Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: matthewdeng <matt@anyscale.com>

#39510) * Update metrics.md (#38512) 1. there are 3 dashboards in the folder now. Refer to the folder instead of only 1 dashboard 2. include "Copy" since people need to copy this from the head node to the Grafana server Signed-off-by: Huaiwei Sun <scottsun94@gmail.com> * polish observability (o11y) docs (#39069) Signed-off-by: Huaiwei Sun <scottsun94@gmail.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: matthewdeng <matt@anyscale.com> * [Doc] Unbold "Use Cases" in sidebar (#39295) Signed-off-by: pdmurray <peynmurray@gmail.com> * [docs] Cleanup for other AIR concepts (#39400) * [doc][clusters] add doc for setting up Ray and K8s (#39408) --------- Signed-off-by: Huaiwei Sun <scottsun94@gmail.com> Signed-off-by: pdmurray <peynmurray@gmail.com> Co-authored-by: Huaiwei Sun <scottsun94@gmail.com> Co-authored-by: matthewdeng <matt@anyscale.com> Co-authored-by: Peyton Murray <peynmurray@gmail.com> Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: matthewdeng <matt@anyscale.com> Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: matthewdeng <matt@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>

scottsun94 requested review from alanwguo, matthewdeng, rkooo567 and angelinalg August 29, 2023 18:13

scottsun94 marked this pull request as ready for review August 29, 2023 18:13

scottsun94 requested review from architkulkarni, wuisawesome, DmitriGekhtman, maxpumperla, pcmoritz, kevin85421 and a team as code owners August 29, 2023 18:13

scottsun94 changed the title ~~polish o11y docs~~ polish observability (o11y) docs Aug 29, 2023

architkulkarni approved these changes Aug 29, 2023

View reviewed changes

rkooo567 reviewed Aug 30, 2023

View reviewed changes

scottsun94 force-pushed the 0828-doc-fix branch 4 times, most recently from 6637b70 to 276a9e3 Compare September 5, 2023 18:11

kevin85421 approved these changes Sep 5, 2023

View reviewed changes

rkooo567 approved these changes Sep 6, 2023

View reviewed changes

scottsun94 force-pushed the 0828-doc-fix branch 2 times, most recently from 4ea2a0b to 3c28699 Compare September 6, 2023 20:28

angelinalg approved these changes Sep 6, 2023

View reviewed changes

scottsun94 added 6 commits September 6, 2023 19:19

polish o11y docs

a7321d3

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

address comments

fc4d3db

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

fix test

7dc1378

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

fixups

d0c197f

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

address comments

1500ac6

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

fixups

5f79c23

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

scottsun94 added 2 commits September 6, 2023 19:19

fix link

e079fab

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

address comments and add one troubleshooting tip

3a27ec6

Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

scottsun94 force-pushed the 0828-doc-fix branch from 5ff24cc to 3a27ec6 Compare September 7, 2023 02:20

scottsun94 and others added 4 commits September 6, 2023 19:20

Update doc/source/ray-observability/user-guides/debug-apps/optimize-p…

7d28da5

…erformance.rst Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Huaiwei Sun <scottsun94@gmail.com>

Merge branch 'master' into 0828-doc-fix

57de822

Merge branch 'master' into 0828-doc-fix

d404ce6

Merge branch 'master' into 0828-doc-fix

6acb2bd

matthewdeng merged commit daf0dc1 into ray-project:master Sep 8, 2023
12 of 15 checks passed

angelinalg added docs An issue or change related to documentation v2.7.0-pick labels Sep 8, 2023

angelinalg mentioned this pull request Sep 9, 2023

[docs][docs infra] [clusters][cherry-pick] Docs cherry-picks for 2.7.0 #39510

Merged

8 tasks

rickyyx mentioned this pull request Oct 24, 2023

[core] microbenchmark regression #40606

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

polish observability (o11y) docs #39069

polish observability (o11y) docs #39069

scottsun94 commented Aug 29, 2023 •

edited

architkulkarni left a comment

rkooo567 left a comment

rkooo567 Aug 30, 2023

scottsun94 Aug 30, 2023

rkooo567 Sep 6, 2023

scottsun94 Sep 6, 2023

rkooo567 Aug 30, 2023

scottsun94 Aug 30, 2023

rkooo567 Sep 6, 2023

scottsun94 Sep 6, 2023

kevin85421 left a comment

rkooo567 Sep 6, 2023

angelinalg left a comment

angelinalg Sep 6, 2023

scottsun94 Sep 7, 2023

angelinalg Sep 6, 2023

scottsun94 Sep 7, 2023

	- Ray Task / Actor profiling
	- Ray Task / Actor timeline tracing

polish observability (o11y) docs #39069

polish observability (o11y) docs #39069

Conversation

scottsun94 commented Aug 29, 2023 • edited

Why are these changes needed?

Related issue number

Checks

architkulkarni left a comment

Choose a reason for hiding this comment

rkooo567 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin85421 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angelinalg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scottsun94 commented Aug 29, 2023 •

edited