[runtime env]: Integrating Nsight to Ray worker process #39998

jonathan-anyscale · 2023-09-29T23:41:01Z

Why are these changes needed?

Nsight internal docs: https://docs.google.com/document/d/11RlNTbGLf6fat7HYARU8yWhodBD9j5uiZCdAB0geEik
Related issue: #39094

Nsight integration with Ray using runtime_env. Currently nsight can't profile the GPU usage from Ray tasks/actors since the processes that can be traced by nsight must be driver processes and it's subprocesses, whereas Ray tasks/actors are run by worker process. Thus, we added nsight native to runtime_env in order to modify the worker process to run with nsys profile which can produce the report for each worker processes once it exits.

The nsight API in the runtime_env can be specified with flags that user want to add to the nsys profile for example

@ray.remote(runtime_env={"nsight": {
"-t": "cuda,nvtx", 
"--cudabacktrace": "true"
}})
def task():
    ....

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

python/ray/_private/runtime_env/nsight.py

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

rkooo567

Hmm I thought we decided to go with "profiler" now? Is this updated correctly?

python/ray/_private/runtime_env/agent/runtime_env_agent.py

python/ray/remote_function.py

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

python/ray/runtime_env/runtime_env.py

python/ray/tests/test_runtime_env_profiler.py

rkooo567 · 2023-10-11T15:18:54Z

python/ray/tests/test_runtime_env_profiler.py

+        os.environ.get("CI") and sys.platform != "linux",
+        reason="Requires PR wheels built in CI, so only run on linux CI machines.",
+    )
+    def test_nsight_custom_name(


Suggested change

def test_nsight_custom_name(

def test_nsight_custom_option(

(since the purpose of the test is to make sure options work, not the custom name. There's no reason to test if custom name works because that's already should've tested with nsight itself)

python/ray/_private/runtime_env/nsight.py

rkooo567 · 2023-10-11T15:25:48Z

python/ray/_private/runtime_env/nsight.py

+            )
+        # add set output path to logs dir
+        nsight_config["-o"] = f"{self._logs_dir}/" + nsight_config.get(
+            "-o", NSIGHT_DEFAULT_CONFIG["-o"]


Is it possible to include worker id to the output file name?

Talked with Jiajun before, since worker isn't launch when creating plugin, we can't pass it to report filename

Hmm yeah that sounds tricky...

python/ray/_private/runtime_env/nsight.py

rkooo567 · 2023-10-11T15:26:03Z

python/ray/_private/runtime_env/nsight.py

+        )
+
+        self.nsight_cmd = parse_nsight_config(nsight_config)
+        return 0


Q: what's the plan for GC policy now?

Also, cc @architkulkarni how do we currently GC runtime resources? Which directory should we use to utilize the default GC features? (Also, should create return the size of artifact to utilize this feature?)

create in runtime_env is used if we requires installing some packages (like pip, or working_dir)
So I set it to 0 as we don't install the dependency for user, only modifying the RuntimeContext

I see. I guess the file generated by a profiler is a separate question. do you plan to tackle it in the followup prs?

rkooo567 · 2023-10-12T00:31:48Z

python/ray/_private/runtime_env/nsight.py

+        )
+
+        self.nsight_cmd = parse_nsight_config(nsight_config)
+        return 0


I see. I guess the file generated by a profiler is a separate question. do you plan to tackle it in the followup prs?

rkooo567 · 2023-10-12T00:32:23Z

python/ray/_private/runtime_env/nsight.py

+            )
+        # add set output path to logs dir
+        nsight_config["-o"] = f"{self._logs_dir}/" + nsight_config.get(
+            "-o", NSIGHT_DEFAULT_CONFIG["-o"]


Hmm yeah that sounds tricky...

rkooo567 · 2023-10-12T00:34:04Z

python/ray/_private/runtime_env/nsight.py

+    """
+
+    # use empty as nsight report test filename
+    nsight_config_copy = copy.deepcopy(nsight_config)


Raylet just stuck and keep trying to relaunch worker process instead of terminate

Hmm this actually seems like a bug. Maybe we cannot catch the error if a worker process fails to start by a command error. Can you create an issue for this?

rkooo567 · 2023-10-12T00:34:10Z

python/ray/_private/runtime_env/nsight.py

+    """
+
+    # use empty as nsight report test filename
+    nsight_config_copy = copy.deepcopy(nsight_config)


for now, the workaround looks okay.

python/ray/_private/runtime_env/nsight.py

rkooo567 · 2023-10-12T00:37:14Z

python/ray/_private/runtime_env/nsight.py

+    """
+
+    # use empty as nsight report test filename
+    nsight_config_copy = copy.deepcopy(nsight_config)


Theoretically the cpp worker pool should be able to catch this (maybe it could be follow-up for you if you have time in the future)

python/ray/_private/runtime_env/nsight.py

rkooo567 · 2023-10-12T00:37:57Z

python/ray/_private/runtime_env/nsight.py

+    Function to check if the provided nsight options are
+    valid nsys profile options and if nsys profile is installed
+
+    The function returns:


seems not changed?

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

ci/docker/core.build.Dockerfile

python/requirements/nsight_mock/nsys_mock.py

python/requirements/nsight_mock/setup.py

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

rkooo567

We should not include the fake nsight to the official docker image.

rkooo567 · 2023-10-16T15:04:13Z

ci/docker/core.build.Dockerfile

@@ -16,3 +16,5 @@ RUN pip install -U --ignore-installed  \
  -r python/requirements.txt \
  -r python/requirements/test-requirements.txt \
  -r python/requirements/ml/dl-cpu-requirements.txt
+
+RUN pip install python/requirements/nsight_fake


Hmm I feel like we shouldn't include this to actual docker image. Can we only include it to a test images? Or install is separately from the build

I'll try move it on basetest.Dockerfile

rkooo567 · 2023-10-16T15:04:16Z

ci/docker/core.build.wanda.yaml

@@ -6,5 +6,7 @@ srcs:
  - python/requirements_compiled.txt
  - python/requirements/test-requirements.txt
  - python/requirements/ml/dl-cpu-requirements.txt
+  - python/requirements/nsight_fake/nsys_fake.py
+  - python/requirements/nsight_fake/setup.py


same as above

python/ray/_private/runtime_env/agent/runtime_env_agent.py

python/ray/_private/runtime_env/nsight.py

rkooo567 · 2023-10-16T15:10:05Z

Follow up:

enhance the API to follow the API proposal
GC policy
other profilers

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale marked this pull request as draft September 29, 2023 23:41

jonathan-anyscale force-pushed the nsight_ray branch from 47c5c1f to 7865f2c Compare September 29, 2023 23:45

ray nsight prototype

8d96d4a

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from 7865f2c to 8d96d4a Compare September 29, 2023 23:46

jonathan-anyscale changed the title ~~Nsight ray~~ [runtime env]: Integrating Nsight to Ray worker process Sep 29, 2023

fix remote func typo

c445e24

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from 3305c6b to c445e24 Compare October 2, 2023 01:09

jonathan-anyscale added 2 commits October 2, 2023 09:58

add check if nsys installed

aadaee4

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

lint

1b70da2

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale assigned jonathan-anyscale and unassigned jonathan-anyscale Oct 2, 2023

jonathan-anyscale added 3 commits October 3, 2023 20:57

replace nsight output to logs dir

a744024

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

enable list of nsys profile options on runtime_env

046f9ca

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

add default config for nsight

27895b1

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from a68df52 to 27895b1 Compare October 5, 2023 19:16

jonathan-anyscale assigned rkooo567 Oct 5, 2023

clean up

c7fe651

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale marked this pull request as ready for review October 7, 2023 06:52

nsight unit tests

bfc7c1a

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from 3519905 to bfc7c1a Compare October 7, 2023 07:27

KarimDarwish reviewed Oct 9, 2023

View reviewed changes

python/ray/_private/runtime_env/nsight.py Outdated Show resolved Hide resolved

fix test

8e6d622

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

rkooo567 reviewed Oct 10, 2023

View reviewed changes

python/ray/_private/runtime_env/agent/runtime_env_agent.py Show resolved Hide resolved

python/ray/remote_function.py Show resolved Hide resolved

add ray actor test

cc2e783

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from d1af0ec to cc2e783 Compare October 10, 2023 16:47

rkooo567 reviewed Oct 11, 2023

View reviewed changes

rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Oct 11, 2023

rkooo567 reviewed Oct 12, 2023

View reviewed changes

subprocess to asyncio

75f757b

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from c9699e7 to 1426054 Compare October 12, 2023 05:35

jonathan-anyscale force-pushed the nsight_ray branch 2 times, most recently from 446f32e to eb2c199 Compare October 12, 2023 06:42

nsight mock

e83a1a0

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from eb2c199 to e83a1a0 Compare October 12, 2023 06:58

fix ci

6ebe5d3

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from 444183d to 6ebe5d3 Compare October 12, 2023 17:02

jonathan-anyscale added 3 commits October 12, 2023 10:25

fix ci

d9cc66e

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

debug ci

88123bb

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

move nsight mock to requirements

9f2b321

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from ccd822e to 9f2b321 Compare October 12, 2023 20:16

jonathan-anyscale requested review from aslonnie, can-anyscale and krfricke as code owners October 12, 2023 20:16

aslonnie reviewed Oct 12, 2023

View reviewed changes

ci/docker/core.build.Dockerfile Outdated Show resolved Hide resolved

python/requirements/nsight_mock/nsys_mock.py Outdated Show resolved Hide resolved

python/requirements/nsight_mock/setup.py Outdated Show resolved Hide resolved

fix nsight fake

6b161cf

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Oct 13, 2023

jonathan-anyscale and others added 2 commits October 13, 2023 13:02

Merge branch 'master' into nsight_ray

f5a1d63

fix redundant build files

d397443

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

rkooo567 approved these changes Oct 16, 2023

View reviewed changes

rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Oct 16, 2023

jonathan-anyscale force-pushed the nsight_ray branch 3 times, most recently from 794abbd to 4c6908e Compare October 16, 2023 19:57

move nsight fake dep to profiler test

ed7f9ee

Signed-off-by: Jonathan Nitisastro <jonathancn@anyscale.com>

jonathan-anyscale force-pushed the nsight_ray branch from 4c6908e to ed7f9ee Compare October 16, 2023 20:31

rkooo567 merged commit 4113ab4 into ray-project:master Oct 17, 2023
63 of 69 checks passed

jonathan-anyscale removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Oct 17, 2023

jonathan-anyscale mentioned this pull request Oct 17, 2023

[Feature] NSight GPU profiler support #19631

Closed

2 tasks

rickyyx mentioned this pull request Oct 26, 2023

Release test long_running_many_actor_tasks failed #40568

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[runtime env]: Integrating Nsight to Ray worker process #39998

[runtime env]: Integrating Nsight to Ray worker process #39998

jonathan-anyscale commented Sep 29, 2023 •

edited

Loading

rkooo567 left a comment

rkooo567 Oct 11, 2023

rkooo567 Oct 11, 2023

jonathan-anyscale Oct 11, 2023

rkooo567 Oct 12, 2023

rkooo567 Oct 11, 2023

rkooo567 Oct 11, 2023

jonathan-anyscale Oct 11, 2023

rkooo567 Oct 12, 2023

rkooo567 Oct 12, 2023

rkooo567 Oct 12, 2023

rkooo567 Oct 12, 2023

rkooo567 Oct 12, 2023

rkooo567 Oct 12, 2023

rkooo567 Oct 12, 2023

rkooo567 left a comment

rkooo567 Oct 16, 2023

jonathan-anyscale Oct 16, 2023

rkooo567 Oct 16, 2023

rkooo567 commented Oct 16, 2023

[runtime env]: Integrating Nsight to Ray worker process #39998

[runtime env]: Integrating Nsight to Ray worker process #39998

Conversation

jonathan-anyscale commented Sep 29, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

rkooo567 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkooo567 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkooo567 commented Oct 16, 2023

jonathan-anyscale commented Sep 29, 2023 •

edited

Loading