[Core] Merge Driver/Job's runtime environment when it conflicts #39208

rkooo567 · 2023-09-01T16:11:12Z

Why are these changes needed?

When the job and driver both specifies the runtime env, driver's runtime env is ignored. This is a confusing behavior and we'd like to merge instead.

This PR merges the runtime envs, and if there's a conflict raises an exception. User can override the job's runtime env by driver's runtime env by specifying the env var.

Related issue number

Closes #37930

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

doc/source/ray-core/handling-dependencies.rst

jjyao · 2023-09-01T17:11:47Z

doc/source/ray-core/handling-dependencies.rst

+  # Driver's specified `runtime_env`
+  {"env_vars": {"C": "c"}}
+
+  # Driver's actual `runtime_env` (merged with Job's)


I assume by Driver's runtime env, you mean runtime_env specified in ray.init(), then the driver's actual runtime env can only be job's runtime env since ray.init runtime env only have effects on tasks and actors.

I'd avoid Driver's runtime_env, by saying runtime_env specified via ray.init()

Also there is another confusing thing: we normally say ray.init() runtime_env as job level runtime env but now there is another job level runtime env specified via job submission.

I guess I will call it job-level runtime env specified ray job submit vs ray.init then

I rewrote this part. Can you check?

angelinalg

Some nits for style/consistency and clarification.

doc/source/cluster/running-applications/job-submission/quickstart.rst

doc/source/ray-core/handling-dependencies.rst

python/ray/runtime_env/runtime_env.py

architkulkarni

Looks good! Some minor concerns:

It looks like we've duplicated the merging code for job->driver and the existing merging code for driver->actor/task. Ideally both of these cases would just call into a single helper function (maybe with an arg raise_exception: bool)
Similarly the merge behavior is described in the docs in two different places, ideally there would be a single source of truth that we just link to from both places.

jjyao · 2023-09-05T19:43:12Z

doc/source/ray-core/handling-dependencies.rst

+Runtime Environment Specified by Both Job and Driver
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+You can specify Ray driver's runtime environment via `ray.init(runtime_env=...)` or `ray job submit --runtime-env`.


Ray driver's runtime env can only be specified via job submission. ray.init runtime env has no effect on the driver.

Oh great catch. ray.init() runtime env only sets the default for any subsequent task or actor's runtime env

wait, so if we do job submission, the driver script will already have a runtime env created? How does it work now? Is the runtime env created by supervisor actor first and the driver is started with the same config?

It looks like it is already written in a pretty confusing way; https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#specifying-a-runtime-environment-per-job (it considers as job submit & ray.init as "runtime env per job"). I think we should technically distinguish them in this case. Let me try rewriting this part

@jjyao @architkulkarni I updated the doc based on the feedback again; 401bf31. Can you guys check again? if this doesn't satisfy the merge bar, I will create a meeting to discuss this.

Also @architkulkarni

It looks like we've duplicated the merging code for job->driver and the existing merging code for driver->actor/task. Ideally both of these cases would just call into a single helper function (maybe with an arg raise_exception: bool)

I think we shouldn't use the Python helper for everything because we'd like to keep the inheritance logic in cpp if possible (so that it will be language independent). That said, we ideally should merge the runtime env in cpp. It seems like the job submit runtime env inheritance is not even supported from other languages, so I think we can do it when we actually support them in multi languages.

Similarly the merge behavior is described in the docs in two different places, ideally there would be a single source of truth that we just link to from both places.

Addressed in 401bf31

Is the runtime env created by supervisor actor first and the driver is started with the same config?

That's right, the driver is started by the actor as a subprocess so it inherits the cwd and env vars automatically without Ray having to do anything.

architkulkarni

Update looks good to me, just some nits about the text.

doc/source/ray-core/handling-dependencies.rst

architkulkarni · 2023-09-06T16:19:29Z

doc/source/ray-core/handling-dependencies.rst


-    This ensures the runtime environment is installed on the cluster before the entrypoint script is run.
+    If ``runtime_env`` is specified from ``ray.init(runtime_env=...)``, the runtime env is only applied to all children Tasks and Actors, not the entrypoint script (Driver) itself.


From what I've seen, tasks, actors and drivers are not capitalized elsewhere in the doc. It would be great if you could double check and make it consistent

This was recommended from @angelinalg. I will just stick to this it for now.

doc/source/ray-core/handling-dependencies.rst

…project#39208) When the job and driver both specifies the runtime env, driver's runtime env is ignored. This is a confusing behavior and we'd like to merge instead. This PR merges the runtime envs, and if there's a conflict raises an exception. User can override the job's runtime env by driver's runtime env by specifying the env var.

…project#39208) When the job and driver both specifies the runtime env, driver's runtime env is ignored. This is a confusing behavior and we'd like to merge instead. This PR merges the runtime envs, and if there's a conflict raises an exception. User can override the job's runtime env by driver's runtime env by specifying the env var. Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>

…project#39208) When the job and driver both specifies the runtime env, driver's runtime env is ignored. This is a confusing behavior and we'd like to merge instead. This PR merges the runtime envs, and if there's a conflict raises an exception. User can override the job's runtime env by driver's runtime env by specifying the env var. Signed-off-by: Victor <vctr.y.m@example.com>

rkooo567 added 5 commits August 28, 2023 11:58

ip

d2ed287

Merge branch 'master' into runtime-env-inheritance-job-submit

8d86a16

ip

3d9c11a

ip

266aaad

Lint

8e5c989

rkooo567 requested review from architkulkarni, wuisawesome, DmitriGekhtman, maxpumperla, pcmoritz, kevin85421 and a team as code owners September 1, 2023 16:11

rkooo567 assigned jjyao, rynewang, architkulkarni and angelinalg Sep 1, 2023

jjyao reviewed Sep 1, 2023

View reviewed changes

angelinalg approved these changes Sep 1, 2023

View reviewed changes

rkooo567 added 2 commits September 5, 2023 21:49

Merge branch 'master' into runtime-env-inheritance-job-submit

8527477

Working.

d09bac8

architkulkarni approved these changes Sep 5, 2023

View reviewed changes

jjyao reviewed Sep 5, 2023

View reviewed changes

rkooo567 added 2 commits September 6, 2023 23:00

Merge branch 'master' into runtime-env-inheritance-job-submit

7532bef

Updated the doc again.

401bf31

architkulkarni approved these changes Sep 6, 2023

View reviewed changes

rkooo567 added 5 commits September 7, 2023 16:02

Address code review.

950bbe7

Merge branch 'master' into runtime-env-inheritance-job-submit

c9c9c4a

Fix doc error

50d0f55

typo

a6e377a

More clarification

81030fa

Remove unnecessary info from a doc.

81272c3

rkooo567 merged commit b1356d7 into ray-project:master Sep 7, 2023
81 of 88 checks passed

GeneDer mentioned this pull request Sep 7, 2023

[Core] Merge Driver/Job's runtime environment when it conflicts (#39208) #39424

Closed

8 tasks

rickyyx mentioned this pull request Oct 24, 2023

[core] microbenchmark regression #40606

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Merge Driver/Job's runtime environment when it conflicts #39208

[Core] Merge Driver/Job's runtime environment when it conflicts #39208

rkooo567 commented Sep 1, 2023

jjyao Sep 1, 2023

rkooo567 Sep 2, 2023

rkooo567 Sep 5, 2023

angelinalg left a comment

architkulkarni left a comment

jjyao Sep 5, 2023

architkulkarni Sep 5, 2023

rkooo567 Sep 6, 2023

rkooo567 Sep 6, 2023

rkooo567 Sep 6, 2023

architkulkarni Sep 6, 2023

architkulkarni left a comment

architkulkarni Sep 6, 2023

rkooo567 Sep 6, 2023


		This ensures the runtime environment is installed on the cluster before the entrypoint script is run.
		If ``runtime_env`` is specified from ``ray.init(runtime_env=...)``, the runtime env is only applied to all children Tasks and Actors, not the entrypoint script (Driver) itself.

[Core] Merge Driver/Job's runtime environment when it conflicts #39208

[Core] Merge Driver/Job's runtime environment when it conflicts #39208

Conversation

rkooo567 commented Sep 1, 2023

Why are these changes needed?

Related issue number

Checks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angelinalg left a comment

Choose a reason for hiding this comment

architkulkarni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

architkulkarni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment