Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate package environment to local workers #43270

Merged
merged 1 commit into from
Feb 25, 2022
Merged

Conversation

fredrikekre
Copy link
Member

Propagate package environment to local workers, fixes #28781, closes #42089.

@fredrikekre fredrikekre added domain:parallelism Parallel or distributed computation domain:packages Package management and loading labels Nov 30, 2021
@fredrikekre fredrikekre force-pushed the fe/distributed-env branch 2 times, most recently from 8d93a56 to 8c2feee Compare December 1, 2021 15:19
Local workers now inherit the package environment of the main process,
i.e. the active project, LOAD_PATH, and DEPOT_PATH. This behavior
can be overridden by passing the new `env` keyword argument, or by
passing `--project` in the `exeflags` keyword argument.

Fixes #28781, and closes #42089.
@fredrikekre fredrikekre merged commit 163c032 into master Feb 25, 2022
@fredrikekre fredrikekre deleted the fe/distributed-env branch February 25, 2022 08:36
staticfloat pushed a commit to JuliaCI/julia-buildkite-testing that referenced this pull request Mar 2, 2022
…ng#43270)

Local workers now inherit the package environment of the main process,
i.e. the active project, LOAD_PATH, and DEPOT_PATH. This behavior
can be overridden by passing the new `env` keyword argument, or by
passing `--project` in the `exeflags` keyword argument.

Fixes JuliaLang#28781, and closes JuliaLang#42089.
LilithHafner pushed a commit to LilithHafner/julia that referenced this pull request Mar 8, 2022
…ng#43270)

Local workers now inherit the package environment of the main process,
i.e. the active project, LOAD_PATH, and DEPOT_PATH. This behavior
can be overridden by passing the new `env` keyword argument, or by
passing `--project` in the `exeflags` keyword argument.

Fixes JuliaLang#28781, and closes JuliaLang#42089.
@fonsp
Copy link
Member

fonsp commented Jan 16, 2023

About the NEWS.md update, it looks like the ENV variables JULIA_LOAD_PATH and JULIA_DEPOT_PATH were already propagated in previous Julia versions:

Tests on Julia 1.8

LOAD_PATH

➜  Downloads JULIA_LOAD_PATH="$(pwd)/hello:$JULIA_LOAD_PATH" julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.3 (2022-11-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> LOAD_PATH
4-element Vector{String}:
 "/Users/fons/Downloads/hello"
 "@"
 "@v#.#"
 "@stdlib"

julia> import Distributed

julia> Distributed.addprocs(1)
1-element Vector{Int64}:
 2

julia> Distributed.remotecall_eval(Main, 2, :LOAD_PATH)
4-element Vector{String}:
 "/Users/fons/Downloads/hello"
 "@"
 "@v#.#"
 "@stdlib"

DEPOT_PATH

➜  Downloads JULIA_DEPOT_PATH="$(pwd)/hello" julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.3 (2022-11-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> st
  Installing known registries into `~/Downloads/hello`
Status `~/Downloads/hello/environments/v1.8/Project.toml` (empty project)

julia> import Distributed

julia> Distributed.addprocs(1)
1-element Vector{Int64}:
 2

julia> Distributed.remotecall_eval(Main, 2, :DEPOT_PATH)
1-element Vector{String}:
 "/Users/fons/Downloads/hello"

I believe this is because the entire ENV is propagated:

julia> Distributed.remotecall_eval(Main, 2, :ENV) == ENV
true

Perhaps the news entry can be reworded? The change is that the active project from the parent process is propagated, independently of how it was set (ENV variable, Pkg.activate, --project, ...). The DEPOT_PATH was already being propagated.

Suggested change: #48301

Keno pushed a commit that referenced this pull request Jun 5, 2024
Local workers now inherit the package environment of the main process,
i.e. the active project, LOAD_PATH, and DEPOT_PATH. This behavior
can be overridden by passing the new `env` keyword argument, or by
passing `--project` in the `exeflags` keyword argument.

Fixes #28781, and closes #42089.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain:packages Package management and loading domain:parallelism Parallel or distributed computation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Workers should inherit Pkg environment
3 participants