Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for ClusterManager and WorkerConfig #31234

Merged
merged 2 commits into from Mar 7, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions stdlib/Distributed/docs/src/index.md
Expand Up @@ -60,6 +60,8 @@ same host, and `SSHManager`, for launching on remote hosts via `ssh`. TCP/IP soc
and transport messages between processes. It is possible for Cluster Managers to provide a different transport.

```@docs
Distributed.ClusterManager
Distributed.WorkerConfig
Distributed.launch
Distributed.manage
Distributed.kill(::ClusterManager, ::Int, ::WorkerConfig)
Expand Down
39 changes: 39 additions & 0 deletions stdlib/Distributed/src/cluster.jl
@@ -1,7 +1,46 @@
# This file is a part of Julia. License is MIT: https://julialang.org/license

"""
ClusterManager

Supertype for cluster managers, which control workers processes as a cluster.
Cluster managers implement how workers can be added, removed and communicated with.
`SSHManager` and `LocalManager` are subtypes of this.
vchuravy marked this conversation as resolved.
Show resolved Hide resolved
"""
abstract type ClusterManager end

"""
WorkerConfig

Type used by [`ClusterManager`](@ref)s to control workers added to their clusters. Some fields
are used by all cluster managers to access a host:
* `io` -- the connection used to access the worker (a subtype of `IO` or `Nothing`)
* `host` -- the host address (either an `AbstractString` or `Nothing`)
* `port` -- the port on the host used to connect to the worker (either an `Int` or `Nothing`)

Some are used by the cluster manager to add workers to an already-initialized host:
* `count` -- the number of workers to be launched on the host
* `exename` -- the path to the Julia executable on the host, defaults to `"\$(Sys.BINDIR)/julia"` or
`"\$(Sys.BINDIR)/julia-debug"`
* `exeflags` -- flags to use when lauching Julia remotely

The `userdata` field is used to store information for each worker by external managers.

Some fields are used by `SSHManager` and similar managers:
* `tunnel` -- `true` (use tunneling), `false` (do not use tunneling), or [`nothing`](@ref) (use default for the manager)
* `bind_addr` -- the address on the remote host to bind to
* `sshflags` -- flags to use in establishing the SSH connection
* `max_parallel` -- the maximum number of workers to connect to in parallel on the host

Some fields are used by both `LocalManager`s and `SSHManager`s:
* `connect_at` -- determines whether this is a worker-to-worker or driver-to-worker setup call
* `process` -- the process which will be connected (usually the manager will assign this during [`addprocs`](@ref))
* `ospid` -- the process ID according to the host OS, used to interrupt worker processes
* `environ` -- private dictionary used to store temporary information by Local/SSH managers
* `ident` -- worker as identified by the [`ClusterManager`](@ref)
* `connect_idents` -- list of worker ids the worker must connect to if using a custom topology
* `enable_threaded_blas` -- `true`, `false`, or `nothing`, whether to use threaded BLAS or not on the workers
"""
mutable struct WorkerConfig
# Common fields relevant to all cluster managers
io::Union{IO, Nothing}
Expand Down