Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ExistingProcessManager cluster manager #148

Closed
wants to merge 1 commit into from
Closed

Add the ExistingProcessManager cluster manager #148

wants to merge 1 commit into from

Conversation

DilumAluthge
Copy link

@DilumAluthge DilumAluthge commented Oct 15, 2020

Closes #147

Summary

This pull request adds the ExistingProcessManager cluster manager. Because this is a non-breaking feature, I have bumped the version number from 0.4.0 to 0.4.1.

This is a resubmission of #147.

Example usage

First, run the following command four times. It can be on the same machine or on different machines.

julia --worker=1234567890abcdef &

Make note of the hosts and port numbers that are printed to stdout. For this example, we will suppose that the hosts and port numbers that were printed to stdout were as follows:

Host Port
192.168.1.151 9684
192.168.1.151 9685
192.168.1.151 9686
192.168.1.151 9687

Now, open a Julia session and run the following: (in the workers array, replace the hosts and port numbers with the hosts and port numbers that you received in the previous step)

julia> import ClusterManagers, Distributed

julia> Distributed.cluster_cookie("1234567890abcdef")

julia> workers = [
       ("192.168.1.151", 9684),
       ("192.168.1.151", 9685),
       ("192.168.1.151", 9686),
       ("192.168.1.151", 9687),
       ]

julia> Distributed.addprocs(ClusterManagers.ExistingProcessManager(workers))

Motivation

The idea here is that for whatever reason, you have already manually started the Julia worker processes by doing julia --worker=my_cluster_cookie. Those workers are now running, and you want to add them as processes so that Distributed is aware of them. This is what the ExistingProcessManager cluster manager allows you to do.

Constructors

There are two ways to construct a ExistingProcessManager:

  1. Pass a vector of tuples, where the first element of each tuple is the host, and the second element of each tuple is the port number.
  2. Pass a vector of Distributed.WorkerConfigs.

@DilumAluthge
Copy link
Author

DilumAluthge commented Oct 15, 2020

I need this functionality in production relatively soon, so I've fleshed this out a bit, added tests and docs, and put it into a dedicated package: https://github.com/bcbi/ExistingProcessManagers.jl

I do still think that this would be useful for users of ClusterManagers.jl. But either way, it is already available in ExistingProcessManagers.jl.

@DilumAluthge
Copy link
Author

I'm closing this PR as stale.

The functionality is available in the ExistingProcessManager.jl package (https://github.com/bcbi/ExistingProcessManagers.jl), which is now registered in the General registry.

@DilumAluthge DilumAluthge deleted the dpa/existing-process-manager branch November 8, 2020 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant