-
-
Notifications
You must be signed in to change notification settings - Fork 712
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] Worker groups and environments #495
Comments
Some thoughts:
|
Hrm, people like @ogrisel might have a use for setup(). He often deploys software with something like the following: def install():
import os
os.system('pip install x y z')
client.run(install) This would be a good and use case for setup, to always install certain libraries when the workers come online. |
Since the |
There are two options:
|
Did we make any progress with this issue? |
No, I don't think so.
…On Wed, Aug 14, 2019 at 7:48 PM kindjacket ***@***.***> wrote:
Did we make any progress with this issue?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#495?email_source=notifications&email_token=AAKAOIWUN3RM3A65VVMAVRDQESRUDA5CNFSM4CO5QBA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KQPYI#issuecomment-521471969>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAKAOISPF3UPVT3KUNVMOX3QESRUDANCNFSM4CO5QBAQ>
.
|
This is a concrete proposal for solving #85. For preliminary discussion, see that issue.
What is an environment:
An environment is defined as an object of type
Environment
, which currently has the following design:The reason for making it a class is grouping the methods together, providing a consistent place to store state, and making the user facing signature consistent (even if more optional methods are added to the class). Using the examples from the original issue, example
Environment
classes might be:How environments are registered
Environments are registered by passing instances of the object to
Executor.register_environment
, along with a name. The decision to use instances instead of classes was made to allow environments to be parametrized (if needed), and contain internal state. An example of registering environments is:Changes to the scheduler/worker state and transitions
When an environment is registered
{name: environment_object}
calledenvironments
.isinstance
method is ran to check if the worker is an instance of the environment. If so, the worker stores it in a mapping of{name: environment_object}
called environments. The (optional)setup
method on the environment is then run, which can initialize any additional needed state.{environment_name: {worker_ids}}
to reflect this. Call thisenvironment_workers
.When a worker is added
When a worker is removed
environment_workers
as neededWhen a task is submitted
workers
keyword tosubmit
will be modified to take in environment names as well. This will be sent along in addition torestrictions
andloose_restrictions
to the scheduler'supdate_graph
method as anenvironments
keyword.update_graph
will update a mapping of{key: {environment_names}}
calledenvironment_restrictions
on the state. The reason for not converting this immediately intorestrictions
is that the workers in the environment may change between task submission time and task run time (new workers may register for example).When selecting tasks to run
ensure_occupied
will be modified to also take into accountenvironment_restrictions
when picking tasks to run and where to run them.Summary of new state
New worker state
environments
: a mapping of{environment_name: environment_object}
New scheduler state
environments
: a mapping of{environment_name: environment_object}
environment_workers
: a mapping of{environment_name: {workers}}
environment_restrictions
: a mapping of{key: {environment_names}}
Accessing environments from inside a task
Tasks can call the
distributed.get_environment('name')
function to get the environment object from inside the worker. The implementation of this might look like:This allows tasks to access whatever state the environment may contain internally. For example, a user may do the following to pull data from a stored database connection:
The text was updated successfully, but these errors were encountered: