-
-
Notifications
You must be signed in to change notification settings - Fork 142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use number of cores as a parameter to scale() #130
Comments
This makes sense to me. I could imagine wanting scale(cores=100) or
scale(memory='1TB') in practice
…On Thu, Aug 16, 2018 at 2:00 PM, Guillaume EB ***@***.***> wrote:
We've recently switched from number of jobs to number of worker processes
as an argument to scale method. This is good for working with adaptive
cluster as defined in distributed, but somehow not really easy to deal with
while using dask-jobqueue, at least for me.
Ideally, we would use unitary job that spawn an entire node of our
cluster. But in real life, I often find myself changing cores and process
parameters between use cases, or to better fit with the current use of my
cluster, when there is other users' job that are not using full nodes for
example.
And at the end, when using scale method, what I want is to scale to a
given number of CPU cores, no matter the number of jobs or workers or
process or threads. So I'm left doing the (simple) maths of translating
between number of cores and number of processes given my JobqueueCluster
initialization.
It would be really easier if we could have the option to specify a cores
kwarg or something like this to scale method.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#130>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AASszNRfOeNgZA9hZCy6Wck8Be2q44P9ks5uRc91gaJpZM4WAfaB>
.
|
I like the ideas here. These kwargs would probably fit in the base cluster, yes? @guillaumeeb, I'm wondering if you want to play with my branch a bit in #97. I'm having trouble wrapping it up but I think the general improvements there would make using scale and adaptive more friendly. |
Yes I believe so too, @mrocklin do you agree? |
Yes, but only if we communicate cores-per-worker and memory-per-worker from
the downstream projects back up to the main class. None of the projects do
this today.
…On Fri, Aug 17, 2018 at 7:45 AM, Guillaume EB ***@***.***> wrote:
I like the ideas here. These kwargs would probably fit in the base
cluster, yes?
Yes I believe so too, @mrocklin <https://github.com/mrocklin> do you
agree?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#130 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AASszLW1OCRdYn71Eg1W3z3DXNHLgCjuks5uRskBgaJpZM4WAfaB>
.
|
This is a case where some coordination between the various dask-foo
projects for deployment would be valuable
On Fri, Aug 17, 2018 at 7:50 AM, Matthew Rocklin <mrocklin@anaconda.com>
wrote:
… Yes, but only if we communicate cores-per-worker and memory-per-worker
from the downstream projects back up to the main class. None of the
projects do this today.
On Fri, Aug 17, 2018 at 7:45 AM, Guillaume EB ***@***.***>
wrote:
> I like the ideas here. These kwargs would probably fit in the base
> cluster, yes?
>
> Yes I believe so too, @mrocklin <https://github.com/mrocklin> do you
> agree?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#130 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszLW1OCRdYn71Eg1W3z3DXNHLgCjuks5uRskBgaJpZM4WAfaB>
> .
>
|
After a dask-jobqueue presentation today with teams doing heavy flight dynamics computation, I believe that this proposal could be an important improvement for easier adoption. They were really enthusiast with the tool, but explaining the scaling parameter was a pain point, even given it's not that hard. Dealing with scheduler jobs not corresponding to number of workers is unclear. I see two ways of doing this:
I'm keen to work on either, solution. It would be nice if I could have some advice, and maybe rough insights on how to do it best if we chose the second option. |
I agree that doing the second option would be nicer. I think that this requires us to develop a convention used by the dask deployment subprojects to record cpu and memory per worker in a standard way. I'll raise an issue upstream. |
Raised upstream at dask/distributed#2208 |
After discussions in dask/distributed#2209 and dask/distributed#2235, and after some thoughts, but without actually trying anything, I feel that it may be simpler to just copy and paste the code from dask/distributed#2209 to dask-jobqueue. But should I:
I don't see other solutions, but I don't really like any of those, an external opinion would be welcomed here. Is this really wrong to duplicate distributed.deploy.Cluster here while waiting for dask/distributed#2235 to be solved? |
… On Tue, Oct 2, 2018 at 5:29 PM Guillaume Eynard-Bontemps < ***@***.***> wrote:
After discussions in dask/distributed#2209
<dask/distributed#2209> and dask/distributed#2235
<dask/distributed#2235>, and after some
thoughts, but without actually trying anything, I feel that it may be
simpler to just copy and paste the code from dask/distributed#2209
<dask/distributed#2209> to dask-jobqueue. But
should I:
- Copy paste adapt, scale, possibly the widget inside JobQueueCluster
class?
- Copy the Cluster class and put it in this repo and between
JobQueueCluster and distributed.deploy.Cluster in the class hierarchy.
- Try using super() calls, knowing that it won't work for the widget
or #152 <#152>.
I don't see other solutions, but I don't really like any of those, an
external opinion would be welcomed here. Is this really wrong to duplicate
distributed.deploy.Cluster here while waiting for dask/distributed#2235
<dask/distributed#2235> to be solved?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#130 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AASszMfXiZ5Z8qtEabCRhSTtdoHtnmkoks5ug9qtgaJpZM4WAfaB>
.
|
I also don't have particularly strong feelings about this. If you remember, I was also heading down the path of overriding the |
We've recently switched from number of jobs to number of worker processes as an argument to
scale
method. This is good for working with adaptive cluster as defined in distributed, but somehow not really easy to deal with while using dask-jobqueue, at least for me.Ideally, we would use unitary job that spawn an entire node of our cluster. But in real life, I often find myself changing cores and process parameters between use cases, or to better fit with the current use of my cluster, when there is other users' job that are not using full nodes for example.
And at the end, when using scale method, what I want is to scale to a given number of CPU cores, no matter the number of jobs or workers or process or threads. So I'm left doing the (simple) maths of translating between number of cores and number of processes given my JobqueueCluster initialization.
It would be really easier if we could have the option to specify a cores kwarg or something like this to scale method.
The text was updated successfully, but these errors were encountered: