Add Scheduler method to compute target cores and memory

[mrocklin]

This moves logic from Adaptive onto the main Scheduler and is a small
tentative step towards restructuring deployment.

cc @jcrist

[guillaumeeb]

@mrocklin is this related to #2235? Not sure I understand why you would want to make target_workers an handler. In your proposed design, adaptive intelligence is still running alongside the scheduler, isn't it?

You are planning things still further than this, with non python scheduler and so one?

[guillaumeeb]

Why making this a handler method that can be called remotely?

[mrocklin]

Not sure I understand why you would want to make target_workers an handler. In your proposed design, adaptive intelligence is still running alongside the scheduler, isn't it?

The decision of "how many workers should the cluster have" is best made by the scheduler, which has all of the information necessary to make this decision.

However the actual creation and destruction of the workers may happen on a separate process (ClusterManager) and so it may want to ask the scheduler for the target number of workers.

You are planning things still further than this

There is still plenty of work to do, but I don't anticipate working on this topic personally in the next month.

with non python scheduler and so one?

I'm not sure I understand what you mean here by non-python scheduler

[guillaumeeb]

However the actual creation and destruction of the workers may happen on a separate process (ClusterManager) and so it may want to ask the scheduler for the target number of workers.

From previous exchanges, I had the impression that it was the Scheduler that should ask the ClusterManager to scale to a given number of cores, not the ClusterManager asking how many workers to launch, but it may be the correct solution, I'm not sure. It depends where Adaptive logic must be run.

I'm not sure I understand what you mean here by non-python scheduler

In another issue, IIRC, you speak about the possibility of Scheduler non being a python piece of software in the future.

[mrocklin]

From previous exchanges, I had the impression that it was the Scheduler that should ask the ClusterManager to scale to a given number of cores, not the ClusterManager asking how many workers to launch, but it may be the correct solution, I'm not sure. It depends where Adaptive logic must be run.

Yeah, I don't have a strong opinion here. My guess though is that it will be easier for a ClusterManager to contact the Scheduler rather than the other way. The scheduler is already running a server and we're accustomed to contacting it. If we can avoid setting up another server for the ClusterManager that sounds ideal. The cluster manager will also start the scheduler, and so will probably know the address to contact it.

[guillaumeeb]

Yep you are right, that's one important design décision for #2235:

• ClusterManager will also run adaptive Logic
• Scheduler must provide remote method to informé resources It needs.

[jrbourbeau]

Closing as I think was implemented elsewhere