Adaptative implementation with a Remote scheduler

[guillaumeeb]

In order to identify the interaction between adaptative mode and the ClusterManager and Scheduler components, it would be good to use current dask-jobqueue implementation.

• Move adaptive logic here, remove distributed.deploy dependency to it.
• Simplify the logic if possible, see if adaptive can only call scale, and not retire_workers itself
• Identify the remote method that adaptive will need to be available from Scheduler.

[guillaumeeb]

@mrocklin @jacobtomlinson since dask/dask-kubernetes#162 was merge, I guess Adaptive mode doesn't need access to Scheduler object anymore and discuss with it only by RPC calls?

[mrocklin]

That's correct

…

On Fri, Oct 11, 2019 at 2:47 PM Guillaume Eynard-Bontemps < ***@***.***> wrote: @mrocklin <https://github.com/mrocklin> @jacobtomlinson <https://github.com/jacobtomlinson> since dask/dask-kubernetes#162 <dask/dask-kubernetes#162> was merge, I guess Adaptive mode doesn't need access to Scheduler object anymore and discuss with it only by RPC calls? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#188?email_source=notifications&email_token=AACKZTHQTPXIGEZJHIND6ODQODJ47A5CNFSM4GAOUHMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBA7OA#issuecomment-541200312>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTFKA4BORRJOPGYXAF3QODJ47ANCNFSM4GAOUHMA> .

[mrocklin]

This was fixed in dask/distributed#2865

…

On Fri, Oct 11, 2019 at 2:48 PM Matthew Rocklin ***@***.***> wrote: That's correct On Fri, Oct 11, 2019 at 2:47 PM Guillaume Eynard-Bontemps < ***@***.***> wrote: > @mrocklin <https://github.com/mrocklin> @jacobtomlinson > <https://github.com/jacobtomlinson> since dask/dask-kubernetes#162 > <dask/dask-kubernetes#162> was merge, I guess > Adaptive mode doesn't need access to Scheduler object anymore and discuss > with it only by RPC calls? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#188?email_source=notifications&email_token=AACKZTHQTPXIGEZJHIND6ODQODJ47A5CNFSM4GAOUHMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBA7OA#issuecomment-541200312>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACKZTFKA4BORRJOPGYXAF3QODJ47ANCNFSM4GAOUHMA> > . >

[guillaumeeb]

Closing this one, fixed upstream.