Add scheduler_options to pass scheduler-specific parameters#384
Conversation
lesteve
changed the title
lesteve
changed the title
| ) | ||
| ) | ||
|
|
||
| if dashboard_address is not None: |
There was a problem hiding this comment.
Not allowing to pass host and dashboard_address is a breaking change. I guess it feels cleaner to be able to pass all the scheduler options through scheduler_options.
Also if we keep host for example, you can pass different things at different levels, e.g. host + interface in scheduler_options which would make the error hard to make user-friendly (and/or the code a bit ugly).
If you think that's too harsh, I can make it a warning until the 0.8 release!
There was a problem hiding this comment.
What feel's weird here is to keep the host and dashboard_address in the constructor kwargs, but thrown an exception when used. Maybe we should add some doc or comment, or simply remove them and catch them through **kwargs argument? Or else we should handle it and log a deprecation warning.
I'm not really familliar with Python best practices in this case.
There was a problem hiding this comment.
I can see two solutions:
- raise an informative error if people are using
hostordasboard_addressto tell them to pass this throughscheduler_options. What I had in mind was to have this user-friendly error in 0.7.1 and removehostanddashboard_addressin 0.8. - warn if people are using
hostordashboard_addresstelling them that this will not be supported in 0.8 and then combine them withscheduler_optionsto do the right thing.
I tend to think 1. is easier because you don't need the combining logic between host dashboard_address and scheduler_options which can be tricky. For example, if you use host and use scheduler_options={'interface': eth0} which one should have the priority and should we raise an error in this case.
Also my feeling is that people ignore warnings until it breaks so that 2. is not that helpful for users in the end.
I'm not really familliar with Python best practices in this case.
It completely depends on the project. For example, in scikit-learn we have a strong backward-compatibility policy: we would have warnings for two major releases before we can break user code. I feel in dask-jobqueue we can afford to be less conservative.
lesteve
force-pushed
the
add-scheduler-options
branch
from
d8994fc to
f08f0d0
Compare
lesteve
force-pushed
the
add-scheduler-options
branch
from
f08f0d0 to
56fe96c
Compare
|
Maybe @willirath you could have a quick look if you are around and not too busy with other things? |
guillaumeeb
left a comment
guillaumeeb
left a comment
There was a problem hiding this comment.
This looks good, just a few interrogations. Do you need this to be tested?
| ) | ||
| ) | ||
|
|
||
| if dashboard_address is not None: |
There was a problem hiding this comment.
What feel's weird here is to keep the host and dashboard_address in the constructor kwargs, but thrown an exception when used. Maybe we should add some doc or comment, or simply remove them and catch them through **kwargs argument? Or else we should handle it and log a deprecation warning.
I'm not really familliar with Python best practices in this case.
| # Use the same network interface as the workers if scheduler ip has not | ||
| # been set through scheduler_options via 'host' or 'interface' | ||
| if "host" not in scheduler_options and "interface" not in scheduler_options: | ||
| scheduler_options["interface"] = interface |
There was a problem hiding this comment.
Should'nt we just put interface in the default_scheduler_options? Won't it be overriden if defined in the scheduler_options?
Or is this because of a problem in distributed if host and interface do not match?
There was a problem hiding this comment.
I should probably test this edge case indeed. My line of reasoning is like this (as I tried to explain in the comment but the wording can probably be improved ...): interface parameter applies primarily to workers. It should apply to the scheduler only if the scheduler IP is not already set by either host or interface in the scheduler_options dict.
|
|
||
| if interface: | ||
| extra = extra + ["--interface", interface] | ||
| kwargs.setdefault("host", get_ip_interface(interface)) |
There was a problem hiding this comment.
After having looked a bit more in details, I am reasonably sure that host in kwargs is not used at all, so this code can be safely removed. In other words this is another manifestation of #386.
This code was introduced in #135. At that time the kwargs were passed to LocalCluster. Nowadays the kwargs are passed to the job_cls. This does not make any sense to pass host=something to the worker arguments where something is computed on the node the scheduler runs (similar comment as #382 (comment)) ...
lesteve
force-pushed
the
add-scheduler-options
branch
from
9f11469 to
82c84f7
Compare
|
I think this one should be ready to merge, if someone has some spare time to look at it, this would be much appreciated! If I don't hear back, I plan to merge this one in a few days. |
|
OK I am going to merge this one, do feel free to comment on this PR (even after the fact) if you have some improvements to suggest! |
lesteve
changed the title
2 participants
fix #355 (regression in 0.7 where
scheduler portwas ignored), fix #382 (different interfaces on scheduler and workers).