Specify a cluster scheduler listener port?

[hawk-sf]

I'm trying to use dask-jobqueue on an SGE cluster, and by default our worker nodes cannot communicate back to the login node. The admins have set aside a small range of ports us to use for this, but I can't seem to specify a port/port range for the scheduler to listen on. I see with dask.distributed's LocalCluster, you can specify the argument scheduler_port, but SGECluster does not seem to take the keyword (though it doesn't complain):

cluster = SGECluster(queue='short.q',
                     cores=10,
                     memory='20GB',
                     walltime='00:20:00',
                     scheduler_port=8804)

cluster.scheduler shows the scheduler still being assigned a random port (always around 40,000):

<Scheduler: "tcp://169.230.126.85:39008" processes: 0 cores: 0>

Is there a way to configure this?

Thanks.

[lesteve]

I can reproduce this and it seems like a regression in 0.7.0. It works as it should with 0.6.3.

from dask_jobqueue import SGECluster

cluster = SGECluster(queue='short.q',
                     cores=10,
                     memory='20GB',
                     walltime='00:20:00',
                     scheduler_port=8804)

print(cluster.scheduler_address)

Output with 0.6.3:

tcp://192.168.0.11:8804

Output with 0.7.0 (random port):

tcp://192.168.0.11:35257

My previous mental model was that FooCluster **kwargs were forwarded until they reach the underlying LocalCluster cluster, but there is no underlying LocalCluster anymore with the SpecCluster refactor in #307.

This would needs to be looked at maybe @mrocklin you have a super rough guestimate how hard this would be to fix ? I am hoping this is a silly mistake that we did not catch and should be reasonably straightforward to figure out where the **kwargs fail to be forwarded to the right place.

At the moment I don't have the bandwidth to investigate, but a PR would certainly be welcome!

[hawk-sf]

Downgrading to 0.6.3 did the trick, for me. Thanks. I'd certainly be willing to take a look at getting **kwargs forwarded correctly, and putting in a PR; is this something you'd want a non-core contributer working on? Are these the only contribution guidelines: https://jobqueue.dask.org/en/latest/develop.html?

[mrocklin]

The relevant lines to affect are here:

dask-jobqueue/dask_jobqueue/core.py

Lines 428 to 437 in c1e0a21

	scheduler = {
	"cls": Scheduler, # Use local scheduler for now
	"options": {
	"protocol": protocol,
	"interface": interface,
	"host": host,
	"dashboard_address": dashboard_address,
	"security": security,
	},
	}

You want to make sure that the options dict here includes "port": .... These values get passed directly to the dask.distributed.Scheduler constructor (see that class's docstring for more information). For a more generic solution we might consider accepting a scheduler_options= keyword that accepted a dictionary of options?

[lesteve]

Thanks. I'd certainly be willing to take a look at getting **kwargs forwarded correctly, and putting in a PR; is this something you'd want a non-core contributer working on?

Sure, help would be more than welcome on this! This is a regression in 0.7 and it would be nice to get it fixed.

Are these the only contribution guidelines: jobqueue.dask.org/en/latest/develop.html?

Yes (there is a link to dask contributing guidelines with more material), there was some talk about adding CONTRIBUTING.md in dask-community dask/community#17.

In any case I am not sure what you mean by "only", but if you spot possible improvements in the contributing documentation, feel free to open a separate PR about this as well!

[guillaumeeb]

see also dask/dask-kubernetes#196 for something similar.

[guillaumeeb]

@hawk-sf would you still be willing to do something about this?

[hawk-sf]

Yes- thanks for the prompt. This had slipped down the priority list, but I’ll have time to start at the end of this week.

[lesteve]

FYI I have a PR in #384.