Add shebang option to support more shells by danpf · Pull Request #189 · dask/dask-jobqueue

danpf · 2018-11-05T01:38:04Z

Previously the shell(shebang) was hardcoded to be '#!/bin/bash'.
This fixes that, because some of the queueing schedulers had their own
shebangs hardcoded as well, so you actually ended up having two shebang
lines. In order to accomodate more shells, we can
add an option to specify shell via shebangs. If you think this
implementation is fine, I can/will add this to the documentation as
well.

Previously the shell(shebang) was hardcoded to be '#!/bin/bash'. This fixes that, because some of the queueing schedulers had their own shebangs hardcoded as well, so you actually ended up having two shebang lines. In order to accomodate more shells, we can add an option to specify shell via shebangs. If you think this implementation is fine, I can/will add this to the documentation as well.

willirath · 2018-11-05T08:45:03Z

            env_extra = dask.config.get('jobqueue.%s.env-extra' % self.scheduler_name)
        if log_directory is None:
            log_directory = dask.config.get('jobqueue.%s.log-directory' % self.scheduler_name)
+        if shebang is None:


I'd reconsider adding a kwarg for this. In my understanding, kwargs are for things you change frequently. This looks like an issue that you have to sort out once for your deployment and then never touch it again. Just adding it to the config should cover all cases.

I'm not sure, and I have no strong feeling on this. Currently it looks like we have kind of a match between kwargs and config file. I'm not sure which way we should go.

I did it this way only because that's how everything else in the yaml's are set. I don't really mind either way. maybe let me know @guillaumeeb ?

Yeah, that is what I thought. If so, leave it as is, as the everything else is done. Maybe we'll need to check whether all the kwargs are justified or not in the future.

willirath · 2018-11-05T08:46:40Z

    local-directory: null       # Location of fast local storage like /scratch or $TMPDIR

    # PBS resource manager options
+    shebang: "#!/bin/bash"


Shouldn't we go for #!/usr/bin/env bash ?

Yep, looks like it, good catch!

willirath · 2018-11-05T08:48:18Z

    with SLURMCluster(walltime='00:02:00', processes=4, cores=8, memory='28GB') as cluster:

        job_script = cluster.job_script()
+        assert job_script.startswith("#!/bin/bash")


This should also go into the tests for the other schedulers.

I believe this should only go in https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/tests/test_jobqueue_core.py as this is not specific to any scheduler.

willirath · 2018-11-05T08:52:08Z

Looks good!

Off Topic:

Looking at this PR, adding all these identical lines to the configs (e.g., https://github.com/dask/dask-jobqueue/pull/189/files#diff-06f4938c65f61f791782337bf4187054R15) feels wrong.

guillaumeeb

Nice catch @danpf, and many thanks for the PR!

It looks good, just address the points highlighted by @willirath. Just not sure about whether it should go into kwargs or not...

guillaumeeb · 2018-11-05T10:17:35Z

    with SLURMCluster(walltime='00:02:00', processes=4, cores=8, memory='28GB') as cluster:

        job_script = cluster.job_script()
+        assert job_script.startswith("#!/bin/bash")


I believe this should only go in https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/tests/test_jobqueue_core.py as this is not specific to any scheduler.

guillaumeeb · 2018-11-05T10:17:50Z

    local-directory: null       # Location of fast local storage like /scratch or $TMPDIR

    # PBS resource manager options
+    shebang: "#!/bin/bash"


Yep, looks like it, good catch!

guillaumeeb · 2018-11-05T10:19:43Z

            env_extra = dask.config.get('jobqueue.%s.env-extra' % self.scheduler_name)
        if log_directory is None:
            log_directory = dask.config.get('jobqueue.%s.log-directory' % self.scheduler_name)
+        if shebang is None:


I'm not sure, and I have no strong feeling on this. Currently it looks like we have kind of a match between kwargs and config file. I'm not sure which way we should go.

guillaumeeb · 2018-11-05T10:27:07Z

Looking at this PR, adding all these identical lines to the configs (e.g., https://github.com/dask/dask-jobqueue/pull/189/files#diff-06f4938c65f61f791782337bf4187054R15) feels wrong.

@willirath you're right, this is not ideal. We did that when implementing multiple config in #68. Feel free to open an issue to discuss other possibilities for that point.

guillaumeeb · 2018-11-05T10:29:36Z

And thanks @willirath for the review here, this is really appreciated.

Previously shell was default "/bin/bash" now it respects the environment. Also shebang tests were only in slurm, now there is one for each scheduler that we support.

guillaumeeb

I think you can simplify tests a bit with pytest parametrize feature.

guillaumeeb · 2018-11-05T20:17:15Z

        assert ' --preload mymodule' in cluster._command_template


+def test_shebang_settings():


You should use parametrized test as the one below for all this!

really cool!!!!

guillaumeeb · 2018-11-05T20:18:42Z

                      env_extra=['export LANG="en_US.utf8"',
                                 'export LANGUAGE="en_US.utf8"',
-                                 'export LC_ALL="en_US.utf8"']
+                                 'export LC_ALL="en_US.utf8"'],


A comma is remaining here.

guillaumeeb · 2018-11-05T22:16:25Z

LGTM, @willirath do you see something else?

I updated the documentation to reflect shebangs being required. because otherwise a batch script could be submitted without an interpreter. I also added a test to cover this.

danpf · 2018-11-05T23:56:17Z

sorry to bother you more, please review the updates to the documentation I added as well.
Also added some functionality to make sure there was always a shebang + test for that.

willirath

I'm not sure the additional requirement mock is needed to solve the problem at hand. In Xarray's tests, they use a context manager to directly change the dask config:
https://github.com/pydata/xarray/blob/4a5d88dba5fd50d48ab00ed2ebaee058287ab0bf/xarray/tests/test_dask.py#L838

So you could just use

    with pytest.raises(ValueError, match="batch script interpreter"):
        with (dask.config.set(shebang=None)):
            Cluster(cores=2, memory='4GB')

willirath · 2018-11-06T12:44:44Z

Apart from the question about the added dependency on mock, this looks very good!

willirath · 2018-11-06T16:23:58Z

+        assert job_script.startswith(default_shebang)
+
+    with pytest.raises(ValueError, match="batch script interpreter"):
+        with (dask.config.set(shebang=None)):


Sorry. My example code was wrong. dask.config.set('jobqueue.%s.shebang' % Cluster.scheduler_name, None) should work.

thanks, my initial idea was way worse.

willirath

Looks good!

willirath · 2018-11-06T17:49:13Z

LGTM, @willirath do you see something else?

@guillaumeeb Good to go from py point of view!

guillaumeeb

We're almost there, but I feel there are two things to address:

I don't think we want to raise an exception if no shebang is define (but I may be wrong)
We shouldn't put the shebang kwarg or config value in all the docs, as the default is perfectly fine for most cases.

Sorry to be a bit pernickety...

guillaumeeb · 2018-11-06T19:39:17Z


+        if shebang is None:
+            raise ValueError("You must specify a batch script interpreter to use"
+                             " like ``#!/usr/bin/env bash``")


Is this really a problem if there is no shebang?
What concerns me here is that this exception might break existing jobqueue configurations, where no shebang is declared. From my point of view it shouldn't.

We could maybe add a deprecation warning? and then a warning to say that if shebang is None everywhere, automatically set it to #!/usr/bin/env bash to preserve the old configuration

You are right that this breaks existing configs in its current state. But you cannot submit a batch script to most schedulers without a shebang:

sbatch: error: This does not look like a batch script. The first sbatch: error: line must start with #! followed by the path to an interpreter. sbatch: error: For instance: #!/bin/sh

Wait, I may be wrong about this. I need to try this PR to see if it breaks my config. Maybe it doesn't, default value should be taken from the module default jobqueue.yaml file.

looking at the code I think you're right. It should default to bash. I'll wait for you to try it though...

Sorry did not find time today, and this will not be possible until friday. You could verify it though, just create a jobqueue.yaml config file, and do not add a shebang line.
I believe the only way to reach that code is to create a local config file, and set shebang to null in it. I'm thus in favor of removing this code, as it seems very unlikely a user run in this case without knowing it.

guillaumeeb · 2018-11-06T19:43:31Z

        name: dask-worker
        cores: 36
        memory: 270GB
+        shebang: '#!/usr/bin/env bash'


You should remove the shebang from "real life" configurations, as keeping the default shebang is perfectly fine apparently for those. This include ARM, but also above Cheyenne and Nersc.

Are these what the DOE are using as their jobqueue.yaml ? in that case they 'should' set the shebang k-v in this updated version. Currently the config is set per-scheduler and the shebang is pulled from the scheduler's k-v config. This is solved if we use the deprecation method I suggested. Although I think it's best practice for the documentation to show these as set so people understand.

I think they shouldn't if they keep the default value, see my response to the other comment. I will try this tomorrow. Default shebang value should be inherited from default config if not specified.

And so based on the above discussion, I think we really should remove the majority of examples containing shebang in the doc, only keeping a few specific. Maybe add the config you are using in your cluster in the example section?

I deleted a bunch, I only left it in places where the docs describe the extra args as well!!

guillaumeeb · 2018-11-06T19:44:13Z


   cluster = PBSCluster(processes=18,
                        threads=4,
+                        shebang='#!/usr/bin/env bash',


Still the same, shebang is optional, maybe put it once, but not everywhere.

guillaumeeb · 2018-11-08T10:05:08Z

You've got flake8 checks failing here.

willirath · 2018-11-09T17:21:42Z

Looks like this is all fine now?

guillaumeeb · 2018-11-09T19:59:58Z

Yes ! Thanks to both of you here, merging.

willirath requested changes Nov 5, 2018

View reviewed changes

guillaumeeb reviewed Nov 5, 2018

View reviewed changes

danpf added 3 commits November 5, 2018 07:19

Switch to env bash, and add more shebang tests

cd46edc

Previously shell was default "/bin/bash" now it respects the environment. Also shebang tests were only in slurm, now there is one for each scheduler that we support.

Update shebang tests to set cores and memory

1b09b77

Add tests for default shebang

eed3069

guillaumeeb requested changes Nov 5, 2018

View reviewed changes

update tests to use pytest.parametrize

2dba863

guillaumeeb approved these changes Nov 5, 2018

View reviewed changes

Update documentation and shebang as required arg

ce7b5dd

I updated the documentation to reflect shebangs being required. because otherwise a batch script could be submitted without an interpreter. I also added a test to cover this.

danpf added 4 commits November 5, 2018 16:03

Add mock as requirement for tests

03b5dc6

Add mock as a true dependency

50f93be

fix dask.config.get import

c4b3f01

fix bad mock logic

251d066

willirath requested changes Nov 6, 2018

View reviewed changes

danpf added 2 commits November 6, 2018 07:37

remove mock requirement

9f60e4c

remove mock requirement

4e66efa

willirath reviewed Nov 6, 2018

View reviewed changes

danpf added 2 commits November 6, 2018 08:28

fix shebang test failing

9afe912

fix config setter to use dictionary

d439ed7

willirath approved these changes Nov 6, 2018

View reviewed changes

guillaumeeb requested changes Nov 6, 2018

View reviewed changes

docs shebang comma to equals sign

257d79b

danpf added 3 commits November 7, 2018 23:12

cut back on shebang documentation

c64f718

merge with remote

a2fbbb2

remove excess shebang test

d55763c

fix flake8 test

72e1546

guillaumeeb approved these changes Nov 9, 2018

View reviewed changes

guillaumeeb merged commit ea1b4ee into dask:master Nov 9, 2018

danpf deleted the danpf/fix_shebang branch November 9, 2018 22:54

		assert ' --preload mymodule' in cluster._command_template


		def test_shebang_settings():

Uh oh!

Conversation

danpf commented Nov 5, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

willirath commented Nov 5, 2018

Uh oh!

guillaumeeb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guillaumeeb commented Nov 5, 2018

Uh oh!

guillaumeeb commented Nov 5, 2018

Uh oh!

guillaumeeb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guillaumeeb commented Nov 5, 2018

Uh oh!

danpf commented Nov 5, 2018

Uh oh!

willirath left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

willirath commented Nov 6, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

willirath left a comment

Choose a reason for hiding this comment

Uh oh!

willirath commented Nov 6, 2018

Uh oh!

guillaumeeb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

willirath left a comment •

edited

Loading