Emphasize which python executable is used to run the workers in SLURM

[zaccharieramzi]

I recently had an error when using a SLURMCluster where a module I thought I had loaded using job_extra couldn't be loaded in my python script.

Specifically I had done module load tensorflow-gpu/py3/2.1.0 in my job_extra and I couldn't import tensorflow (this on Jean Zay).

It took @lesteve 5 minutes to notice this when looking at the job script generated but it wasn't obvious to me to even look there, so I thought it deserved a bit more emphasis maybe here, that the python used to run the workers is the one used to launch the scheduler.

I am planning to submit a small PR.

[Ovec8hkin]

Just checking, are you aware you can directly specify the python executable to use for each dask worker by using the python parameter when you initialize your cluster object?

Or is this more of a "documentation is unclear" question?

[lesteve]

The idea was to have it as some confusing of weird error that can happen if you are not careful. Also the documentation is not mentioning that Dask workers and Dask scheduler should have the same Dask, distributed and dask-jobqueue version otherwise subtle but painful bugs may hit you.

@Ovec8hkin in general I find it easier to use the same conda environment on both sides (Dask scheduler and Dask workers) rather than using the python parameter. I would be interested if you have use cases where python parameter is particularly useful or even required.

And while I am at it a little tip but client.get_versions(check=True) is useful when weird things happen to make sure that the versions are the same on the Dask scheduler and the Dask workers.

[willirath]

I would be interested if you have use cases where python parameter is particularly useful or even required.

As we're seeing increased availability of container envs (mostly) singularity across HPC centres, I expect to use / need a mechanism for having the dask worker call executed in a container. Overriding / prefixing the python kwarg would allow for doing this.

[ocaisa]

I would be interested if you have use cases where python parameter is particularly useful or even required.

We also use it when we have the scheduler on one node type and workers on our KNL partition, the path to python can change in such a use case.

[zaccharieramzi]

Closing this since #409 has been merged.