Add support for PBS clusters

[jcrist]

Adds initial support for the PBS job queue system.

This works by using an intermediate script (dask-gateway-jobqueue-launcher) to do the heavy lifting of submitting/killing jobs. The gateway user then needs sudo access for this script for all dask users.

The gateway also needs to be a PBS operator in order to check the job status of all users. This could be worked around, but avoiding being an operator would likely be more expensive as sudo would have to be used for status queries as well.

The configurable fields exposed here were my best attempt at determining which parameters were important for users (balancing flexibility with ease of use). Not being a PBS user myself, I've likely done a poor job at determining what's important.

[mrocklin]

@jcrist do you have any thoughts on how you might leverage the dask-jobqueue infrastructure or community? It would be unfortunate to cause a significant technology split here. Maybe there is some component in specifying jobs that could be shared between the two projects?

cc @jhamman @guillaumeeb

[jcrist]

I don't see a way to use the same codebase - there's too much work that needs to be done to adequately support multiple users in an efficient and secure manner.

FWIW, this didn't take terribly long to write up, and I suspect much of this could be pulled into a base class to support other schedulers.

Leveraging the community would be useful though, I'm not familiar with these systems and may not be doing things 100% correct. Code review would be quite welcome.

[mrocklin]

I'm not suggesting that you use the dask-jobqueue codebase as is, I'm suggesting that there might be some component that could be designed that both systems could use. The advantage of this is that dask-gateway would then benefit from a bunch of other users. Otherwise I suspect that you'll be needled with hundreds of "oh, my job scheduler uses this special parameter, can you add that too please?" requests. However the dask-jobqueue folks have handled things seems to have handled things.

[mrocklin]

I think the first step here would be for you to look through the dask-jobqueue codebase, talk with the maintainers of that project, and then maybe discuss ways in which job-specification might be generalized.

[jcrist]

I think the first step here would be for you to look through the dask-jobqueue codebase

I have been using their codebase as inspiration, but suspect that for now it's easier to copy and augment than it is to figure out what should be shared. Shared components could maybe be extracted at a later point once it's clearer what that means.

I would be more than happy to talk to the maintainers of dask-jobqueue, but think figuring out a shared dependency at this point is more effort than it's worth.

[guillaumeeb]

Ok I took a quick look at the code from this PR, some thoughts:

• It would be really good if we could share some components, it's a shame that we do some similar things in two different Dask modules.
• dask-gateway jobqueue system has some advantages or specificities:
  • It it designed to launch jobs from a central place for multiple users, so has some sudo logic in it, this could probably be added to dask-jobqueue.
  • It handles the submission of Dask Scheduler as a separated job, a long awaited feature for dask-jobqueue.
• dask-jobqueue could help dask-gateway :
  • It handles many options and Scheduler specificities :
    • avoid using qstat too often for ensuring jobs status (we ask our users not to use it automatically more than one per minute)
    • provide HPC good configuration with local-directory and interface (Infiniband) parameters
  • It is already compatible with multiple schedulers

This also looks quite similar to batchspawner from Jupyterhub, which is not surprising if I understand correctly the "Daskhub" nature of dask-gateway?

Could or do you use Dask Cluster objects for other infrastructure gateway? Don't you think we could design both Cluster and dask-gateway to be compatible?

[jcrist]

Thanks @guillaumeeb for reading through the PR. Responding to a few comments inline:

It it designed to launch jobs from a central place for multiple users, so has some sudo logic in it, this could probably be added to dask-jobqueue.

Why would dask-jobqueueneed support for launching jobs as other users?

avoid using qstat too often for ensuring jobs status (we ask our users not to use it automatically more than one per minute)

I'm not super familiar with these systems. Why the severe restriction (does it result in a noticeable lag on the scheduler)? In this case we make many requests to qstat, but it's a single request for tracking all user's jobs, not one-per-job. The frequency could be turned down, the reason we check at all is to catch failures early rather than waiting for a failed job to timeout on startup.

This also looks quite similar to batchspawner from Jupyterhub, which is not surprising if I understand correctly the "Daskhub" nature of dask-gateway?

This is correct - batchspawner's code was also looked at for inspiration.

Could or do you use Dask Cluster objects for other infrastructure gateway? Don't you think we could design both Cluster and dask-gateway to be compatible?

We don't rely on any of the existing dask cluster implementations inside dask gateway. The gateway server library (dask-gateway-server) doesn't depend on dask at all (requirements here), it only handles authentication and process management. I think we could potentially find a way to share code, but right now I think focusing on that right now is the wrong solution. The requirements for a dask-gateway ClusterManager and an existing dask Cluster implementation are sufficiently different that many things have to be rewritten (we have to support multiple users securely, save state to a database, handle authentication, robust failure handling, etc...). Think of it like ipyparallel - that codebase already had logic for launching jobs on many job queue systems, but it didn't make sense for dask-jobqueue to rely on it for starting jobs.

---

For now I plan to expand support for the various existing dask backends. If at some point it becomes clear how we could share some code we can think about splitting that out then. Of course I welcome any feedback/features/issues/comments you might have - so far this has been a single-person project and I'm definitely not as familiar with job-queue systems as you/the dask-jobqueue team are.

[guillaumeeb]

Why would dask-jobqueue need support for launching jobs as other users?

I was unclear, I just meant you could configure or extend dask-jobqueue to launch job with sudo.

Why the severe restriction (does it result in a noticeable lag on the scheduler)?

Exactly, it can overwhelm the scheduler if a lot of users do frequent qstat. We tell our users to avoid more than one per minute in their program. This is probably very conservative, but it prevent missuses.

it's a single request for tracking all user's jobs, not one-per-job.

OK, that's a good thing!

The requirements for a dask-gateway ClusterManager and an existing dask Cluster implementation are sufficiently different that many things have to be rewritten (we have to support multiple users securely, save state to a database, handle authentication, robust failure handling, etc...)

I think we were trying to define this ClusterManager in dask-distributed (dask/distributed#2235). I agree this is not the same as what you describe, but this is a step towards mutualisation.

Anyway, I'm happy to try to address this later on (and I just don't have the time to give you support right now), let's just keep in mind that there could be some generalization.