Autoscaling doesn't work for pending jobs

[basnijholt]

I am using cyclecloud with a SLURM queueing system in combination with dask (a Python package that manages the bookkeeping and task distribution such that one doesn't have to write jobscripts and collect the data.)

It's possible to use that adaptive scaling feature of dask which means, that as the load on all my nodes becomes high, it automatically creates new jobs using (in my case) the following job script:

#!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -e TMPDIR/dask-worker-%J.err
#SBATCH -o TMPDIR/dask-worker-%J.out
#SBATCH -p debug
#SBATCH -A WAL
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=15G
#SBATCH -t 240:00:00
JOB_ID=${SLURM_JOB_ID%;*}

export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1

/gscratch/home/t-banij/miniconda3/envs/py37_min/bin/python -m distributed.cli.dask_worker tcp://10.75.64.5:43267 --nthreads 1 --memory-limit 16.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60 --local-directory TMPDIR

This results in many PENDING jobs. Unfortunately, no new nodes are started automatically, I still need to go into the web interface and start new nodes by hand, rendering the autoscale feature I am using useless.

Seems related to #1, #5, and #9.