Add new worker specs in groups

[mrocklin]

Currently a subclass of SpecCluster can generate new workers for the spec with SpecCluster.new_worker_spec, which returns a name and a worker configuration. This is typically called by scale.

while len(cluster.worker_spec) < target_workers:
    name, spec = cluster.new_worker_spec()
    cluster.worker_spec[name] = spec

However, in both HPC and GPU cases we want/need to create workers in groups. For this reason it might be nice to change the API here to return a dictionary of new workers

while len(cluster.worker_spec) < target_workers:
    d = cluster.new_worker_spec()
    cluster.worker_spec.update(d)

See dask/dask-jobqueue#307 (comment) for motivation. This also comes up with GPU work, where we want to add a new node and have one worker per GPU on that node.

[mrocklin]

@jacobtomlinson this would probably affect you

[jacobtomlinson]

This would also be useful in ECSCluster as tasks can be submitted in batches and are throttled at one API call per second..

[mrocklin]

It sounds like we might want to consider telling new_worker_spec how many workers we might want. This would also help with array jobs in HPC centers.

[mrocklin]

For now, I'm going to dict PR as a first step, but keep this open for discussion on increasing the size of the dict in the future.