Allow users to specify batches per worker

[vishalbollu]

Description

Currently, clients must specify the number of workers required for the Job in the Job Submission. If the client doesn't know the total number of batches, they may over/under provision the required for number of workers for the job to complete in a timely manner. Allow clients to specify BatchesPerWorker to scale the number of workers based on the number batches in the job. Make max_workers a required field to prevent the users from provisioning an unexpected number of workers (if there is a mistake in the job submission).

A proposed solution for worker schema in the job submission might be:

"workers": {
  "desired":
  "batches_per_worker":
  "max_workers": 
}

Validations:

• only desired or batches_per_worker is specified but not both
• if batches_per_worker is specified, max_workers must be specified

Motivation

• Client can provide the desired number of workers to complete the job in a timely manner without figuring the number of batches in the job.