Aws Batch Job Utilities

Overview

I want to run a lot of distributed batch jobs on AWS Batch with AWS Fargate Spot instances (very cheap, but interruptible). To do that, you need to

And some more. This is what these scripts do.

docker_upload.py can build a docker container around a command
aws_batch.py can be used to setup, submit (many times) and teardown the jobs.
aws_dispatcher.py is a little trampoline script that is embedded inside a Docker container to read the mapped input (this is an "array job") and invoke the actual command with this input as stdin.

Configure:

Create a file like config.json

Upload a docker container to be run (you have to build it first):

docker_upload.py <docker-image-tag>

Or, build and upload docker container around a Python command:

docker_upload.py <docker-image-tag> --build <program> <arguments>

Setup AWS resources:

aws_batch.py setup

Submit a job:

aws_batch.py submit <docker-image-tag> <input-file>

The input file has one line per worker. The input will be provided to the command as stdin.

Teardown AWS resources:

aws_batch.py teardown