Improve the visibility and resiliency of AWS Batch processing of CSVs

[jwalgran]

Overview

AWS Batch jobs can fail. We designed the processing to be idempotent, so it is safe to rerun failed jobs, but we don't have an automated system for doing so.

Is your feature request related to a problem? Please describe.

You cannot see that a job has failed within the application, only in the AWS Batch console.

Describe the solution you'd like

• Evaluate whether or not the built-in AWS Batch job retry feature is appropriate for our workflow.
• Re-run failed jobs a fixed number of times.
• Alert development/operations when jobs fail (Reference: https://docs.aws.amazon.com/batch/latest/userguide/batch_sns_tutorial.html)

[hectcastro]

I would be interested in discussing this a bit with whoever pulls it. In RF we've made use of:

• Job retries (which require some additional application logic to track the retry count)
• Job timeouts (to prevent jobs that hang from hogging up resources)
• Send Rollbar notifications when jobs fail (A Ruby Goldberg device that uses part of the strategy in SNS article above, but to wake up a Lambda that invokes Rollbar)