Scheduled tasks

[ejholmes]

https://github.com/remind101/empire/wiki/Roadmap#scheduled-tasks

Closes #300.

This adds first class support for scheduled tasks. It uses a combination of CloudWatch events, lambda, and ECS to run scheduled jobs in separate containers.

Lots of advantages to this, over doing some kind of in process cron:

1. Memory/CPU constraints can be controlled for each scheduled task, since the jobs run in a short lived ecs tasks.

2. emp restarts won't affect the scheduled tasks. Scheduled tasks that are already running will continue to run.

3. Scheduled tasks will show up in emp ps:

   $ emp deploy remind101/acme-inc:scheduled-job
   $ emp scale scheduled-job=1
   $ emp ps
   v54.scheduled-job.9b649d34-b4f5-4fb7-bfe2-889d80dbd3c9  1X  RUNNING  11s  "sleep 60"
   v54.web.fd130482-675f-4611-a599-eb0da1879a10            1X  RUNNING   9m  "acme-inc server"
   $ sleep 60
   $ emp ps
   v54.scheduled-job.0aec4323-e518-400c-9c50-e2b0cbd5babf  1X  RUNNING  12s  "sleep 60"
   v54.web.fd130482-675f-4611-a599-eb0da1879a10            1X  RUNNING  15m  "acme-inc server"

Scheduled tasks are configured in the Procfile, using the new extended Procfile format. To mark a process as a scheduled job, you add a cron: field to the process, with a cron expression for when it should run:

web:
  command: ./bin/web
scheduled-job:
  command: ./bin/scheduled-job
  cron: * * * * * # Run once every minute

By default, scheduled processes are disabled by default (like other processes), and need to be scaled up to run:

$ emp scale scheduled-job=1

You can scale past 1 to run multiple tasks at once when the cron event triggers:

$ emp scale scheduled-job=2 # run 2 of these when triggered
$ emp ps
v56.scheduled-job.16d9458f-99d8-4c7b-8631-da2d16b45676  1X  RUNNING  39s  "sleep 60"
v56.scheduled-job.28edbe42-3b2d-4cb6-8434-8f9331ad1ccf  1X  RUNNING  39s  "sleep 60"
v56.web.1507b937-529d-4179-aaef-63fa18f073f0            1X  RUNNING   9m  "acme-inc server"

And you can scale back down to 0 to disable the scheduled process:

$ emp scale scheduled-job=0

TODO

• Throw errors in legacy ECS backend.
• Validate cron expression.
• Dedup the lambda function?
• More tests
• Docs

[ejholmes]

As part of this, I ended up cleaning up some of the template building by replicating some of troposphere in Go. If it gets cumbersome to review, I can pull that out into a separate PR and rebase this.

[ejholmes]

And, to explain the implementation a little more, it's actually pretty simple:

For processes that have a Schedule attached to them, we create 3 main resources:

1. An AWS::ECS::TaskDefinition for the process.
2. An AWS::Events::Rule with an appropriate ScheduleExpression (e.g. cron(...)), which triggers the following lambda function:
3. An AWS::Lambda::Function that simply performs an ecs.RunTask call. It sets startedBy so that it shows up in emp ps.

So, it's just CloudWatch Events -> Lambda -> ECS task.

[ejholmes]

Remove this

[phobologic]

👍 So cool.