Show attached runs in emp ps.

[ejholmes]

Closes #905

This allows us to easily show attached runs in the output of emp ps:

$ emp run sh -a acme-inc
$ emp ps -a acme-inc
v33.run.dcc5bf133f56                          256:512.00mb  RUNNING  11s  sh
v33.web.f25d25e2-6b39-4fd2-833a-b2353082feb1  1X            RUNNING  23h  "acme-inc server"

This is implemented by using ListContainers to find any containers tagged with attached-run. Obviously this won't work consistently if you have more than 1 instance of Empire pointed at multiple Docker daemons.

My suggestion is that we update the docs to encourage people to point Empire at a single Docker daemon (either a single physical Docker daemon, or a Docker swarm). There's other benefits to doing this:

1. docker pulls's when deploying will generally be faster, since it will hit cache more frequently with a single Docker daemon.
2. It's easier to lock down security group rules if the Docker daemon is off of the host that Empire is running on (if you use the same Docker daemon that Empire is running as, people in emp run's have the same permissions as Empire to create infrastructure).
3. It gives you more control over when runs are shutdown. For example, it's not uncommon that we roll out new AMI's and that process will kill long running migrations. Having a dedicated cluster/instance for emp run's gives you more control to ensure that no runs are in progress.

TODO

• Update Production Best Practices docs about pointing Empire at a separate Docker cluster.
• Update CHANGELOG.
• Run Instances method in goroutine.

[ejholmes]

This should get kicked off in a goroutine.

[phobologic]

I haven't messed w/ docker swarm, but is there anyway we could run docker swarm's API on both empire hosts, and have them controlling the same docker instances? Then we could just point the Empire daemon at the local swarm, and still have a separate set of instances to do this on?

[mwildehahn]

should this be updated?

[ejholmes]

@phobologic yep, that would work well too. Although, I think our production recommendation should be to have emp run's be off of the host that Empire runs in for better security.

[ejholmes]

I think probably the easiest thing to do for most people would be to run a single EC2 host dedicated to emp run's, expose Docker over TCP, then point the Empire daemon at that host. Docker swarm can be used to scale out when 1 host isn't enough (probably unlikely). Getting it off host is good so that emp run's don't get the same IAM perms that Empire has.

I think for now, we should make this feature experimental and only enable it with a flag. I'm sure for some people, the extra complexity in getting Docker over TCP securely isn't worth it.

[ejholmes]

We should also implement the Stop method in this, so that attached runs can be stopped like everything else.

[ejholmes]

This should be good to go. I went ahead and feature flagged it with an --x.showattached flag, since it requires some infrastructure changes.

[mwildehahn]

shouldn't this be fmt.Sprintf("%s=%s", runLabel, "attached")?

[ejholmes]

good catch! I'll update.

[mwildehahn]

👍