Notification when deploy finishes

[bmarini]

After a deploy, I usually poll emp ps to know when the last release has been completely phased out.

Wouldn't it be nice to receive a notification instead.

[ejholmes]

I think this would be great if we could do it. Just to use this to record some ideas:

It would be nice if we had a done channel that we could listen on after a scheduler.Submit to know when the deployment fully completes (all of the ECS services have been updated with the new task definitions, and their running count matches their desired count) or when it's canceled (e.g. a new deployment supercedes the old one).

Within the Custom::ECSService we could return the id of the ECS deployment that the update creates, and set this as an output on the stack. Once the stack update completes, wait for all of the deployments to stabilize (move to ACTIVE state, or INACTIVE).

That would be relatively straightforward to do.

[ejholmes]

It would also be nice if the done channel was just a channel of status updates, so we could put some useful information in the emp deploy stream, and then also update the status of a release in emp releases (e.g. this release is pending, inactive, active, etc).

[ejholmes]

I worked on a proposal for how I think we should do this, and put it in the wiki. I think fundamentally what we should do, is make the empire.Deploy method return when the new release is actually running on the cluster, rather than simply submitted to the backend.

[mwildehahn]

Part one of this will be done with: #917

[mwildehahn]

@ejholmes the last part of this i wanted to implement was always sending a notification when the release has stabilized, regardless of whether or not you specified streaming the status updates.

Right now, DeployEvent is either triggered after the release has stabilized if you've enabled the stream, or after it has been submitted if you haven't.

It feels weird that the empire event behaves differently depending on a small detail like whether or not the user wanted to stream updates on the cli.

I like how all events are published within https://github.com/remind101/empire/blob/master/empire.go, to maintain consistency with that, I think we should do something like:

• publish a new DeployStartedEvent here: https://github.com/remind101/empire/blob/master/empire.go. something like:

mhahn started deploying remind101/acme-inc:1370b0390e4f4c572e521713d416248057b2932d to remind101/acme-inc dev (v99)

• create a done channel that gets passed to Deploy: https://github.com/remind101/empire/blob/master/empire.go#L572
• when the done channel closes, publish the DeployEvent

that way the events are always being published at the same time, regardless of whether streaming was specified and we also don't introduce publishing events in other parts of the stack.

the DeployEvent then means that the deploy has been released and the services have stabilized.

We can also setup a timeout within the Deploy function to fire a DeployTimeoutEvent when the deploy is taking longer than expected to alert people to issues.

Thoughts?

[ejholmes]

With the recent changes that @mhahn did, I think we can go ahead and close this. When the -s flag is provided to emp deploy, the deployment will wait until all services are stable before returning. This is also the default behavior in the GitHub Deployments integration.