production not on master/ unhandled promise rejection

[vsmart]

While running out the eng standards inventory (#141) I noticed that latest master is not deployed to production. production is still on 112e8ae0.

I started the update and it failed - no healthy tasks could start up because of this error:

(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: maxJobDuration: not a number

next steps

• what introduced this regression?
• deploy a safe fix

cc @mapbox/assembly-line

[vsmart]

notes: I can see that the maxJobDuration was added to ecs-watchbot in mapbox/ecs-watchbot#230.
However, it looks like this is still running on 4.0.0 since #130 so I'm not sure where this issue comes from.

I can reproduce this issue by deploying the current master branch to staging.

[vsmart]

💡 I think I tracked down the issue here.

npm i @mapbox/watchbot^4.0.0, which is run in the Dockerfile, installs watchbot 4.10.0 inside the container. any version of watchbot > 4.5.0 requires the maxJobDuration env variable.

but: the template itself is locked from package-lock.json at watchbot 4.2.0, so the template does not expose this env variable.

deploying #134 will fix this, since it upgrades the locked version of watchbot to 4.10.0 (i.e. above 4.5.0). I've verified this in staging.

for the future, we should also lock the version in the Dockerfile, and consider a way to make sure these too are the same and there are no conflicts.

[rclark]

We should follow ecs-watchbot's installation recommendation and install a prebuilt binary instead of running npm install in the Dockerfile.

[vsmart]

👍 @rclark, the binary installation recommendation makes sense. However, I just ran into the same issue over at ecs-telephone, which is already using the binary version. The binary install in the Dockerfile was grabbing a newer version than the one in package.json. on startup the tasks crash with maxJobDuration is not a number.