agent does not always gracefully shut down

[bradrydzewski]

Watchtower (utility to auto-upgrade Docker images) is not properly stopping the Agent on upgrade, which causes stuck builds. The root cause is unknown and requires more research. Hopefully it is something simple ...

[bradrydzewski]

Note that the inability to gracefully shut down properly also impacts autoscaling. In some cases the agent receives a SIGTERM and blocks indefinitely even though no builds are running. The end result is the autoscaler runs docker stop and it takes the full timeout period to stop the container even though no builds are running.

---

the underlying problem is with how we used Context and has been fixed in the next upcoming iteration of the docker runner, which can be found at drone-runners/drone-runner-docker.

[c2h5oh]

@bradrydzewski it's -t 60 or --time=60 and the unit is seconds, not minutes

[bradrydzewski]

sorry, I should have been more specific ... the autoscaler does not actually call docker stop -t it makes the equivalent API call. The API requires the timeout value in milliseconds.

[bradrydzewski]

we have a new implementation of the docker runner that solves this issue. This should be a drop-in replacement for the agent. Some of the environment variables changed, but we do have some logic to try and map old variables to new variables here (if something is missing please send a pull request)

drone/drone-runner-docker:latest

I am going to leave this issue open since we are still testing this runner at cloud.drone.io. Once testing is complete we will tag a release, update the documentation, and then close this issue.