Builds failing with docker errors

[lox]

It looks like we've had a regression on #266 sometime after 2.3.0 where builds are occasionally failing with docker connection errors.

E.g "transport: dial unix /var/run/docker/containerd/docker-containerd.sock: connect: connection refused"

My suspicion is that these are related to a race condition where we configure docker on boot and then restart it. We merged #377 to address this and will be looking to put out a release soon.

[gugahoi]

We've had this happen today again. Have the agent handy if anything is needed but from the brief inspection this is related to containerd not being ready when docker is starting up. Is docker responsible for invoking containerd with docker-containerd?

Also is it possible this is somewhat related: moby/moby#36173 ?

[lox]

A speculative fix went out for this in https://github.com/buildkite/elastic-ci-stack-for-aws/releases/tag/v2.3.4, are you on that @gugahoi?

[lox]

Yup, that moby bug looks definitely possible, there are regular classes of docker bugs around race conditions on docker's daemons restarting, which we try and mitigate by being very careful about how we do that on stack bootup.

[lox]

Keeping this open until we confirm it's fixed.

[lox]

My other suspicion is the incredibly old version of Upstart on Amazon Linux. We might be seeing a variant of moby/moby#6647.

[gugahoi]

Is it possible we can fail the agent when docker ps is not responsive over 5 tries here ?

It seems buildkite is simply ignoring the fact that docker ps is not responding and moving on. Would it be better to have the agent not register to schedule builds then? Maybe reboot, or do a more forced sudo service docker restart or something along those lines?

[ankurd1]

Hey, we are running stack 2.3.5 and seeing a similar error.

An EC2 instance was started by the stack and didn't have docker running. Buildkite kept sending jobs to it and all the jobs kept failing.

I have the machine still running so I can get you any logs that you need to debug. We've been seeing such errors more and more and people have started blaming the CI team for flakiness :(

Would really appreciate if you guys could look into this! Thanks!

[lox]

Haven't seen this in a long while, closing this out.