x/build: fix broken Docker on arm64 packet builders

[FiloSottile]

Some of the arm64 packet builders stopped working.

Over time, the docker daemon lost track of more and more exited containers.

root@go-builder:~# docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
e9da62e0e524        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 seconds ago       Up 2 seconds                            packet07
be4cad637e87        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet09
4198b1b3aaf5        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet15
48bada399702        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet19
5ca10cbf91ad        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet20
8c78fd256b38        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   2 hours ago         Up 2 hours                              packet16
4b89ff505063        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet01
b7918b0fc69e        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet03
3ea996a7951c        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet11
51d2bb94fca2        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet13
7e3530aa3598        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 hours ago         Up 4 hours                              packet02
bed226aba206        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 weeks ago         Up 3 weeks                              packet04
4e4eba80cfb6        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 weeks ago         Up 4 weeks                              packet12
aae8adfd568c        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   5 weeks ago         Up 5 weeks                              packet17
a56f27c8b515        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 months ago        Up 3 months                             packet14
39f54b77d230        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 months ago        Up 4 months                             packet08
e64dd9975d1c        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 months ago        Up 4 months                             packet06
595355800eba        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 months ago        Up 4 months                             packet05
69559e6aa207        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   5 months ago        Up 5 months                             packet10
39b1689d4c2a        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   5 months ago        Up 5 months                             packet18

root@go-builder:~# docker logs packet04
2018/03/21 03:40:36 network is up.
2018/03/21 03:40:36 Downloading https://storage.googleapis.com/go-builder-data/buildlet.linux-arm64 to ./buildlet.exe ...
2018/03/21 03:40:37 Downloaded ./buildlet.exe (7302642 bytes)
2018/03/21 03:40:37 buildlet starting.
2018/03/21 03:40:37 Dialing coordinator farmer.golang.org:443 ...
2018/03/21 03:40:37 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
2018/03/21 03:40:38 Registering reverse mode with coordinator...
2018/03/21 03:40:38 Connected to coordinator; reverse dialing active
2018/03/21 08:37:19 buildlet reverse mode exiting.

buildlet reverse mode exiting is followed by os.Exit.

Trying to exec suggests Docker lost track of them.

root@go-builder:~# docker exec -it packet04 bash
rpc error: code = 2 desc = containerd: container not found

There is no spot-on Moby issue that matches this, but some similar ones over the last year. Upgrading and restarting Docker assuming it's a known issue. Will reopen if it keeps happening.

[bradfitz]

What's the current version of Docker and what are we upgrading to?

[FiloSottile]

Before:

root@go-builder:~# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Tue Jan 31 23:35:02 2017
 OS/Arch:      linux/arm64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Tue Jan 31 23:35:02 2017
 OS/Arch:      linux/arm64

The following packages will be upgraded:
  apparmor apt apt-utils base-files bsdutils cloud-guest-utils cloud-init distro-info-data dnsmasq-base docker.io dpkg initramfs-tools initramfs-tools-bin
  initramfs-tools-core iproute2 isc-dhcp-client isc-dhcp-common klibc-utils kmod libapparmor-perl libapparmor1 libapt-inst2.0 libapt-pkg5.0
  libaudit-common libaudit1 libavahi-client3 libavahi-common-data libavahi-common3 libblkid1 libcryptsetup4 libfdisk1 libgcrypt20 libgnutls-openssl27
  libgnutls30 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libklibc libkmod2 libmount1 libpam-modules libpam-modules-bin libpam-runtime libpam0g
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib librados2 librbd1 libseccomp2 libsmartcols1 libsystemd0 libudev1 libuuid1 logrotate mdadm
  mount ntp ntpdate open-iscsi python python-apt python-apt-common python-minimal python2.7 python2.7-minimal python3-apt runc ssh-import-id sysstat
  systemd systemd-sysv ubuntu-fan udev unattended-upgrades util-linux uuid-runtime xdg-user-dirs

After systemctl restart docker:

root@go-builder:~# docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.6.2
 Git commit:   092cba3
 Built:        Thu Nov  2 06:39:28 2017
 OS/Arch:      linux/arm64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.6.2
 Git commit:   092cba3
 Built:        Thu Nov  2 06:39:28 2017
 OS/Arch:      linux/arm64
 Experimental: false

[FiloSottile]

Builders seem happy. Closing and hoping not to reopen.