x/build: ppc64le buildlet stuck dead after GCS failure

[bradfitz]

The ppc64 buildlets run via a systemd unit that's:

[Unit]
Description=Go builder buildlet
After=network.target

[Install]
WantedBy=network-online.target

[Service]
Type=simple
ExecStartPre=/bin/sh -c 'cd /usr/local/bin; /usr/bin/curl -R -f -z buildlet-stage0 -o buildlet-stage0 https://storage.googleapis.com/go-builder-data/buildlet-stage0.linux-ppc64le && chmod +x buildlet-stage0'
ExecStart=/usr/local/bin/buildlet-stage0
Restart=always
RestartSec=2
StartLimitInterval=0

But I noticed that 1 of the 5 was always offline. Looking at the logs:

Jul 01 17:42:28 go-ppc64le-1 systemd[1]: buildlet.service: main process exited, code=exited, status=1/FAILURE
Jul 01 17:42:28 go-ppc64le-1 systemd[1]: Unit buildlet.service entered failed state.
Jul 01 17:42:30 go-ppc64le-1 sh[27020]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Jul 01 17:42:30 go-ppc64le-1 sh[27020]: Dload  Upload   Total   Spent    Left  Speed
Jul 01 17:42:30 go-ppc64le-1 sh[27020]: [155B blob data]
Jul 01 17:42:30 go-ppc64le-1 buildlet-stage0[27025]: 2016/07/01 17:42:30 error untarring /usr/local/go-bootstrap.tar.gz to /usr/local/go-bootstrap:
Jul 01 17:42:30 go-ppc64le-1 buildlet-stage0[27025]: gzip: stdin: not in gzip format
Jul 01 17:42:30 go-ppc64le-1 buildlet-stage0[27025]: tar: Child returned status 1
Jul 01 17:42:30 go-ppc64le-1 buildlet-stage0[27025]: tar: Error is not recoverable: exiting now
Jul 01 17:42:30 go-ppc64le-1 systemd[1]: buildlet.service: main process exited, code=exited, status=1/FAILURE
Jul 01 17:42:30 go-ppc64le-1 systemd[1]: Unit buildlet.service entered failed state.

Why isn't it a valid gzip?

# cat /usr/local/go-bootstrap.tar.gz
Service Unavailable

Ah.

That's bogus.

[gopherbot]

Change https://golang.org/cl/179421 mentions this issue: cmd/buildlet/stage0: use httpdl instead of curl for getting bootstrap tarball