Add linux-armv6l builders (armhf)

[dagood]

• Related: Add linux-arm64 builders #173

The arm64 hardware we have access to should be able to build this. It looks like Google is using arm64 machines to build their armv6l releases:

https://github.com/golang/build/blob/ce4ab8eee5f9e0633d12454f2e5b19824b73f4a7/dashboard/builders.go#L562-L581
https://github.com/golang/build/tree/d652ff4bfce54f226acee801facd43b1dc1174fb/env/linux-arm/aws

The Docker Hub official golang images use the armv6l archive for armhf images:
https://github.com/docker-library/golang/blob/9f431d40c96988dd4302eee93789fa6f071354d2/1.17/bullseye/Dockerfile#L33-L43

		'armel') \
			export GOARCH='arm' GOARM='5' GOOS='linux'; \
			;; \
		'armhf') \
			url='https://dl.google.com/go/go1.17.8.linux-armv6l.tar.gz'; \
			sha256='3287ca2fe6819fa87af95182d5942bf4fa565aff8f145812c6c70c0466ce25ae'; \
			;; \
		'arm64') \
			url='https://dl.google.com/go/go1.17.8.linux-arm64.tar.gz'; \
			sha256='57a9171682e297df1a5bd287be056ed0280195ad079af90af16dcad4f64710cb'; \
			;; \

[dagood]

I've been having trouble getting a hold of a known-good armv6 base image to use in CI.

Upstream's Dockerfile uses FROM arm32v6/debian:buster, which doesn't exist. (Anymore?)

I'm able to docker pull arm32v5/debian:buster and docker pull arm32v7/debian:buster, but not arm32v6.

According to https://github.com/docker-library/official-images#architectures-other-than-amd64, v5 is not on the list of supported architectures. But, I don't know if the omission is intentional.

Ubuntu 18.04 (our current base image for Go builds) only has v7+, arm32v7/ubuntu:18.04.

The only Linux distro I see in arm32v6 is alpine, but it uses musl instead of glibc so we can't use it here.

Options I see to get something shipped:

• Cross-compile linux-armv6l using armv7 (armhf). Hard to say how dangerous this might be. Getting cgo working may be a significant challenge. (See beginning of Add linux-arm64 builders #173 before we got the hardware to build arm64 natively.)
• Ship an armv7 (armhf) build of Go as if it were linux-armv6l. This is ok for the image scenario (because we're delivering this binary in an image labeled "armhf"), but potentially not ok for anyone directly downloading our binaries to an armv6 machine.
• Diverge from upstream's set of supported platforms: build linux-armhf, deliver it like that, and ship it in our armhf images. This might cause confusion due to the mismatch, in particular because upstream calls armv6l "linux-arm" in some places, like GOOS/GOARCH. (There is GOARM=6 vs. GOARM=7 when GOARCH=arm, but not clear to me yet if this clears it up enough.)

[dagood]

It's looking like cross-compilation will work:

A major thing I found that I didn't realize when I tried cross-compilation for #173 is that linux-arm doesn't support the race compiler, so there's no need to worry about building the race stdlib. 😄 (That's where I ran into issues with cgo support when I was trying to cross-build linux-amd64 -> linux-arm64.)

Actually, even though linux-arm64 does support the race compiler, upstream's build of Go doesn't include a prebuilt race-compiled stdlib like ours currently does! It's possible that removing it would let us use cross-compilation there, too. If we did that, we could consolidate our build hardware on amd64 machines. Either way, we should remove the prebuilt race-compiled stdlib from our arm64 build to match upstream and save ~30 MB.

About go cgo support: a cross-compiled amd64 -> bin/linux_arm/go does report CGO_ENABLED=1 when I run bin/linux_arm/go env under QEMU, so we may be fine there. I also tried building armv6 Go on an RPi 4 (Debian arm64) that I set up yesterday, and it reported the same. (No QEMU necessary here, because this particular arm64 processor can run armv6 programs as long as they're statically linked, like go.)

On my RPi, if I install gcc-arm-linux-gnueabi and build a statically-linked cgo-using app with CC=arm-linux-gnueabi-gcc go build -ldflags "-linkmode external -extldflags -static" ., I'm also able to run that. (A dynamically linked cgo app won't run on my machine, I think because my system's ld isn't what it expects.)

(Edit: Later, I also installed Debian armhf on the RPi 4, and it worked without static linking.)

I'm feeling fairly confident now that cross-compiling is fine.

The potentially fiddly part now is packaging up a cross-build as if it's an ordinary build. The host platform binaries need to be removed and the target platform binaries shuffled around.

[dagood]

The way we're approaching this for now (cross-compilation) is different from the official builds. Summing it up here to have the info in one place.

Official builds all native to the OS:

• Host Linux amd64: linux-amd64 Go -> linux-amd64 official Go image
• Host Windows amd64: windows-amd64 Go -> windows-amd64 official Go image
• Host Linux arm/v8:
  • Image arm/v8:
    • Linux arm/v8 Go -> Linux arm/v8 official Go image
  • Image arm/v6:
    • Linux arm/v6 Go -> Linux arm/v7 official Go image
      • Note: arm/v6 build of Go is used for the arm/v7 image. ARM backward compatibility is being relied on.
      • It is possible to run an arm/v7 build of Go, but this build flavor is not officially produced or used.
      • This may appear to be cross-compilation because the host is arm/v8 but the image is arm/v6. It isn't, because the OS in the arm/v6 Docker image is arm/v6, and that's what's building Go.

(Above, stuck with "arm/v8" to mean arm64, "arm/v7" for armhf, "arm/v6" to mean armv6l, armel. There's more nuance to it--not all ARM processors are backward compatible, arm/v8 can be in 32 or 64 bit modes, and some of these terms aren't actually synonyms. Using "v6-v8" just makes it easier to keep track of.)

We're still building mostly native, but cross-compiling for arm/v6:

• Host Linux amd64:
  • linux-amd64 Go -> linux-amd64 image
  • Cross-compile Linux arm/v6 Go -> Linux arm/v7 image
• Host Windows amd64: windows-amd64 Go -> windows-amd64 image
• Host Linux arm/v8: Linux arm/v8 Go -> Linux arm/v8 image

[dagood]

Completed. MCR now has amd64/arm64/armhf manifest tags, where the armhf images contain the Go toolset compiled targeting armv6l.