x/sys/cpu: add support for ARM

[jpap]

It would be nice to have feature detection for ARM (32-bit) in x/sys/cpu.

Concretely, a new cpu.ARM struct that closely resembles the existing cpu.ARM64 struct, tailored to the ARM specific hardware capabilities. The following fields are proposed, which map directly to the {HWCAP, HWCAP2} auxiliary vector values on Linux and FreeBSD:

HasSWP      bool // SWP instruction support
HasHALF     bool // Half-word load and store support
HasTHUMB    bool // ARM Thumb instruction set
Has26BIT    bool // Address space limited to 26-bits
HasFASTMUL  bool // 32-bit operand, 64-bit result multiplication support
HasFPA      bool // Floating point arithmetic support
HasVFP      bool // Vector floating point support
HasEDSP     bool // DSP Extensions support
HasJAVA     bool // Java instruction set
HasIWMMXT   bool // Intel Wireless MMX technology support
HasCRUNCH   bool // MaverickCrunch context switching and handling
HasTHUMBEE  bool // Thumb EE instruction set
HasNEON     bool // NEON instruction set
HasVFPv3    bool // Vector floating point version 3 support
HasVFPv3D16 bool // Vector floating point version 3 D8-D15
HasTLS      bool // Thread local storage support
HasVFPv4    bool // Vector floating point version 4 support
HasIDIVA    bool // Integer divide instruction support in ARM mode
HasIDIVT    bool // Integer divide instruction support in Thumb mode
HasIDIV     bool // Integer divide instruction support in ARM and Thumb mode
HasVFPD32   bool // Vector floating point version 3 D15-D31
HasLPAE     bool // Large Physical Address Extensions
HasEVTSTRM  bool // Event stream support
HasAES      bool // AES hardware implementation
HasPMULL    bool // Polynomial multiplication instruction set
HasSHA1     bool // SHA1 hardware implementation
HasSHA2     bool // SHA2 hardware implementation
HasCRC32    bool // CRC32 hardware implementation

As I look around, I see code detecting CPU features based on the runtime.goarm value (which is set by the GOARM environment variable at link time), rather than a runtime check. This means that:

1. As runtime.goarm is not const, the fast-path (e.g. using NEON) and slow-path fallback are being compiled into the binary, but only one path can ever be used. It would be nice if both paths can be used via run-time detection instead.
2. Using the above, one cannot have a "universal binary" that is especially problematic on Android.

In one of my projects, I have resorted to parsing /proc/cpuinfo for run-time detection of NEON, which only works on Linux. I'd love to instead use the standard library.

[smasher164]

x/sys/cpu has still not been totally updated to match internal/cpu, which has some of the feature flags you are looking for.
Note that so far, we have only exposed flags that are needed for the runtime and standard library. That is, CL 114826 exposes HasIDIVA for use in the runtime, and CL 126315 exposes HasVFPv4 for use in the math package. I could see the argument for exposing all of the HWCAP flags listed in linux, for use in external code.

I will defer to @martisch's judgement on that.

[martisch]

Note that changing internal/cpu seems a separate topic (it also already supports arm) and there are no plans to export those variables outside the runtime/standard library.

If x/sys/cpu support for linux/arm is enough (not *BSD or windows/arm) then it should be as easy as implementing the hwCap/hwCap2 bit checks in https://github.com/golang/sys/blob/master/cpu/cpu_arm.go since hwCap/hwCap2 should already be populated on Linux by: https://github.com/golang/sys/blob/master/cpu/cpu_linux.go. I think its fine to expose all linux supported but general arm applicable flags in x/sys/cpu.

Note that the above would not support android as /proc/self/auxv might not be accessible and it wont change the runtime and standard libs detection of arm features (since that uses internal/cpu and not x/sys/cpu) and the effects of setting goarm are not effected as they override any cpu package .e.g.:

go/src/runtime/os_freebsd_arm.go

Line 14 in 3b6216e

func checkgoarm() {

go/src/math/sqrt_arm.s

Line 9 in 0df9fa2

MOVB runtime·goarm(SB), R11

[jpap]

Thanks for the detailed response, Martin.

Note that changing internal/cpu seems a separate topic (it also already supports arm) and there are no plans to export those variables outside the runtime/standard library.

That's right; I'm happy to open a new issue to discuss that separately if you wish. I would propose that the internal implementation be separate (a "copy") of the x/sys/cpu version.

Since you mention it, I was reading the CL for #28148 (see this source file), and noticed how the use of detection of NEON could be improved by checking the hardware-caps instead of using the static runtime.goarm value. I did notice other similar uses throughout the Go internal/standard library, as you have linked to. While it might be some work to review all of them for a possible switch to runtime hardware-cap checks (esp where we can use a NEON code-path), it might be beneficial to do so over time?

(As said above, I am doing run-time detection at present by parsing /proc/cpuinfo which could also be improved.)

If x/sys/cpu support for linux/arm is enough (not *BSD or windows/arm) then it should be as easy as implementing the hwCap/hwCap2 bit checks in https://github.com/golang/sys/blob/master/cpu/cpu_arm.go since hwCap/hwCap2 should already be populated on Linux by: https://github.com/golang/sys/blob/master/cpu/cpu_linux.go. I think its fine to expose all linux supported but general arm applicable flags in x/sys/cpu.

I am mostly interested in Android, so support for Linux would be a win. (Apple has moved exclusively to aarch64 for some time now.)

It looks like FreeBSD has added support back in 2017 (see D12290 and D12291), so it might be possible to do something similar there also.

As far as I can tell, Go bring-up for windows/arm has stalled (c.f. #26148), so that would need to wait regardless?

Note that the above would not support android as /proc/self/auxv might not be accessible and it wont change the runtime and standard libs detection of arm features (since that uses internal/cpu and not x/sys/cpu) and the effects of setting goarm are not effected as they override any cpu package .e.g.:

BoringSSL provides a good reference implementation: they weak-link getauxval which is available on Android from API level 20, and fall back to /proc/self/auxv and /proc/cpuinfo in that order. So it appears we might be able to improve on the implementation you have linked to.

[martisch]

That's right; I'm happy to open a new issue to discuss that separately if you wish. I would propose that the internal implementation be separate (a "copy") of the x/sys/cpu version.

The scope of the feature request is currently unclear to me when reading the title and first post content. From the comments I would infer it is to add arm support for android. It also does not seem restricted to x/sys/cpu but part of the proposal is to change the runtimes use of goarm with dynamic detection. As internal/cpu IRC already has support for arm the runtime could already use that in some places where goarm is used (no need to copy from x/sys/cpu). However this would be a behaviour change and some platforms would still need to use goarm (to populate the cpu variables). I think that is better discussed decoupled from any x/sys/cpu support.

The Go compiler uses the goarm setting to make decisions about the emitted instructions and thereby does not produce "universal binaries". Changing this would be a larger change than x/sys/cpu support and likely have larger performance and binary size implications when switched to runtime detection:

go/src/cmd/compile/internal/arm/ssa.go

Line 644 in f125b32

if objabi.GOARM >= 6 {

go/src/cmd/compile/internal/ssa/regalloc.go

Line 599 in 8a317eb

if s.f.Config.arch == "arm" && objabi.GOARM == 5 {

Since you mention it, I was reading the CL for #28148 (see this source file), and noticed how the use of detection of NEON could be improved by checking the hardware-caps instead of using the static runtime.goarm value.

Seems like a good candidate for using x/sys/cpu but it seems it would need a fallback in x/sys/cpu to goarm for platforms not supporting CPU feature detection via AUXV or syscalls/proc.

It looks like FreeBSD has added support back in 2017 (see D12290 and D12291), so it might be possible to do something similar there also.

The Go runtime sets hwcap on arm in internal/cpu on FreeBSD by reading the auxv thats provided after argv. I dont think x/sys/cpu has direct access to that. So I think we can not simply copy that approach to x/sys/cpu.

go/src/runtime/os_freebsd.go

Line 385 in a62b572

sysauxv(auxv[:])

As far as I can tell, Go bring-up for windows/arm has stalled (c.f. #26148), so that would need to wait regardless?

No having only partial support in x/sys/cpu is fine. However to not regress in performance for other platforms we would seem to need a fallback to setting the cpu features based on goarm on those not supporting runtime detection.

BoringSSL provides a good reference implementation: they weak-link getauxval which is available on Android from API level 20, and fall back to /proc/self/auxv and /proc/cpuinfo in that order. So it appears we might be able to improve on the implementation you have linked to.

Thank you. Thats a nice reference. If we can make the same work under go then that looks like an option to gain feature detection support on android in x/sys/cpu.

[jpap]

The scope of the feature request is currently unclear to me when reading the title and first post content. From the comments I would infer it is to add arm support for android. It also does not seem restricted to x/sys/cpu but part of the proposal is to change the runtimes use of goarm with dynamic detection. As internal/cpu IRC already has support for arm the runtime could already use that in some places where goarm is used (no need to copy from x/sys/cpu). However this would be a behaviour change and some platforms would still need to use goarm (to populate the cpu variables). I think that is better discussed decoupled from any x/sys/cpu support.

Let me be more clear...

This issue is about:

1. Improving CPU hardware-cap detection in x/sys/cpu for third parties.
2. Exposing all available arm (32-bit) properties like whether Advanced SIMD (NEON) is supported on the device.
3. Support for Linux seems straightforward, by using BoringSSL as a reference. That implementation goes beyond the current implementation that exists in internal/cpu.
4. Support for FreeBSD may be possible by querying the elf32_freebsd_sysvec symbol in libc; see posted links above to the FreeBSD site for further details.

This issue is not about,

1. Changes to internal/cpu and how CPU hardware-cap detection is done internally in the Go standard library.
2. How the Go compiler emits opcodes, in general, for the different arm architecture versions.

however a separate issue can be created that proposes:

1. Improvements to the CPU hardware-cap detection in internal/cpu; for example, to determine whether Advanced SIMD (NEON) is supported on the device on Linux. The BoringSSL implementation for Linux could be used as a reference here also.
2. I previously assumed there were more NEON/VFP accelerated implementations in the Go standard library, and my suggestion was to review the path taken to them if they relied on runtime.goarm. After a quick look, it appears there is no support for NEON-accelerated algorithms on arm; only a reference to "will want to revision NEON, when support is better", which is actually on arm64 where NEON is always supported. (That reference is to Plan 9 assembly support, I am guessing.)
3. This separate issue could go toward promoting development of SIMD-accelerated algorithms in the Go standard library. For example, x/crypto/sha3: add SHA3 assembly implementation for ARMv7 #28148 could determine at run-time whether the NEON accelerated path is taken, rather than relying on GOARM which is set at link time. See also a comment in cmd/asm: add neon, vector instructions for arm #7300 for a similar suggestion.
4. I am not proposing that the Go compiler emits a "fat" or "universal" binary equivalent where at runtime, the ARM architecture version is determined, and optimized instructions used for math operations, and other granular instructions. I am however proposing that run-time detection of CPU hardware-caps like NEON support could allow some algorithm implementations (i.e. larger, dedicated functions; e.g. as used in crypto) to take a fast-path.

The Go compiler uses the goarm setting to make decisions about the emitted instructions and thereby does not produce "universal binaries". Changing this would be a larger change than x/sys/cpu support and likely have larger performance and binary size implications when switched to runtime detection:

I understand this issue, but I would never propose it. My reference to a "universal binary" was with reference to one of my projects where I detect NEON, and take a fast-path accordingly. A true universal binary -- that is, a program supporting multiple architectures -- is often achieved though shipping separate texts for each architecture in a sandwiched "fat binary" as is possible on macOS, and only exists as a proof-of-concept on Linux. I am not proposing that here.

Seems like a good candidate for using x/sys/cpu but it seems it would need a fallback in x/sys/cpu to goarm for platforms not supporting CPU feature detection via AUXV or syscalls/proc.

Yes -- and sure beats developers individually using go:linkname to punch a hole through to runtime.goarm.

The Go runtime sets hwcap on arm in internal/cpu on FreeBSD by reading the auxv thats provided after argv. I dont think x/sys/cpu has direct access to that. So I think we can not simply copy that approach to x/sys/cpu.

I have not tried it, but it appears a symbol exists on libc that gets you access to the information without having to go via auxv.

[gopherbot]

Change https://golang.org/cl/190525 mentions this issue: cpu: add support for ARM (32-bit) feature detection.

[jpap]

@martisch, I've submitted a CL for this issue as described by my past post, and have tested it on linux/arm and freebsd/arm. Are you able to review it?

[martisch]

@martisch, I've submitted a CL for this issue as described by my past post, and have tested it on linux/arm and freebsd/arm. Are you able to review it?

Sure. Thanks for working on it. I added a first round of high level comments.
I been meaning to reply more in depth about how we can approach this but unfortunately did not find the time earlier.

There is already support for linux auxv reading in cpu_linux. We should first leverage that existing detection to add the hwcap bits to variable mapping and then I think basic linux/arm support should already work. Continuing from there we can add additional support for android to work around /proc/self/auxv not necessary being accessible to fill hwcap in x/sys/cpu. Afterwards we can extend the support for arm on other platforms and wheel in help from testers with hardware on those.

What makes arm special in x/sys/cpu vs other archs is that absent hwcap detection we should fall back to the minimal set of hwcap bits set that is mandated by the goarm variable.

[rsc]

As far as the API is concerned, which is what the proposal process would care about, it seems that the proposal is to add a cpu.ARM that is very similar to cpu.ARM64 with appropriate modifications for the actual CPU features that might be present.

/cc @tklauser @martisch @ianlancetaylor for feedback

[martisch]

For the API I think we should stay consistent with the other architectures and internal/cpu which means to expose a cpu.ARM struct with fields named HasNAME where NAME is the corresponding HWCAP name of the feature. If needed the client of x/sys/cpu can create combinations such as HasIDIVA | HasIDIVT outside of x/sys/cpu but we do not need to create these as fields inside the cpu.ARM struct.

In general (for adding other architectures) I think if the existing naming schema is followed we do not need to have an separate proposal for each API addition for additional CPU feature structs/variables in x/sys/cpu.

[jpap]

@martisch, I have updated the original post to include a concrete list of fields.

If needed the client of x/sys/cpu can create combinations such as HasIDIVA | HasIDIVT outside of x/sys/cpu but we do not need to create these as fields inside the cpu.ARM struct.

I would argue that having "derived" fields, such as the proposed HasIDIV makes the API easier to consume by the user. Anyone else keen to chime in?