Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: arm FreeBSD should work around missing ELF auxiliary vectors #47006

Open
clausecker opened this issue Jul 1, 2021 · 9 comments
Open

runtime: arm FreeBSD should work around missing ELF auxiliary vectors #47006

clausecker opened this issue Jul 1, 2021 · 9 comments

Comments

@clausecker
Copy link

@clausecker clausecker commented Jul 1, 2021

What version of Go are you using (go version)?

Go 1.16.5 as provided by the FreeBSD ports tree in lang/go.

Does this issue reproduce with the latest release?

Not tested.

What operating system and processor architecture are you using (go env)?

arm with GOARM=7 on arm64 FreeBSD 13.0-RELEASE.

What did you do?

Attempt to build and run any armv7 Go program in an armv7 jail on arm64 FreeBSD 13.0-RELEASE.

What did you expect to see?

A working Go program.

What did you see instead?

runtime: this CPU has no floating point hardware, so it cannot run
this GOARM=7 binary. Recompile using GOARM=5.

Additional notes

Due to a bug (PR #256897, PR #256903) in FreeBSD 13.0-RELEASE, the AT_HWCAP and AT_HWCAP2 ELF auxiliary vectors are not provided to arm programs running under arm64 FreeBSD 13.0-RELEASE. Unfortunately, the Go runtime uses these vectors to check for floating point support and erroneously believe that no such support is available if the auxiliary vectors are missing.

To make running arm Go programs on arm64 FreeBSD 13.0-RELEASE possible, I suggest to change the floating point support detection code in src/runtime/os_freebsd_arm.go to be changed such that it checks for the presence of AT_HWCAP and AT_HWCAP2 and only denies execution if both vectors are present and indicate lack of support. If the vectors are not present, we are on a system that indicates neither presence nor absence of floating point support via the ELF auxilary vectors and proceed with execution. I believe that only affects arm64 FreeBSD 13.0-RELEASE which should always have GOARM=7 compatible floating point support as that is architectural for ARMv8-A.

@martisch
Copy link
Contributor

@martisch martisch commented Jul 3, 2021

questions:

  1. To reliably distinguish between not supported features and auxiliary vector not initialised: If the arm auxiliary vector is populated does it have at least one bit set on any arm machine (arm and arm64 hosts) that FreeBSD supports?
  2. I dont think Go runtime knows if its running a arm binary on arm64 host or arm binary on arm host. So it can not currently use that information to assume all of ARMv8-A is supported. If Go should detect running on an arm64 host that how would it do that on Freebsd with low overhead and early in the runtime (therefore ideally from the aux vector itself)?
  3. Is the request to assume GOARM=x is supported when auxiliary vector is known to be not initialised and binary is compiled with GOARM=x and/or that all of GOARM=7 is supported when host is detectable as being ARMv8-a?
  4. Does it suffice to only skip the runtime checks or is it needed to populate the the internal/cpu and x/sys/cpu feature bits as if assumed the features for GOARM are supported. The bits are AFAIK only needed for performance.
  5. Is it correct to assume FreeBSD will fix the bug to correctly fill the auxiliary vectors in the future? (approximate timeline?)

We may look into a workaround (if easy) for go1.17/go1.18 but im not sure since this is a bug in FreeBSD and it seems has a workaround (compile with GOARM=5) this qualifies for a backport. I would assume the number of active uses of running go
arm binary on a arm64 host are low.

@martisch
Copy link
Contributor

@martisch martisch commented Jul 3, 2021

Looking at the PR it seems the HWCap auxilary vectors are not provided at all (not just 0). Therefore the easiest workaround seems to record a bool in Go runtime for FreeBSD and if its not true skip the runtime checks for goarm. Likely with a hint that the expected aux values were not supplied.

@clausecker
Copy link
Author

@clausecker clausecker commented Jul 3, 2021

Hi @martinsch, let me try to address your questions:

  1. On the defective version of FreeBSD, this means the auxiliary vector is not present, not just not populated.
  2. One option would be to check the kern.supported_archs sysctl for presence of the string aarch64 indicating support for running aarch64 binaries (implying an ARMv8-A CPU), but this might be complex to implement. As for another option, the hw.machine_arch sysctl could be used. This sysctl gives armv6 on ARMv6 platforms and armv7 on ARMv7 platforms (older version of FreeBSD may have also returned armv5). This sysctl has a fixed number (see HW_MACHINE_ARCH in sys/sysctl.h) and can thus be retrieved easily.
  3. The request is to assume the former with the intent that the runtime is not able to detect if the binary can run successfully, so it'll give it a best effort. The only system where this is known to happen is based on ARMv8 on which this works out, so it seems to be a reasonable compromise between ease of implementation and usefulness. Perhaps a floating point instruction could be placed into the runtime initialisation code path in this case so the binary at least crashes as early as possible if it turns out the CPU does not support the required instructions.
  4. I don't know.
  5. I am certain this will be fixed for 13.1-RELEASE as it seems to be a rather important issue. It might even be fixed in an upcoming patch release to 13.0-RELEASE but I can't say.

The main purpose of running armv7 Go on arm64 is to build the FreeBSD ports tree (I believe this is why this support was added in the first place). Ever since the advent of fast arm64 machines, this has been desirable over building armv7 ports on QEMU and AFAIK the armv7 ports builders run by the FreeBSD project do it this way.

So this issue means that no port based on Go has armv7 binary packages available at the moment. It also means that Go developers on FreeBSD cannot test their software on armv7 on a machine with arm64 kernel and will have to buy a separate ARM machine for armv7 testing.

@martisch
Copy link
Contributor

@martisch martisch commented Jul 3, 2021

Thanks for the context. I think we can just reintroduce the code removed in: 3b6216e with a new comment explaining FreeBSD 13 Release doesnt set HWCap for arm on arm64 hosts.

I will send a CL early next week and let the go release team decide if they allow a backport or if FreeBSD team should patch it in until its resolved in FreeBSD.

@clausecker
Copy link
Author

@clausecker clausecker commented Jul 15, 2021

@martisch Any progress with this one?

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 20, 2021

Change https://golang.org/cl/335791 mentions this issue: runtime: assume arm VFP support when AT_HWCAP is missing on FreeBSD

@martisch
Copy link
Contributor

@martisch martisch commented Jul 20, 2021

@dmitshur @golang/release would it be possible to add the reintroduced workaround for missing AT_HWCAP on FreeBSD arm into go 1.17 release (or if preferred a patch release)?

@clausecker please test if https://golang.org/cl/335791 works in an armv7 jail on arm64.

@clausecker
Copy link
Author

@clausecker clausecker commented Aug 22, 2021

Any progress on the CL?

Meanwhile, FreeBSD has fixed the issue kernel side and the fix is slated to appear in FreeBSD 13.1-RELEASE. However, that release is not going to be released until next year I believe.

@martisch
Copy link
Contributor

@martisch martisch commented Aug 23, 2021

The CL would need to be accepted as a backport. This I can not ratify alone. Without a backport this may land in go1.18 if accepted, but it seems this would by the time be solved by FreeBSD 13.1-RELEASE itself. So it seems the only question is if this warrants a back port or not to work around omissions of setting HWCAP in the FreeBSD 13 Release.

/cc @aclements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants