Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MIPS32: switch to fpxx from fp32 or even fp64 #39289

Open
wzssyqa opened this issue May 28, 2020 · 31 comments
Open

MIPS32: switch to fpxx from fp32 or even fp64 #39289

wzssyqa opened this issue May 28, 2020 · 31 comments

Comments

@wzssyqa
Copy link

@wzssyqa wzssyqa commented May 28, 2020

https://web.archive.org/web/20180828210612/https://dmz-portal.mips.com/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking

Let's make clear of the word:

fp32: paired-float64 support:
all fpr are float32
even number can be used as float64, with the pair of the odd-number fpr behind with it.
lwc1 is used to load low/high part of float64 into two fpr.

FPXX: (require MIPS II)
odd-number fpr should not be used
even-number fpr can both work with float32 and float64.
lwc1 is used to load float32 to even-number fpr
ldc1 is used to load float64 to even-number fpr

FP64 (require MIPS32r2)
all FPR is full-function 32/64 float.
lwc1 is used to load float32 to any fpr
ldc1 is used to load float64 to any fpr

Compatibility
FP32 can interlink and co-work with FPXX
FPXX can interlink and co-work with FP64
FP32 can NOT interlink or co-work with FP64

Currently, for Linux distributions, like Debian, we have finished the migrate from FP32 to FPXX, and we are planing for FP64.
Since currently, golang for mips32le requires mips32r2, we can switch to FP64 directly,
aka we can use the same asm code between mips64 and mips32 now.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented May 28, 2020

We find this problem is that go crash on Cavium Octeon machine:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960674

@randall77
Copy link
Contributor

@randall77 randall77 commented May 28, 2020

Since currently, golang for mips32le requires mips32r2, we can switch to FP64 directly,

I think we require only mips32r1, at least according to https://github.com/golang/go/wiki/MinimumRequirements

What's the difference between "MIPS II", "MIPS32r1", and "MIPS32r2" (at least, relevant parts for floating-point support)?

It's easy enough to ban odd-numbered FP registers. I'm not sure what is involved with changing lwc* uses.

We find this problem is that go crash on Cavium Octeon machine:

What's special about that machine? Is it something hardware-related? OS? Is it running FP64-only?

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented May 28, 2020

Since currently, golang for mips32le requires mips32r2, we can switch to FP64 directly,

I think we require only mips32r1, at least according to https://github.com/golang/go/wiki/MinimumRequirements

What's the difference between "MIPS II", "MIPS32r1", and "MIPS32r2" (at least, relevant parts for floating-point support)?

It's easy enough to ban odd-numbered FP registers. I'm not sure what is involved with changing lwc* uses.

For FP64/FPXX mode, we can only use ldc1 to load float64 into a single FPR.
Since it may be really a single 64bit FPR.

We find this problem is that go crash on Cavium Octeon machine:

What's special about that machine? Is it something hardware-related? OS? Is it running FP64-only?
It is a Octeon 7130 CPU
https://db.debian.org/machines.cgi?host=eller

Yes, it is in FP64 mode. We find this problem due to that we enable CONFIG_MIPS_O32_FP64_SUPPORT

@randall77
Copy link
Contributor

@randall77 randall77 commented May 28, 2020

For FP64/FPXX mode, we can only use ldc1 to load float64 into a single FPR.
Since it may be really a single 64bit FPR.

Is that instruction available in mips32r1?

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented May 28, 2020

For FP64/FPXX mode, we can only use ldc1 to load float64 into a single FPR.
Since it may be really a single 64bit FPR.

Is that instruction available in mips32r1?

Yes. ldc1 is from MIPS II, which is quite old.
and that's why FPXX can work with MIPS II but not MIPS I.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented May 28, 2020

Another tip: MTC1, MFC1 should not be used to access the high part of a float64.

for mips32r1, it is also possible to support FP64, if we:
don't use MTHC1, MFHC1 to access the high part of a float64,
since MTHC1, MFHC1 are introduced in mips32r2
Instead, we have to use memory as a bridge.

@draganmladjenovic
Copy link

@draganmladjenovic draganmladjenovic commented May 28, 2020

@wzssyqa If I remeber corectly, GO has an issue of not being able to guarantee the 8-byte alignment for float64 on mips, at least on stack. So using ldc1 and sdc1 could trap.

This issue sounds something more like a https://go-review.googlesource.com/c/go/+/97735. Maybe the culprit is in https://github.com/golang/go/blob/master/src/runtime/preempt_mipsx.s#L78. It seems to load odd FP regs first if that means something. Maybe there are other places that do the same.

@randall77 Is there a way to run a bootstrap process w/o async preemption to check this?

@randall77
Copy link
Contributor

@randall77 randall77 commented May 28, 2020

GODEBUG=asyncpreemptoff=1 should turn off async preemption.

@andybons andybons added this to the Unplanned milestone May 28, 2020
@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented May 28, 2020

@wzssyqa If I remeber corectly, GO has an issue of not being able to guarantee the 8-byte alignment for float64 on mips, at least on stack. So using ldc1 and sdc1 could trap.

If so, we may have 2 choice:
raise ISA level to r2, so that we can use MTHC1/MFHC1
use memory to help alignment.

This issue sounds something more like a https://go-review.googlesource.com/c/go/+/97735. Maybe the culprit is in https://github.com/golang/go/blob/master/src/runtime/preempt_mipsx.s#L78. It seems to load odd FP regs first if that means something. Maybe there are other places that do the same.

@randall77 Is there a way to run a bootstrap process w/o async preemption to check this?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented May 28, 2020

I think there is some alignment issue. Currently float64 is only 4-byte aligned on 32-bit architectures. If we use 64-bit FP load/store and the address is not 8-byte aligned, it faults. This is why we use two MOVFs instead of one MOVD.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented May 28, 2020

Preemption is not a problem if we don't use odd-numbered FP registers. We can change preempt_mipsx.s easily.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

currently, go is FP32, while it claims that it is FPXX.
So, on FP64 enabled machine, the kernel will set FPU mode to FR=1 for it,
make it cannot work.

I cannot find where it is set.

@randall77
Copy link
Contributor

@randall77 randall77 commented Jun 2, 2020

How are we claiming FPXX? Is that in the elf header somewhere? If we explicitly mark as FP32, will that fix the issue?

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

How are we claiming FPXX? Is that in the elf header somewhere? If we explicitly mark as FP32, will that fix the issue?

In section of .MIPS.abiflags.
objdump -s -j .MIPS.abiflags
told me that it does have this section, while I cannot find it in our source code.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

the content of this section is:
typedef struct
{
/* Version of flags structure. /
uint16_t version;
/
The level of the ISA: 1-5, 32, 64. /
uint8_t isa_level;
/
The revision of ISA: 0 for MIPS V and below, 1-n otherwise. /
uint8_t isa_rev;
/
The size of general purpose registers. /
uint8_t gpr_size;
/
The size of co-processor 1 registers. /
uint8_t cpr1_size;
/
The size of co-processor 2 registers. /
uint8_t cpr2_size;
/
The floating-point ABI. /
uint8_t fp_abi;
/
Processor-specific extension. /
uint32_t isa_ext;
/
Mask of ASEs used. /
uint32_t ases;
/
Mask of general flags. */
uint32_t flags1;
uint32_t flags2;
} Elf_Internal_ABIFlags_v0;

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

ohh, my fault, it doesn't have this section at all.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

Strange thing, this section is added by some tool of Debian...

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

anyway, it is time for us to consider to migrate to FPXX.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

Now we get the real problem: when link with C libraires, go-link will copy the section from libc.

While in fact we should avoid to just copy .MIPS.abiflags, since
FP32 + FPXX -> FP32
FP32 + FP64 -> rejected

@draganmladjenovic
Copy link

@draganmladjenovic draganmladjenovic commented Jun 2, 2020

@wzssyqa It seems that binutils exhibits this "wrong" behavior, at least on 2.31.

$ echo "int main(void) { return 0; }" > test.c

$ gcc -EL  -mabi=32 -mfp32 test.c -c -o test.o
$ gcc -EL  -mabi=32 test.o  -o test.exe

$ readelf -a test.exe | grep FP
  Tag_GNU_MIPS_ABI_FP: Hard float (double precision)
FP ABI: Hard float (double precision)

$ objcopy --remove-section .gnu.attributes test.o
$ objcopy --remove-section .MIPS.abiflags test.o
$ gcc -EL  -mabi=32 test.o  -o test.exe

$ readelf -a test.exe | grep FP

  Tag_GNU_MIPS_ABI_FP: Hard float (32-bit CPU, Any FPU)
FP ABI: Hard float (32-bit CPU, Any FPU)

Edit: If I preserve the .gnu.attributes the abiflags gets correctly inferred. I guess we have to start emitting gnu.attributes here with Tag_GNU_MIPS_ABI_FP set to Val_GNU_MIPS_ABI_FP_DOUBLE.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 2, 2020

Yes. I think that we should do so, aka add these attributions.
.gnu.attributes and .MIPS.abiflags etc

@randall77
Copy link
Contributor

@randall77 randall77 commented Jun 2, 2020

Ok, then on the affected machine, can someone try adding a FP32 marker to a Go binary which would otherwise fail, to see if it then passes?

@draganmladjenovic
Copy link

@draganmladjenovic draganmladjenovic commented Jun 2, 2020

@wzssyqa @randall77 I beileve just placing .gnu_attribute 4, 1 under #ifndef __mips_soft_float in src/runtime/cgo/gcc_mipsx.S should do the trick, but I cannot check it for some time. I assume that this file gets linked on mips every time we do externel linking.

@randall77
Copy link
Contributor

@randall77 randall77 commented Jun 2, 2020

That will work for cgo builds but that file won't be included (and the external linker won't be called) for pure Go builds.
We'll need internal linking fix also. Any maybe a cgo-disabled external link solution also? (Not sure why one would do that, but I think it is possible.)

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Jun 2, 2020

I think we can just emit the marker in the linker, as long as we know what the right marker should be. Softfloat is an interesting case. What happens if we emit the marker in softfloat mode (where FP registers are not used at all)? Does it hurt? (Currently the linker doesn't know/care softfloat or not. We could plumbing the information through, if necessary.)

@draganmladjenovic
Copy link

@draganmladjenovic draganmladjenovic commented Jun 2, 2020

@cherrymui @randall77 I did a quick check linking object with only hard-float .gnu.attibutes into soft-float exe and I get a warring from bfd-ld that I'm mixing soft-float and hard-float objects.
Ideally this should emited in the linker (I'm just proposing a quick fix, that can be easily backpacked), but this only matters when we do external linking. Internal-linked binaries w/o .MIPS.abiflags are treated same as FP32 by the kernel.
Does the ARM have the same problem?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Jun 2, 2020

@draganmladjenovic just to be clear, do you mean that for internally linked executables it already works fine, and we only need the marker for external linking? Thanks.

@draganmladjenovic
Copy link

@draganmladjenovic draganmladjenovic commented Jun 2, 2020

@cherrymui Yes, sorry, we only need marker for external linking, because mips doesn't support internal linking w/ cgo (edit: Actually that is irrelevant, even if mips target did support that use case, marker is needed only for external linking).

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Jun 2, 2020

@draganmladjenovic thanks for clarifying.

For ARM, #15862 seems related. But it doesn't seem as bad there.

@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 6, 2020

I tried to add a .gnu.attributes section for non-cgo, or static linked exe.
It does have a .gnu.attributes section, while the kernel cannot recognize it.

I think that there must be something I am wrong.

diff --git a/src/cmd/link/internal/ld/elf.go b/src/cmd/link/internal/ld/elf.go
index 78298be..e2414c3 100644
--- a/src/cmd/link/internal/ld/elf.go
+++ b/src/cmd/link/internal/ld/elf.go
@@ -207,6 +207,7 @@ const (
 	SHT_SYMTAB_SHNDX     = 18
 	SHT_LOOS             = 0x60000000
 	SHT_HIOS             = 0x6fffffff
+	SHT_GNU_ATTRIBUTES   = 0x6ffffff5
 	SHT_GNU_VERDEF       = 0x6ffffffd
 	SHT_GNU_VERNEED      = 0x6ffffffe
 	SHT_GNU_VERSYM       = 0x6fffffff
@@ -467,6 +468,8 @@ var (
 	shdr [NSECT]*ElfShdr
 
 	interp string
+
+	BigEndian bool
 )
 
 type Elfstring struct {
@@ -487,6 +490,9 @@ var buildinfo []byte
 func Elfinit(ctxt *Link) {
 	ctxt.IsELF = true
 
+	if ctxt.IsBigEndian () {
+		BigEndian = true
+	}
 	if ctxt.Arch.InFamily(sys.AMD64, sys.ARM64, sys.MIPS64, sys.PPC64, sys.RISCV64, sys.S390X) {
 		elfRelType = ".rela"
 	} else {
@@ -826,6 +832,37 @@ func elfwriteinterp(out *OutBuf) int {
 	return int(sh.size)
 }
 
+func elfgnuattributes(sh *ElfShdr, startva uint64, resoff uint64) int {
+	n := 16
+	sh.addr = startva + resoff - uint64(n)
+	sh.off = resoff - uint64(n)
+	println("sh.off in elfgnuattributes: ", sh.off);
+	sh.size = uint64(n)
+
+	return n
+}
+
+func elfgnuattributescontent() []byte {
+	if BigEndian {
+		return []byte{0x41, 0x00, 0x00, 0x00,
+				0x0f, 0x67, 0x6e, 0x75,
+				0x00, 0x01, 0x00, 0x00,
+				0x00, 0x07, 0x04, 0x05}
+	}
+	return []byte{0x41, 0x0f, 0x00, 0x00,
+			0x00, 0x67, 0x6e, 0x75,
+			0x00, 0x01, 0x07, 0x00,
+			0x00, 0x00, 0x04, 0x05}
+}
+func elfwritegnuattributes(out *OutBuf) int {
+	sh := elfshname(".gnu.attributes")
+	sh.type_ = SHT_GNU_ATTRIBUTES
+	out.SeekSet(int64(sh.off))
+	println("sh.off in elfwritegnuattributes: ", sh.off);
+	out.Write(elfgnuattributescontent())
+	return int(sh.size)
+}
+
 func elfnote(sh *ElfShdr, startva uint64, resoff uint64, sz int) int {
 	n := 3*4 + uint64(sz) + resoff%4
 
@@ -1490,6 +1527,20 @@ func addgonote(ctxt *Link, sectionName string, tag uint32, desc []byte) {
 	s.SetAlign(4)
 }
 
+func addgnuattributes(ctxt *Link) {
+	ldr := ctxt.loader
+	s := ldr.CreateSymForUpdate(".gnu.attributes", 0)
+	s.SetReachable(true)
+	s.SetType(sym.SELFROSECT)
+	s.AddUint32(ctxt.Arch, uint32(16))
+	s.AddBytes(elfgnuattributescontent())
+	for len(s.Data())%4 != 0 {
+		s.AddUint8(0)
+	}
+	s.SetSize(int64(len(s.Data())))
+	s.SetAlign(4)
+}
+
 func (ctxt *Link) doelf() {
 	ldr := ctxt.loader
 
@@ -1507,6 +1558,9 @@ func (ctxt *Link) doelf() {
 	shstrtab.Addstring(".noptrbss")
 	shstrtab.Addstring("__libfuzzer_extra_counters")
 	shstrtab.Addstring(".go.buildinfo")
+	if ctxt.IsMIPS() {
+		shstrtab.Addstring(".gnu.attributes")
+	}
 
 	// generate .tbss section for dynamic internal linker or external
 	// linking, so that various binutils could correctly calculate
@@ -1557,6 +1611,9 @@ func (ctxt *Link) doelf() {
 			shstrtab.Addstring(elfRelType + ".data.rel.ro")
 		}
 		shstrtab.Addstring(elfRelType + ".go.buildinfo")
+		if ctxt.IsMIPS() {
+			shstrtab.Addstring(elfRelType + ".gnu.attributes")
+		}
 
 		// add a .note.GNU-stack section to mark the stack as non-executable
 		shstrtab.Addstring(".note.GNU-stack")
@@ -1760,6 +1817,9 @@ func (ctxt *Link) doelf() {
 	if ctxt.LinkMode == LinkExternal && *flagBuildid != "" {
 		addgonote(ctxt, ".note.go.buildid", ELF_NOTE_GOBUILDID_TAG, []byte(*flagBuildid))
 	}
+	if ctxt.IsMIPS() {
+		addgnuattributes(ctxt)
+	}
 }
 
 // Do not write DT_NULL.  elfdynhash will finish it.
@@ -1886,6 +1946,11 @@ func Asmbelf(ctxt *Link, symo int64) {
 			sh.flags = SHF_ALLOC
 		}
 
+		if ctxt.IsMIPS() {
+			sh := elfshname(".gnu.attributes")
+			sh.type_ = SHT_NOTE
+		}
+
 		goto elfobj
 	}
 
@@ -2003,6 +2068,16 @@ func Asmbelf(ctxt *Link, symo int64) {
 		phsh(pnote, sh)
 	}
 
+	if ctxt.IsMIPS() {
+		sh := elfshname(".gnu.attributes")
+		resoff -= int64(elfgnuattributes(sh, uint64(startva), uint64(resoff)))
+
+		pnote := newElfPhdr()
+		pnote.type_ = PT_NOTE
+		pnote.flags = PF_R
+		phsh(pnote, sh)
+	}
+
 	// Additions to the reserved area must be above this line.
 
 	elfphload(&Segtext)
@@ -2344,6 +2419,10 @@ elfobj:
 		a += int64(elfwritenetbsdpax(ctxt.Out))
 	}
 
+	if ctxt.IsMIPS() {
+		a += int64(elfwritegnuattributes(ctxt.Out))
+	}
+
 	if a > elfreserve {
 		Errorf(nil, "ELFRESERVE too small: %d > %d with %d text sections", a, elfreserve, numtext)
 	}
@wzssyqa
Copy link
Author

@wzssyqa wzssyqa commented Jun 18, 2020

This is blocked by #39677

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.