cmd/compile: "exec format error" when running newly-built binary on linux-loong64-3a5000

[bcmills]

#!watchflakes
post <- builder ~ `^linux-loong64` && `exec format error`

##### ../test
# go run run.go -- linkname.go
fork/exec /tmp/workdir-host-linux-loong64-3a5000/tmp/13900000/a.exe: exec format error

FAIL	linkname.go	0.262s

greplogs -l -e '(?ms)\Alinux-loong64.*exec format error'
2022-05-26T20:17:08-ec92580/linux-loong64-3a5000
2022-05-26T18:56:07-e6d8b05/linux-loong64-3a5000
2022-05-20T23:05:38-bc2c85a-c3470ca/linux-loong64-3a5000
2022-05-18T15:15:29-5e4e11f-bc2e961/linux-loong64-3a5000
2022-05-02T18:38:26-af99c20/linux-loong64-3a5000

(attn @golang/loong64)

[gopherbot]

Change https://go.dev/cl/408938 mentions this issue: dashboard: add known issues for linux/loong64

[xen0n]

Can't reproduce locally as well.

@golang/loong64, would you try reproducing in a fresh environment (you could use a Gentoo chroot) and latest kernel?

[limeidan]

This is an occasional problem, sometimes running all.bash can reproduce it locally, but the cause has not been found out for the time being, need futher investigation.

[xen0n]

This is an occasional problem, sometimes running all.bash can reproduce it locally, but the cause has not been found out for the time being, need futher investigation.

Can you try to somehow save a copy of the faulty executable next time this happens? Might be helpful for debugging.

[limeidan]

Can you try to somehow save a copy of the faulty executable next time this happens? Might be helpful for debugging.

FYI. And we found that the file was malformed lost 16KB of data at its beginning.

bad-bin-compile.tar.gz

[xen0n]

Update: problem may be pinpointed, it's likely due to some interesting race involving fallocate and/or mmap. The Loongson team should be working on this shortly, and I'll be helping.

[abner-chenc]

On some linux/loong64 machines, an "exec format error" error occasionally occurs when executing the make.bash or all.bash. Check these executable files with errors, it is found that the data of 16K(only one pagesize) at the beginning of the file is missing. The result of hexdump is as follows:

   1  0000000 0000 0000 0000 0000 0000 0000 0000 0000
   2  *
   3  0004000 0020 4c00 3d24 1c01 d084 02fc 2064 29c0
   4  0004010 9004 02c0 4064 29c0 4800 576a 0000 0014
   5  0004020 62d3 28c0 8e73 0012 1260 5c00 0025 0015

This problem is caused by the occasional loss of 16K data at the beginning of the file after the link process calls the syscall.Fallocate function for the second time. However, not all machines can reproduce this error (the file system formats I tested include xfs, ext4, tmfs ).

[gopherbot]

Change https://go.dev/cl/445835 mentions this issue: cmd/link: preallocate new space at the end of the file on Linux

[bcmills]

No recent failures reported by watchflakes. @golang/loong64 — was this fixed on the builder, or by some mitigation in the compiler or linker, or is there more to do here?

[gopherbot]

Change https://go.dev/cl/465156 mentions this issue: dashboard: unmark known-issues with low failure rates

[xen0n]

No recent failures reported by watchflakes. @golang/loong64 — was this fixed on the builder, or by some mitigation in the compiler or linker, or is there more to do here?

Hi, I've revisited this and it indeed seems to be fixed by kernel people in v6.1 (it was a folio regression across the board from 5.17 to 6.0), discovered by our investigation, so IMO this issue could be closed as completed. Thanks for the triage.