Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: SIGILL in amd64 optimized memmove during VMOVDQU on AMD Epyc platform #49063

Closed
hansinator opened this issue Oct 19, 2021 · 5 comments
Closed

Comments

@hansinator
Copy link

@hansinator hansinator commented Oct 19, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.9 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/hansinator/.cache/go-build"
GOENV="/home/hansinator/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/hansinator/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/hansinator/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.9"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/hansinator/work/prj/immune-git/guard/agent/v2/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2081677666=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am using syscall.Memmap to map 0xFF000000 (iomem where the BIOS is) into userspace and copy it, see the following snippet:

		fd, err := os.OpenFile(common.DefaultDevMemPath, os.O_RDWR, 0)
		if err != nil {
			return nil, err
		}
		defer fd.Close()
		mem, err := syscall.Mmap(int(fd.Fd()), int64(biosRegionMmap), maxMappedMem, syscall.PROT_READ, syscall.MAP_SHARED)
		if err != nil {
			return nil, err
		}
		defer syscall.Munmap(mem)
		r = bytes.NewReader(mem)

		return io.ReadAll(r)

What did you expect to see?

io.ReadAll returning an array.

What did you see instead?

SIGILL: illegal instruction
PC=0x46c6aa m=2 sigcode=2
instruction bytes: 0xc5 0xfe 0x6f 0x26 0x4c 0x1 0xde 0x48 0x29 0xc3 0xc5 0xfe 0x6f 0x6 0xc5 0xfe

goroutine 1 [running]:
runtime.memmove(0xc000110000, 0x7fada2715000, 0x200)
	/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/memmove_amd64.s:354 +0x40a fp=0xc00014a1a0 sp=0xc00014a198 pc=0x46c6aa
bytes.(*Reader).Read(0xc00014e180, 0xc000110000, 0x200, 0x200, 0xc00014e180, 0x0, 0xc000032400)
	/opt/hostedtoolcache/go/1.16.9/x64/src/bytes/reader.go:45 +0xb1 fp=0xc00014a1d0 sp=0xc00014a1a0 pc=0x4a9971
io.ReadAll(0x8ce640, 0xc00014e180, 0xff000000, 0x1000000, 0x1, 0x1, 0x7fada2715000)
	/opt/hostedtoolcache/go/1.16.9/x64/src/io/io.go:633 +0xdf fp=0xc00014a250 sp=0xc00014a1d0 pc=0x49d4df
github.com/immune-gmbh/agent/v2/pkg/firmware/biosflash.readBiosFlashMMap(0x0, 0x0, 0x0, 0x8cebe0, 0xc00014e150)
	/home/runner/work/guard/guard/agent/v2/pkg/firmware/biosflash/biosflash_linux.go:39 +0x14e fp=0xc00014a328 sp=0xc00014a250 pc=0x77096e
github.com/immune-gmbh/agent/v2/pkg/firmware/biosflash.ReportBiosFlash(0xc00014a5f8, 0xc000000006, 0xc00014a470)
	/home/runner/work/guard/guard/agent/v2/pkg/firmware/biosflash/biosflash.go:13 +0x94 fp=0xc00014a3f8 sp=0xc00014a328 pc=0x7704d4
github.com/immune-gmbh/agent/v2/pkg/firmware.GatherFirmwareData(0x0, 0x0, 0xc00013e2b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/runner/work/guard/guard/agent/v2/pkg/firmware/report.go:44 +0xf2 fp=0xc00014a6d8 sp=0xc00014a3f8 pc=0x790cd2
github.com/immune-gmbh/agent/v2/pkg/attestation.Attest(0x8d5a10, 0xc000096000, 0xc000080198, 0x0, 0x0, 0x8d8940, 0xc00014e060, 0xc00013e200, 0x0, 0x0, ...)
	/home/runner/work/guard/guard/agent/v2/pkg/attestation/attest.go:30 +0xbb fp=0xc00014b6c8 sp=0xc00014a6d8 pc=0x7917bb
main.(*attestCmd).Run(0xae1cb8, 0xc000080190, 0x0, 0x0)
	/home/runner/work/guard/guard/agent/v2/cmd/main.go:151 +0x17f fp=0xc00014b7c0 sp=0xc00014b6c8 pc=0x7967df
runtime.call32(0xc000130c00, 0xc0000aa0f8, 0xc0000b3c40, 0x1000000020)
	/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/asm_amd64.s:551 +0x3e fp=0xc00014b7f0 sp=0xc00014b7c0 pc=0x469c5e
reflect.Value.call(0x7e1160, 0xae1cb8, 0x213, 0x84e4ac, 0x4, 0xc0000a2918, 0x1, 0x1, 0x1, 0x0, ...)
	/opt/hostedtoolcache/go/1.16.9/x64/src/reflect/value.go:476 +0x8e7 fp=0xc00014b9f8 sp=0xc00014b7f0 pc=0x4cef87
reflect.Value.Call(0x7e1160, 0xae1cb8, 0x213, 0xc0000a2918, 0x1, 0x1, 0x0, 0x1, 0xc000136398)
	/opt/hostedtoolcache/go/1.16.9/x64/src/reflect/value.go:337 +0xb9 fp=0xc00014ba78 sp=0xc00014b9f8 pc=0x4ce459
github.com/alecthomas/kong.callMethod(0x84e315, 0x3, 0x7e8ee0, 0xae1cb8, 0x199, 0x7e1160, 0xae1cb8, 0x213, 0xc000130b70, 0x203000, ...)
	/home/runner/go/pkg/mod/github.com/alecthomas/kong@v0.2.17/callbacks.go:71 +0x4ba fp=0xc00014bba8 sp=0xc00014ba78 pc=0x608f7a
github.com/alecthomas/kong.(*Context).RunNode(0xc0000fe080, 0xc0001081c0, 0xc0000c7e70, 0x1, 0x1, 0x0, 0x825460)
	/home/runner/go/pkg/mod/github.com/alecthomas/kong@v0.2.17/context.go:715 +0x545 fp=0xc00014bcd8 sp=0xc00014bba8 pc=0x610965
github.com/alecthomas/kong.(*Context).Run(0xc0000fe080, 0xc0000c7e70, 0x1, 0x1, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/alecthomas/kong@v0.2.17/context.go:732 +0x99 fp=0xc00014bd38 sp=0xc00014bcd8 pc=0x610b59
main.run(0xc000082058)
	/home/runner/work/guard/guard/agent/v2/cmd/main.go:362 +0x5f0 fp=0xc00014bf70 sp=0xc00014bd38 pc=0x798470
main.main()
	/home/runner/work/guard/guard/agent/v2/cmd/main.go:285 +0x25 fp=0xc00014bf88 sp=0xc00014bf70 pc=0x797e65
runtime.main()
	/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/proc.go:225 +0x256 fp=0xc00014bfe0 sp=0xc00014bf88 pc=0x4388d6
runtime.goexit()
	/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00014bfe8 sp=0xc00014bfe0 pc=0x46b4e1

rax    0x80
rbx    0x1e0
rcx    0x7fada2715200
rdx    0xc00014e180
rdi    0xc000110020
rsi    0x7fada2715000
rbp    0xc00014a1c0
rsp    0xc00014a198
r8     0x7cd680
r9     0x200
r10    0xc000110000
r11    0x20
r12    0x1
r13    0x40
r14    0x40
r15    0x0
rip    0x46c6aa
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0

Important Note:

When I deliberately defeat the optimization by copying 256-byte chunks, then the code works. It works on other AMD Zen-based systems, but not this one:

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 23
model		: 49
model name	: AMD EPYC Processor
stepping	: 0
microcode	: 0x1000065
cpu MHz		: 2495.312
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat umip rdpid arch_capabilities
bugs		: sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 4990.62
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

Code is running inside a VM.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Oct 19, 2021

VMOVDQU is an AVX instruction and the CPU flags says it has AVX. Why it doesn't work?

Do we do CPU feature detection incorrectly or the machine (VM) is lying?

cc @randall77 @martisch

Loading

@cherrymui cherrymui added this to the Backlog milestone Oct 19, 2021
@hansinator
Copy link
Author

@hansinator hansinator commented Oct 19, 2021

@cherrymui It really is an AMD Epyc with AVX, I'll try to get access to the VM host though
edit: filed a support ticket with the hoster, this may take a while

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Oct 19, 2021

Can you reproduce using regular memory instead of your mmap'd memory?

package main

//go:noinline
func mycopy(x, y []byte) {
	copy(x, y)
}
func main() {
	x := make([]byte, 512)
	y := make([]byte, 512)
	mycopy(x, y)
}

I don't understand how weirdly mapped memory would cause a SIGILL, but worth trying.

Loading

@hansinator
Copy link
Author

@hansinator hansinator commented Oct 19, 2021

@randall77 this does work.
The machine in question has

CONFIG_STRICT_DEVMEM=y
CONFIG_IO_STRICT_DEVMEM=y

and the mapped memory is from /dev/mem, however, as I said, copying small chunks works and produces the expected data.

edit: doing a dd on the memory range does not work, it quits with bad address but does so gracefully.

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Oct 19, 2021

My guess: the BIOS memory controller barfs on a 32-byte wide load.
In any case, I don't think this is a bug in Go, so I'm going to close. Please reopen if you find evidence otherwise.

Loading

@randall77 randall77 closed this Oct 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants