Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: illegal instruction on ARMv5 #18694

Closed
RobHumphrey opened this issue Jan 17, 2017 · 21 comments

Comments

Projects
None yet
6 participants
@RobHumphrey
Copy link

commented Jan 17, 2017

I've been trying to get Syncthing working on a WD My Book World Edition (White Light) NAS box, but encountered illegal instructions. Whilst investigating I found issue #15869, which seemed very similar, so hopefully I can continue to help resolve this issue.

What version of Go are you using (go version)?

go version go1.7.4 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/local/bin/go"
GOTOOLDIR="/usr/local/bin/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build191961496=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

Set up a simple infinite loop like:

package main

func main() {
    for true {
    }
}

Compile with:

env GOARCH=arm GOARM=5 GOOS=linux go build loopTest.go

Transfer the output across to the NAS box and run it.

If I resize the terminal window then it crashes with Illegal instruction. Additionally, as reported in #15869, if I kill it with -SIGPIPE then it also crashes with Illegal instruction.

cpuinfo of target machine:

Processor	: ARM926EJ-S rev 5 (v5l)
BogoMIPS	: 183.09
Features	: swp half thumb fastmult edsp java 
CPU implementer	: 0x41
CPU architecture: 5TEJ
CPU variant	: 0x0
CPU part	: 0x926
CPU revision	: 5
Cache type	: write-back
Cache clean	: cp15 c7 ops
Cache lockdown	: format C
Cache format	: Harvard
I size		: 32768
I assoc		: 4
I line length	: 32
I sets		: 256
D size		: 32768
D assoc		: 4
D line length	: 32
D sets		: 256

Hardware	: Oxsemi NAS
Revision	: 0000
Serial		: 000007a0362c2ec0

Under gdb (I resized the window after starting it running):

(gdb) file loopTest
Reading symbols from /shares/Download/loopTest...done.
(gdb) r
Starting program: /shares/Download/loopTest 

Program received signal SIGILL, Illegal instruction.
0xffff0514 in ?? ()
(gdb) bt
#0  0xffff0514 in ?? ()
#1  0x0004ad2c in runtime.sigtrampgo (sig=Unhandled dwarf expression opcode 0x9c
)
    at /usr/local/bin/go/src/runtime/signal_sigtramp.go:57
#2  0x0000001c in ?? ()
Backtrace stopped: frame did not save the PC
@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 17, 2017

No further development is happening on the Go 1.7 branch, except for security fixes.

Can you try Go 1.8?

@minux

This comment has been minimized.

Copy link
Member

commented Jan 17, 2017

@RobHumphrey

This comment has been minimized.

Copy link
Author

commented Jan 17, 2017

Updating to master doesn't seem to have made any difference.

go version devel +81a61a9 Tue Jan 17 22:37:19 2017 +0000 linux/amd64

Kernel version is 2.6.24.4. What's the minimum required for ARM architecture, as this is newer than the version given on the Minimum Requirements page?

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Jan 18, 2017

CL https://golang.org/cl/35352 mentions this issue.

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@RobHumphrey

This comment has been minimized.

Copy link
Author

commented Jan 18, 2017

On the command line I don't have a problem stopping the executable with Ctrl+C either.

I have no experience using gdb, but hope some of the following is useful.

Adding a hardware breakpoint with hbreak *0xffff0514 fails with No hardware breakpoint support in the target., and adding a software breakpoint at this address gives the warning

Cannot insert breakpoint 1.
Error accessing memory address 0xffff0514: Input/output error.

Adding a breakpoint at main.main works. Neither resizing the window nor kill -SIGPIPE have any impact when I step again - it just remains in the infinite loop.

Using the code built from master gives

Program received signal SIGILL, Illegal instruction.
0xffff0514 in ?? ()
(gdb) bt
#0  0xffff0514 in ?? ()
#1  0x00043780 in runtime.sigtrampgo (sig=Unhandled dwarf expression opcode 0x9c
)
    at /home/rob/src/go/src/runtime/signal_unix.go:258
#2  0x0000001c in ?? ()
Backtrace stopped: frame did not save the PC

Adding a breakpoint at 0x43780 gives

Breakpoint 1, runtime.sigtrampgo (sig=Unhandled dwarf expression opcode 0x9c
)
    at /home/rob/src/go/src/runtime/signal_unix.go:259
259	in /home/rob/src/go/src/runtime/signal_unix.go
(gdb) bt
#0  runtime.sigtrampgo (sig=Unhandled dwarf expression opcode 0x9c
) at /home/rob/src/go/src/runtime/signal_unix.go:259
#1  0x0005cf50 in runtime.sigtramp ()
    at /home/rob/src/go/src/runtime/sys_linux_arm.s:360
#2  0xffff050c in ?? ()

I've been trying to find a way of seeing what's happening when the signal occurs. I can stop the code with handle SIGPIPE stop. However, it seems that once I've reached the tight loop then whenever I try to step then gdb automatically detects that we're in an infinite loop and so doesn't do anything. Does that sound likely? If I continue instead of step then the Illegal instruction occurs.

Is there a way to add something simple into the loop (eg an increment or nop) that won't get optimised away? Hopefully that would then stop gdb detecting the infinite loop, and so allow me to step and see what's happening.

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@minux

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@RobHumphrey

This comment has been minimized.

Copy link
Author

commented Jan 18, 2017

Dumping the registers as requested, still without the patch applied:

r0             0xd	13
r1             0x10309c60	271621216
r2             0x10309ce0	271621344
r3             0x0	0
r4             0x10312094	271655060
r5             0x38	56
r6             0x0	0
r7             0x758dc	481500
r8             0x3fffffff	1073741823
r9             0xffffffe1	4294967265
r10            0x103000f0	271581424
r11            0x758dc	481500
r12            0x0	0
sp             0x10309c60	0x10309c60
lr             0xffff050c	4294903052
pc             0x5cf28	0x5cf28 <runtime.sigtramp>
fps            0x0	0
cpsr           0x10	16

The LR contains the same value I saw on the back trace last night. Stepping through until it crashed, the final few instructions were:

0x0005bfc8	689	in /home/rob/src/go/src/runtime/asm_arm.s
0x5bfc8 <setg+8>:	ldrsb	r0, [r11]
690	in /home/rob/src/go/src/runtime/asm_arm.s
0x5bfcc <setg+12>:	cmp	r0, #0	; 0x0
691	in /home/rob/src/go/src/runtime/asm_arm.s
0x5bfd0 <setg+16>:	beq	0x5bfd8 <setg+24>
694	in /home/rob/src/go/src/runtime/asm_arm.s
0x5bfd8 <setg+24>:	mov	r0, r10
695	in /home/rob/src/go/src/runtime/asm_arm.s
0x5bfdc <setg+28>:	add	pc, lr, #0	; 0x0
runtime.sigtrampgo (sig=Unhandled dwarf expression opcode 0x9c) at /home/rob/src/go/src/runtime/signal_unix.go:259
259	/home/rob/src/go/src/runtime/signal_unix.go: No such file or directory.
	in /home/rob/src/go/src/runtime/signal_unix.go
0x43780 <runtime.sigtrampgo+392>:	ldrb	r0, [sp, #20]
0x00043784	259	in /home/rob/src/go/src/runtime/signal_unix.go
0x43784 <runtime.sigtrampgo+396>:	cmp	r0, #0	; 0x0
0x00043788	259	in /home/rob/src/go/src/runtime/signal_unix.go
0x43788 <runtime.sigtrampgo+400>:	beq	0x437c4 <runtime.sigtrampgo+460>
262	in /home/rob/src/go/src/runtime/signal_unix.go
0x437c4 <runtime.sigtrampgo+460>:	ldr	pc, [sp], #96
runtime.sigtramp () at /home/rob/src/go/src/runtime/sys_linux_arm.s:361
361	/home/rob/src/go/src/runtime/sys_linux_arm.s: No such file or directory.
	in /home/rob/src/go/src/runtime/sys_linux_arm.s
0x5cf50 <runtime.sigtramp+40>:	ldr	pc, [sp], #16
Program received signal SIGILL, Illegal instruction.
0xffff0514 in ?? ()

Putting a breakpoint at 0x437c4 (two instructions before the crash), looking also at the registers and the data at the SP address:

Breakpoint 1, runtime.sigtrampgo (sig=Unhandled dwarf expression opcode 0x9c
)
    at /home/rob/src/go/src/runtime/signal_unix.go:262
262	/home/rob/src/go/src/runtime/signal_unix.go: No such file or directory.
	in /home/rob/src/go/src/runtime/signal_unix.go
(gdb) info registers
r0             0x0	0
r1             0x0	0
r2             0x0	0
r3             0x1	1
r4             0x10309c54	271621204
r5             0x10309c54	271621204
r6             0x10302000	271589376
r7             0x758dc	481500
r8             0x3fffffff	1073741823
r9             0xffffffe1	4294967265
r10            0x103000f0	271581424
r11            0xae36b	713579
r12            0x0	0
sp             0x10309bf0	0x10309bf0
lr             0x43780	276352
pc             0x437c4	0x437c4 <runtime.sigtrampgo+460>
fps            0x0	0
cpsr           0x60000010	1610612752
(gdb) x 0x10309bf0
0x10309bf0:	0x0005cf50
(gdb) si
runtime.sigtramp () at /home/rob/src/go/src/runtime/sys_linux_arm.s:361
361	/home/rob/src/go/src/runtime/sys_linux_arm.s: No such file or directory.
	in /home/rob/src/go/src/runtime/sys_linux_arm.s
(gdb) info registers
r0             0x0	0
r1             0x0	0
r2             0x0	0
r3             0x1	1
r4             0x10309c54	271621204
r5             0x10309c54	271621204
r6             0x10302000	271589376
r7             0x758dc	481500
r8             0x3fffffff	1073741823
r9             0xffffffe1	4294967265
r10            0x103000f0	271581424
r11            0xae36b	713579
r12            0x0	0
sp             0x10309c50	0x10309c50
lr             0x43780	276352
pc             0x5cf50	0x5cf50 <runtime.sigtramp+40>
fps            0x0	0
cpsr           0x60000010	1610612752
(gdb) x 0x10309c50
0x10309c50:	0xffff050c
(gdb) si

Program received signal SIGILL, Illegal instruction.
0xffff0514 in ?? ()
(gdb) info registers
r0             0x0	0
r1             0x0	0
r2             0x0	0
r3             0x1	1
r4             0x10309c54	271621204
r5             0x10309c54	271621204
r6             0x10302000	271589376
r7             0xad	173
r8             0x3fffffff	1073741823
r9             0xffffffe1	4294967265
r10            0x103000f0	271581424
r11            0xae36b	713579
r12            0x0	0
sp             0x10309c60	0x10309c60
lr             0x43780	276352
pc             0xffff0514	0xffff0514
fps            0x0	0
cpsr           0x60000010	1610612752

If you need more information than this then do let me know.

@RobHumphrey

This comment has been minimized.

Copy link
Author

commented Jan 18, 2017

Realised I should have added that the patch does stop it crashing, even if it's not fixing quite the bug you thought was happening.

@aclements

This comment has been minimized.

Copy link
Member

commented Jun 8, 2017

Ping @minux.

I'm surprised by this:

info registers
lr             0xffff050c	4294903052
...
0x5cf50 <runtime.sigtramp+40>:	ldr	pc, [sp], #16
Program received signal SIGILL, Illegal instruction.
0xffff0514 in ?? ()

It looks to me like we very slightly stomped the saved LR somehow.

@RobHumphrey, thanks for all the debugging. If you can still reproduce this (sorry, I know it's been a while), could you redo your last experiment where you stopped right before the crash, but when you're on the ldr pc, [sp], #16 line, dump more of the stack frame with, e.g., x/24wx $sp. In particular, this instruction means we're restoring PC from what's saved at SP+16 (rather than SP, which is what you examined). It's mighty suspicious that the expected LR is at SP+0 at that point, though.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 14, 2017

@aclements, this isn't by chance related to your "runtime: save r11 in ARM addmoduledata" 87a51a0 fix, is it?

@aclements

This comment has been minimized.

Copy link
Member

commented Jul 14, 2017

@aclements, this isn't by chance related to your "runtime: save r11 in ARM addmoduledata" 87a51a0 fix, is it?

It's possible it's related, but it's certainly not the same bug since the code modified in 87a51a0 only runs in dynamically linked Go libraries. But maybe we failed to follow the C ABI in some signal-related code.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 10, 2017

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@DrPeterVC

This comment has been minimized.

Copy link

commented Jan 15, 2018

I too have a problem running go compiled program running on a WD My Book World Edition (White Light) NAS box, I think my issue is identical. I realise this thread was closed - but was it ever resolved?

I can provide further details if requested.

@aclements

This comment has been minimized.

Copy link
Member

commented Jan 15, 2018

@DrPeterVC, thanks for checking. Go ahead and open a new issue (with a link to this issue).

@golang golang locked and limited conversation to collaborators Jan 15, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.