New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: nice -20 causes thread exhaustion on ARM #17803

Open
shahinv opened this Issue Nov 4, 2016 · 24 comments

Comments

Projects
None yet
8 participants
@shahinv

shahinv commented Nov 4, 2016

What version of Go are you using (go version)?

go version go1.7.1 linux/arm

What operating system and processor architecture are you using (go env)?

Linux/ARM (RPi 2/3)

What did you do?

sudo nice -n -20 program

What did you expect to see?

Program works correctly

What did you see instead?

Number of threads increase randomly until program crashes with following error "runtime/cgo: pthread_create failed: Resource temporarily unavailable"

gdb reports large number of stalled threads
33 Thread 0x671ff460 (LWP 7863) "printer" 0x000a7aa8 in runtime.futex ()
32 Thread 0x669ff460 (LWP 7864) "printer" 0x000a7aa8 in runtime.futex ()
31 Thread 0x679ff460 (LWP 7821) "printer" 0x000a7aa8 in runtime.futex ()
30 Thread 0x681ff460 (LWP 7820) "printer" 0x000a7aa8 in runtime.futex ()
29 Thread 0x689ff460 (LWP 7811) "printer" 0x000a7aa8 in runtime.futex ()
28 Thread 0x691ff460 (LWP 7810) "printer" 0x000a7aa8 in runtime.futex ()
27 Thread 0x6a1ff460 (LWP 7806) "printer" 0x000a7aa8 in runtime.futex ()
26 Thread 0x699ff460 (LWP 7807) "printer" 0x000a7aa8 in runtime.futex ()
25 Thread 0x6a9ff460 (LWP 7804) "printer" 0x000a7aa8 in runtime.futex ()
24 Thread 0x6b1ff460 (LWP 7802) "printer" 0x000a7aa8 in runtime.futex ()
23 Thread 0x6b9ff460 (LWP 7801) "printer" 0x000a7aa8 in runtime.futex ()
22 Thread 0x6c1ff460 (LWP 7775) "printer" 0x000a7aa8 in runtime.futex ()
21 Thread 0x6c9ff460 (LWP 7774) "printer" 0x000a7aa8 in runtime.futex ()
20 Thread 0x6d1ff460 (LWP 7771) "printer" 0x000a7aa8 in runtime.futex ()
19 Thread 0x6d9ff460 (LWP 7770) "printer" 0x000a7aa8 in runtime.futex ()

Before crashes usually load where around 7~10. RPi's normal load should be less than 0.2.

I have find out time.Sleep() is the part of the issue specially inside tight loops. Longer delays cause issue appear much less frequently. It gets better or worst depend on structure of code. I could not reproduce it on amd64.

Removing "nice" make the issue goes away completely.
Unfortunately I could not prepare small program to reproduce the issue.

@shahinv shahinv changed the title from Changing process priority on Arm crashes the program to Changing process priority on ARM crashes the program Nov 4, 2016

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 4, 2016

Are you able to paste the complete text of the panic message?

On Sat, 5 Nov 2016, 09:53 shahinv notifications@github.com wrote:

What version of Go are you using (go version)?

go version go1.7.1 linux/arm
What operating system and processor architecture are you using (go env)?

Linux/ARM (RPi 2/3)
What did you do?

sudo nice -n -20 program
What did you expect to see?

Program works correctly
What did you see instead?

Number of threads increase randomly until program crashes with following
error "runtime/cgo: pthread_create failed: Resource temporarily unavailable"

gdb reports large number of stalled threads
33 Thread 0x671ff460 (LWP 7863) "printer" 0x000a7aa8 in runtime.futex ()
32 Thread 0x669ff460 (LWP 7864) "printer" 0x000a7aa8 in runtime.futex ()
31 Thread 0x679ff460 (LWP 7821) "printer" 0x000a7aa8 in runtime.futex ()
30 Thread 0x681ff460 (LWP 7820) "printer" 0x000a7aa8 in runtime.futex ()
29 Thread 0x689ff460 (LWP 7811) "printer" 0x000a7aa8 in runtime.futex ()
28 Thread 0x691ff460 (LWP 7810) "printer" 0x000a7aa8 in runtime.futex ()
27 Thread 0x6a1ff460 (LWP 7806) "printer" 0x000a7aa8 in runtime.futex ()
26 Thread 0x699ff460 (LWP 7807) "printer" 0x000a7aa8 in runtime.futex ()
25 Thread 0x6a9ff460 (LWP 7804) "printer" 0x000a7aa8 in runtime.futex ()
24 Thread 0x6b1ff460 (LWP 7802) "printer" 0x000a7aa8 in runtime.futex ()
23 Thread 0x6b9ff460 (LWP 7801) "printer" 0x000a7aa8 in runtime.futex ()
22 Thread 0x6c1ff460 (LWP 7775) "printer" 0x000a7aa8 in runtime.futex ()
21 Thread 0x6c9ff460 (LWP 7774) "printer" 0x000a7aa8 in runtime.futex ()
20 Thread 0x6d1ff460 (LWP 7771) "printer" 0x000a7aa8 in runtime.futex ()
19 Thread 0x6d9ff460 (LWP 7770) "printer" 0x000a7aa8 in runtime.futex ()

Before crashes usually load where around 7~10. RPi's normal load should be
less than 0.2.

I have find out time.Sleep() is the part of the issue specially inside
tight loops. Longer delays cause issue appear much less frequently. It gets
better or worst depend on structure of code. I could not reproduce it on
amd64.

Removing "nice" make the issue goes away completely.
Unfortunately I could not prepare small program to reproduce the issue.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#17803, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAcAwhPTEi4FPmwp96TapQNXhfe8qGIks5q67djgaJpZM4KqEOq
.

@shahinv

This comment has been minimized.

shahinv commented Nov 4, 2016

@davecheney

It takes up to a day to have it crashed. But I have large number of bug reports from people who are using the program. They look similar.

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x76c8af70 m=148

goroutine 0 [idle]:

goroutine 1 [IO wait, 5 minutes]:
net.runtime_pollWait(0x75213f00, 0x72, 0x69f360)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).wait(0x10b6a4b8, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).waitRead(0x10b6a4b8, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).accept(0x10b6a480, 0x0, 0x69e838, 0x10ba2c70)
    /usr/local/go/src/net/fd_unix.go:419 +0x21c
net.(*TCPListener).accept(0x10b2ab18, 0x2470d1ff, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:132 +0x30
net.(*TCPListener).AcceptTCP(0x10b2ab18, 0xa04b0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock.go:209 +0x9c
net/http.tcpKeepAliveListener.Accept(0x10b2ab18, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2608 +0x3c
net/http.(*Server).Serve(0x10b3c140, 0x6a0b78, 0x10b2ab18, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2273 +0x200
net/http.(*Server).ListenAndServe(0x10b3c140, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2219 +0xf4
main.webInit()
    /home/pi/printer/app/web.go:939 +0x263c
main.main()
    /home/pi/printer/app/server.go:70 +0x158

goroutine 17 [syscall, 159 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm.s:998 +0x4

goroutine 38 [runnable]:
syscall.Syscall(0x3, 0xa, 0x11158e00, 0x200, 0xef, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8
syscall.read(0xa, 0x11158e00, 0x200, 0x200, 0xf954900, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_arm.go:783 +0x78
syscall.Read(0xa, 0x11158e00, 0x200, 0x200, 0x5c718, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:161 +0x4c
os.(*File).read(0x10afc1a8, 0x11158e00, 0x200, 0x200, 0x1ffffff, 0x0, 0x0)
    /usr/local/go/src/os/file_unix.go:228 +0x54
os.(*File).Read(0x10afc1a8, 0x11158e00, 0x200, 0x200, 0x0, 0x0, 0x0)
    /usr/local/go/src/os/file.go:101 +0x7c
bytes.(*Buffer).ReadFrom(0x10b42eb8, 0x69e7c0, 0x10afc1a8, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bytes/buffer.go:176 +0x22c
io/ioutil.readAll(0x69e7c0, 0x10afc1a8, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/ioutil/ioutil.go:33 +0x11c
io/ioutil.ReadFile(0x4f9f68, 0x12, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/ioutil/ioutil.go:70 +0x16c
main.WifiStatus()
    /home/pi/printer/app/wireless.go:94 +0x150
created by main.wifiConnect
    /home/pi/printer/app/wireless.go:22 +0x7c

goroutine 6 [sleep]:
time.Sleep(0x540be400, 0x2)
    /usr/local/go/src/runtime/time.go:59 +0xec
main.(*StatusStruct).FindIP(0x6b7088)
    /home/pi/printer/app/db.go:472 +0xe0
created by main.(*StatusStruct).Load
    /home/pi/printer/app/db.go:438 +0x58

goroutine 7 [sleep]:
time.Sleep(0x2a05f200, 0x1)
    /usr/local/go/src/runtime/time.go:59 +0xec
main.faultDetection()
    /home/pi/printer/app/hardware-interface.go:303 +0x3a4
created by main.hwInit
    /home/pi/printer/app/hardware-interface.go:29 +0x21c

goroutine 11 [syscall]:
syscall.Syscall(0x3, 0x4, 0x11279efc, 0x1, 0x1b, 0x10aece40, 0x1c)
    /usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8
syscall.read(0x4, 0x11279efc, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_arm.go:783 +0x78
syscall.Read(0x4, 0x11279efc, 0x1, 0x1, 0x76bb3258, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:161 +0x4c
os.(*File).read(0x10a0c8f0, 0x11279efc, 0x1, 0x1, 0x10aa8320, 0x0, 0x0)
    /usr/local/go/src/os/file_unix.go:228 +0x54
os.(*File).Read(0x10a0c8f0, 0x11279efc, 0x1, 0x1, 0x79, 0x0, 0x0)
    /usr/local/go/src/os/file.go:101 +0x7c
github.com/tarm/serial.(*Port).Read(0x10a0c8f8, 0x11279efc, 0x1, 0x1, 0x10e2ad80, 0x0, 0x0)
    /home/pi/go/src/github.com/tarm/serial/serial_posix.go:129 +0x50
_/home/pi/printer/app/shield.(*Conf).Read(0x6b6c98, 0x11279efc, 0x1, 0x1, 0x0, 0x0, 0x0)
    /home/pi/printer/app/shield/shield.go:90 +0x98
main.(*termStruct).readLine(0x6b6c70, 0x0, 0x0, 0x0)
    /home/pi/printer/app/term.go:54 +0x100
main.(*termStruct).Reader(0x6b6c70)
    /home/pi/printer/app/term.go:39 +0xdc
created by main.main
    /home/pi/printer/app/server.go:54 +0xac

goroutine 12 [sleep]:
time.Sleep(0x540be400, 0x2)
    /usr/local/go/src/runtime/time.go:59 +0xec
main.(*monitorStruct).watch(0x6c5dc8)
    /home/pi/printer/app/slice-monitor.go:84 +0x548
created by main.main
    /home/pi/printer/app/server.go:55 +0xc8

goroutine 14 [sleep]:
time.Sleep(0x2a05f200, 0x1)
    /usr/local/go/src/runtime/time.go:59 +0xec
_/home/pi/printer/app/hw.Monitor()
    /home/pi/printer/app/hw/hw-monitor_linux.go:127 +0x7c
created by main.main
    /home/pi/printer/app/server.go:57 +0xf0

goroutine 24 [runnable]:
syscall.Syscall(0x4, 0x2, 0x10a7ab00, 0x78, 0x78, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8
syscall.write(0x2, 0x10a7ab00, 0x78, 0x160, 0x9, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_arm.go:1064 +0x78
syscall.Write(0x2, 0x10a7ab00, 0x78, 0x160, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:180 +0x4c
os.(*File).write(0x10a0c1c0, 0x10a7ab00, 0x78, 0x160, 0x0, 0x0, 0x0)
    /usr/local/go/src/os/file_unix.go:249 +0x78
os.(*File).Write(0x10a0c1c0, 0x10a7ab00, 0x78, 0x160, 0x34f4b79c, 0x0, 0x0)
    /usr/local/go/src/os/file.go:142 +0x7c
log.(*Logger).Output(0x10a4c090, 0x2, 0x10b3a840, 0x5d, 0x0, 0x0)
    /usr/local/go/src/log/log.go:166 +0x328
log.Println(0x11277ddc, 0x1, 0x1)
    /usr/local/go/src/log/log.go:295 +0x5c
_/home/pi/printer/app/mlog.(*memLog).Add(0x6b6c48, 0x4f3ed7, 0x5, 0x487fff, 0x11277e9c, 0x1, 0x1)
    /home/pi/printer/app/mlog/mlog.go:55 +0x6c4
main.memlog(0x4f3ed7, 0x5, 0x10e181ff, 0x10af66b0, 0x2, 0x2)
    /home/pi/printer/app/server.go:162 +0x140
main.showFrame(0x10c55ee0, 0x15, 0x15)
    /home/pi/printer/app/hardware-interface.go:268 +0x13c
main.printCalibration(0x1)
    /home/pi/printer/app/printing-calibration.go:55 +0xbcc
created by main.printerStart
    /home/pi/printer/app/web.go:699 +0xf8

goroutine 36 [select]:
net/http.(*Transport).getConn(0x10a800a0, 0x10ab6080, 0x0, 0x4ff127, 0x4, 0x10e5c5e0, 0x12, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:885 +0x8e0
net/http.(*Transport).RoundTrip(0x10a800a0, 0x10a0ae80, 0x10a800a0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:367 +0xa18
net/http.send(0x10a0ae80, 0x69e550, 0x10a800a0, 0x0, 0x0, 0x0, 0x0, 0x10ab6070, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:256 +0x4fc
net/http.(*Client).send(0x6b6b48, 0x10a0ae80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:146 +0x144
net/http.(*Client).doFollowingRedirects(0x6b6b48, 0x10a0ae80, 0x53f900, 0x502266, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:528 +0xa50
net/http.(*Client).Post(0x6b6b48, 0x4ff127, 0x1b, 0x502266, 0x21, 0x69e8e0, 0x10e5c580, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:579 +0xd0
net/http.(*Client).PostForm(0x6b6b48, 0x4ff127, 0x1b, 0x10e5c460, 0x48f878, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:605 +0xd4
net/http.PostForm(0x4ff127, 0x1b, 0x10e5c460, 0x110a1fc4, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:593 +0x50
main.sendDashboardData()
    /home/pi/printer/app/server.go:195 +0x884
main.updateControlPanel()
    /home/pi/printer/app/server.go:172 +0x14
created by main.afterIP
    /home/pi/printer/app/server.go:84 +0xf8

goroutine 3227 [IO wait]:
net.runtime_pollWait(0x75213618, 0x72, 0x11090000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).wait(0x10d33238, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).waitRead(0x10d33238, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x10d33200, 0x11090000, 0x1000, 0x1000, 0x0, 0x69f360, 0x10a10090)
    /usr/local/go/src/net/fd_unix.go:243 +0x1f0
net.(*conn).Read(0x10eb4848, 0x11090000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0xb8
net/http.(*connReader).Read(0x10e3ba80, 0x11090000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:586 +0x1d8
bufio.(*Reader).fill(0x111f4270)
    /usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(*Reader).ReadSlice(0x111f4270, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:330 +0x264
bufio.(*Reader).ReadLine(0x111f4270, 0x0, 0x0, 0x0, 0x94500, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:359 +0x60
net/textproto.(*Reader).readLineSlice(0x10d6ae00, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:55 +0x58
net/textproto.(*Reader).ReadLine(0x10d6ae00, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:36 +0x34
net/http.readRequest(0x111f4270, 0x53f800, 0x10dc5400, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:793 +0x80
net/http.(*conn).readRequest(0x10d33280, 0x6a0f90, 0x10e3bac0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:765 +0x3d4
net/http.(*conn).serve(0x10d33280, 0x6a0f90, 0x10e3bac0)
    /usr/local/go/src/net/http/server.go:1532 +0x910
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2293 +0x470

goroutine 3294 [chan receive]:
net.goLookupIPOrder(0x6a0fe0, 0x10a4ad80, 0x10e5c5e0, 0xf, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/dnsclient_unix.go:492 +0x37c
net.lookupIP(0x6a0fe0, 0x10a4ad80, 0x10e5c5e0, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/lookup_unix.go:76 +0xfc
net.glob..func11(0x6a0fe0, 0x10a4ad80, 0x53f7dc, 0x10e5c5e0, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/hook.go:19 +0x58
net.lookupIPContext.func1(0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/lookup.go:119 +0x78
internal/singleflight.(*Group).doCall(0x6b6920, 0x10e84090, 0x10e5c5e0, 0xf, 0x10e5c6c0)
    /usr/local/go/src/internal/singleflight/singleflight.go:93 +0x1c
created by internal/singleflight.(*Group).DoChan
    /usr/local/go/src/internal/singleflight/singleflight.go:86 +0x37c

goroutine 3318 [runnable]:
main.shutterOpen()
    /home/pi/printer/app/hardware-interface.go:217
created by main.printCalibration
    /home/pi/printer/app/printing-calibration.go:38 +0x4f4

goroutine 3296 [runnable]:
net.runtime_pollWait(0x75213ca8, 0x72, 0x11158000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).wait(0x10a4b038, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).waitRead(0x10a4b038, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x10a4b000, 0x11158000, 0x200, 0x200, 0x0, 0x69f360, 0x10a10090)
    /usr/local/go/src/net/fd_unix.go:243 +0x1f0
net.(*conn).Read(0x10afc170, 0x11158000, 0x200, 0x200, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0xb8
net.dnsRoundTripUDP(0x69fdb8, 0x10afc170, 0x10a4afc0, 0x50d629ef, 0x0, 0x0)
    /usr/local/go/src/net/dnsclient_unix.go:64 +0x18c
net.(*UDPConn).dnsRoundTrip(0x10afc170, 0x10a4afc0, 0x35064362, 0x0, 0x0)
    /usr/local/go/src/net/dnsclient_unix.go:47 +0x44
net.exchange(0x6a0fe0, 0x10a4ae80, 0x10aecc80, 0xf, 0x10ab60d0, 0x10, 0x6a001c, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/dnsclient_unix.go:164 +0x400
net.tryOneName(0x6a0fe0, 0x10a4ae80, 0x10bb4140, 0x10ab60d0, 0x10, 0x1c, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/net/dnsclient_unix.go:193 +0x418
net.goLookupIPOrder.func1(0x6a0fe0, 0x10a4ad80, 0x10bb4140, 0x10afc148, 0x10a4ae00, 0x110a001c)
    /usr/local/go/src/net/dnsclient_unix.go:487 +0x4c
created by net.goLookupIPOrder
    /usr/local/go/src/net/dnsclient_unix.go:489 +0x308

goroutine 3209 [IO wait]:
net.runtime_pollWait(0x752137f8, 0x72, 0x10c50000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).wait(0x10dee738, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).waitRead(0x10dee738, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x10dee700, 0x10c50000, 0x1000, 0x1000, 0x0, 0x69f360, 0x10a10090)
    /usr/local/go/src/net/fd_unix.go:243 +0x1f0
net.(*conn).Read(0x10ab4df8, 0x10c50000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0xb8
net/http.(*connReader).Read(0x10e37fc0, 0x10c50000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:586 +0x1d8
bufio.(*Reader).fill(0x10bb0270)
    /usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(*Reader).ReadSlice(0x10bb0270, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:330 +0x264
bufio.(*Reader).ReadLine(0x10bb0270, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:359 +0x60
net/textproto.(*Reader).readLineSlice(0x10e87460, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:55 +0x58
net/textproto.(*Reader).ReadLine(0x10e87460, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/textproto/reader.go:36 +0x34
net/http.readRequest(0x10bb0270, 0x53f800, 0x10e63780, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:793 +0x80
net/http.(*conn).readRequest(0x10dee780, 0x6a0f90, 0x10be0140, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:765 +0x3d4
net/http.(*conn).serve(0x10dee780, 0x6a0f90, 0x10be0140)
    /usr/local/go/src/net/http/server.go:1532 +0x910
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2293 +0x470

goroutine 3317 [runnable]:
main.(*StatusStruct).Save(0x6b7088)
    /home/pi/printer/app/db.go:430
created by main.(*StatusStruct).SetCurrentHeightMm
    /home/pi/printer/app/db.go:396 +0xac

goroutine 3295 [runnable]:
net.goLookupIPOrder.func1(0x6a0fe0, 0x10a4ad80, 0x10bb4140, 0x10afc148, 0x10a4ae00, 0x110a0001)
    /usr/local/go/src/net/dnsclient_unix.go:486
created by net.goLookupIPOrder
    /usr/local/go/src/net/dnsclient_unix.go:489 +0x308

goroutine 3293 [select]:
net.lookupIPContext(0x6a0fe0, 0x10a4ad80, 0x10e5c5e0, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/lookup.go:122 +0x6d0
net.internetAddrList(0x6a0fe0, 0x10a4ad80, 0x4f362e, 0x3, 0x10e5c5e0, 0x12, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/net/ipsock.go:241 +0x4e0
net.resolveAddrList(0x6a0fe0, 0x10a4ad80, 0x4f3acc, 0x4, 0x4f362e, 0x3, 0x10e5c5e0, 0x12, 0x0, 0x0, ...)
    /usr/local/go/src/net/dial.go:179 +0x624
net.(*Dialer).DialContext(0x10a4a200, 0x6a0fe0, 0x10a4ad80, 0x4f362e, 0x3, 0x10e5c5e0, 0x12, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/net/dial.go:329 +0x474
net.(*Dialer).DialContext-fm(0x6a0fb8, 0x10a10188, 0x4f362e, 0x3, 0x10e5c5e0, 0x12, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:43 +0x6c
net/http.(*Transport).dial(0x10a800a0, 0x6a0fb8, 0x10a10188, 0x4f362e, 0x3, 0x10e5c5e0, 0x12, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/net/http/transport.go:821 +0x78
net/http.(*Transport).dialConn(0x10a800a0, 0x6a0fb8, 0x10a10188, 0x0, 0x4ff127, 0x4, 0x10e5c5e0, 0x12, 0x6b70f8, 0x0, ...)
    /usr/local/go/src/net/http/transport.go:962 +0x17b8
net/http.(*Transport).getConn.func4(0x10a800a0, 0x6a0fb8, 0x10a10188, 0x10e5c600, 0x10a4ab40)
    /usr/local/go/src/net/http/transport.go:880 +0x40
created by net/http.(*Transport).getConn
    /usr/local/go/src/net/http/transport.go:882 +0x34c

trap    0x0
error   0x0
oldmask 0x0
r0      0x0
r1      0x5f24
r2      0x6
r3      0x0
r4      0x76d9b094
r5      0x7d751460
r6      0x0
r7      0x10c
r8      0x1
r9      0x58204920
r10     0x111281e0
fp      0x698c94
ip      0x7d751920
sp      0x7d750ac0
lr      0x76c8af44
pc      0x76c8af70
cpsr    0x20000010
fault   0x0`
@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 5, 2016

Thanks for posting the repro. What does ulimit -a print for the account
that is running that program.

On Sat, 5 Nov 2016, 10:15 shahinv notifications@github.com wrote:

@davecheney https://github.com/davecheney

It takes up to a day to have it crashed. But I have large number of bug
reports from people who are using the program. They looks similar.

`runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x76c8af70 m=148

goroutine 0 [idle]:

goroutine 1 [IO wait, 5 minutes]:
net.runtime_pollWait(0x75213f00, 0x72, 0x69f360)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(

_pollDesc).wait(0x10b6a4b8, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34 net.(_pollDesc).waitRead(0x10b6a4b8,
0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(

_netFD).accept(0x10b6a480, 0x0, 0x69e838, 0x10ba2c70)
/usr/local/go/src/net/fd_unix.go:419 +0x21c net.(_TCPListener).accept(0x10b2ab18,
0x2470d1ff, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x30
net.(

_TCPListener).AcceptTCP(0x10b2ab18, 0xa04b0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock.go:209 +0x9c
net/http.tcpKeepAliveListener.Accept(0x10b2ab18, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2608 +0x3c net/http.(_Server).Serve(0x10b3c140,
0x6a0b78, 0x10b2ab18, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2273 +0x200
net/http.(*Server).ListenAndServe(0x10b3c140, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2219 +0xf4
main.webInit()
/home/pi/printer/app/web.go:939 +0x263c
main.main()
/home/pi/printer/app/server.go:70 +0x158

goroutine 17 [syscall, 159 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4

goroutine 38 [runnable]:
syscall.Syscall(0x3, 0xa, 0x11158e00, 0x200, 0xef, 0x0, 0x0)
/usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8
syscall.read(0xa, 0x11158e00, 0x200, 0x200, 0xf954900, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_arm.go:783 +0x78
syscall.Read(0xa, 0x11158e00, 0x200, 0x200, 0x5c718, 0x0, 0x0)
/usr/local/go/src/syscall/syscall_unix.go:161 +0x4c
os.(

_File).read(0x10afc1a8, 0x11158e00, 0x200, 0x200, 0x1ffffff, 0x0, 0x0)
/usr/local/go/src/os/file_unix.go:228 +0x54 os.(_File).Read(0x10afc1a8,
0x11158e00, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/os/file.go:101 +0x7c
bytes.(*Buffer).ReadFrom(0x10b42eb8, 0x69e7c0, 0x10afc1a8, 0x0, 0x0, 0x0,
0x0)
/usr/local/go/src/bytes/buffer.go:176 +0x22c
io/ioutil.readAll(0x69e7c0, 0x10afc1a8, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0)
/usr/local/go/src/io/ioutil/ioutil.go:33 +0x11c
io/ioutil.ReadFile(0x4f9f68, 0x12, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/io/ioutil/ioutil.go:70 +0x16c
main.WifiStatus()
/home/pi/printer/app/wireless.go:94 +0x150
created by main.wifiConnect
/home/pi/printer/app/wireless.go:22 +0x7c

goroutine 6 [sleep]:
time.Sleep(0x540be400, 0x2)
/usr/local/go/src/runtime/time.go:59 +0xec
main.(

_StatusStruct).FindIP(0x6b7088) /home/pi/printer/app/db.go:472 +0xe0
created by main.(_StatusStruct).Load
/home/pi/printer/app/db.go:438 +0x58

goroutine 7 [sleep]:
time.Sleep(0x2a05f200, 0x1)
/usr/local/go/src/runtime/time.go:59 +0xec
main.faultDetection()
/home/pi/printer/app/hardware-interface.go:303 +0x3a4
created by main.hwInit
/home/pi/printer/app/hardware-interface.go:29 +0x21c

goroutine 11 [syscall]:
syscall.Syscall(0x3, 0x4, 0x11279efc, 0x1, 0x1b, 0x10aece40, 0x1c)
/usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8
syscall.read(0x4, 0x11279efc, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_arm.go:783 +0x78
syscall.Read(0x4, 0x11279efc, 0x1, 0x1, 0x76bb3258, 0x0, 0x0)
/usr/local/go/src/syscall/syscall_unix.go:161 +0x4c
os.(

_File).read(0x10a0c8f0, 0x11279efc, 0x1, 0x1, 0x10aa8320, 0x0, 0x0)
/usr/local/go/src/os/file_unix.go:228 +0x54 os.(_File).Read(0x10a0c8f0,
0x11279efc, 0x1, 0x1, 0x79, 0x0, 0x0)
/usr/local/go/src/os/file.go:101 +0x7c
github.com/tarm/serial.(

_Port).Read(0x10a0c8f8, 0x11279efc, 0x1, 0x1, 0x10e2ad80, 0x0, 0x0)
/home/pi/go/src/github.com/tarm/serial/serial_posix.go:129
http://github.com/tarm/serial/serial_posix.go:129 +0x50
_/home/pi/printer/app/shield.(_Conf).Read(0x6b6c98, 0x11279efc, 0x1, 0x1,
0x0, 0x0, 0x0)
/home/pi/printer/app/shield/shield.go:90 +0x98
main.(

_termStruct).readLine(0x6b6c70, 0x0, 0x0, 0x0)
/home/pi/printer/app/term.go:54 +0x100 main.(_termStruct).Reader(0x6b6c70)
/home/pi/printer/app/term.go:39 +0xdc
created by main.main
/home/pi/printer/app/server.go:54 +0xac

goroutine 12 [sleep]:
time.Sleep(0x540be400, 0x2)
/usr/local/go/src/runtime/time.go:59 +0xec
main.(*monitorStruct).watch(0x6c5dc8)
/home/pi/printer/app/slice-monitor.go:84 +0x548
created by main.main
/home/pi/printer/app/server.go:55 +0xc8

goroutine 14 [sleep]:
time.Sleep(0x2a05f200, 0x1)
/usr/local/go/src/runtime/time.go:59 +0xec
_/home/pi/printer/app/hw.Monitor()
/home/pi/printer/app/hw/hw-monitor_linux.go:127 +0x7c
created by main.main
/home/pi/printer/app/server.go:57 +0xf0

goroutine 24 [runnable]:
syscall.Syscall(0x4, 0x2, 0x10a7ab00, 0x78, 0x78, 0x0, 0x0)
/usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8
syscall.write(0x2, 0x10a7ab00, 0x78, 0x160, 0x9, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_arm.go:1064 +0x78
syscall.Write(0x2, 0x10a7ab00, 0x78, 0x160, 0x0, 0x0, 0x0)
/usr/local/go/src/syscall/syscall_unix.go:180 +0x4c
os.(

_File).write(0x10a0c1c0, 0x10a7ab00, 0x78, 0x160, 0x0, 0x0, 0x0)
/usr/local/go/src/os/file_unix.go:249 +0x78 os.(_File).Write(0x10a0c1c0,
0x10a7ab00, 0x78, 0x160, 0x34f4b79c, 0x0, 0x0)
/usr/local/go/src/os/file.go:142 +0x7c
log.(

_Logger).Output(0x10a4c090, 0x2, 0x10b3a840, 0x5d, 0x0, 0x0)
/usr/local/go/src/log/log.go:166 +0x328 log.Println(0x11277ddc, 0x1, 0x1)
/usr/local/go/src/log/log.go:295 +0x5c _/home/pi/printer/app/mlog.(_memLog).Add(0x6b6c48,
0x4f3ed7, 0x5, 0x487fff, 0x11277e9c, 0x1, 0x1)
/home/pi/printer/app/mlog/mlog.go:55 +0x6c4
main.memlog(0x4f3ed7, 0x5, 0x10e181ff, 0x10af66b0, 0x2, 0x2)
/home/pi/printer/app/server.go:162 +0x140
main.showFrame(0x10c55ee0, 0x15, 0x15)
/home/pi/printer/app/hardware-interface.go:268 +0x13c
main.printCalibration(0x1)
/home/pi/printer/app/printing-calibration.go:55 +0xbcc
created by main.printerStart
/home/pi/printer/app/web.go:699 +0xf8

goroutine 36 [select]:
net/http.(

_Transport).getConn(0x10a800a0, 0x10ab6080, 0x0, 0x4ff127, 0x4,
0x10e5c5e0, 0x12, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:885 +0x8e0 net/http.(_Transport).RoundTrip(0x10a800a0,
0x10a0ae80, 0x10a800a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:367 +0xa18
net/http.send(0x10a0ae80, 0x69e550, 0x10a800a0, 0x0, 0x0, 0x0, 0x0,
0x10ab6070, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:256 +0x4fc
net/http.(

_Client).send(0x6b6b48, 0x10a0ae80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:146 +0x144 net/http.(_Client).doFollowingRedirects(0x6b6b48,
0x10a0ae80, 0x53f900, 0x502266, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:528 +0xa50
net/http.(

_Client).Post(0x6b6b48, 0x4ff127, 0x1b, 0x502266, 0x21, 0x69e8e0,
0x10e5c580, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/client.go:579 +0xd0
net/http.(_Client).PostForm(0x6b6b48, 0x4ff127, 0x1b, 0x10e5c460,
0x48f878, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:605 +0xd4
net/http.PostForm(0x4ff127, 0x1b, 0x10e5c460, 0x110a1fc4, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:593 +0x50
main.sendDashboardData()
/home/pi/printer/app/server.go:195 +0x884
main.updateControlPanel()
/home/pi/printer/app/server.go:172 +0x14
created by main.afterIP
/home/pi/printer/app/server.go:84 +0xf8

goroutine 3227 [IO wait]:
net.runtime_pollWait(0x75213618, 0x72, 0x11090000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(

_pollDesc).wait(0x10d33238, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34 net.(_pollDesc).waitRead(0x10d33238,
0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(

_netFD).Read(0x10d33200, 0x11090000, 0x1000, 0x1000, 0x0, 0x69f360,
0x10a10090) /usr/local/go/src/net/fd_unix.go:243 +0x1f0 net.(_conn).Read(0x10eb4848,
0x11090000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0xb8
net/http.(

connReader).Read(0x10e3ba80, 0x11090000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:586 +0x1d8 bufio.(

Reader).fill(0x111f4270)
/usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(

_Reader).ReadSlice(0x111f4270, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:330 +0x264 bufio.(_Reader).ReadLine(0x111f4270,
0x0, 0x0, 0x0, 0x94500, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:359 +0x60
net/textproto.(

_Reader).readLineSlice(0x10d6ae00, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:55 +0x58 net/textproto.(_Reader).ReadLine(0x10d6ae00,
0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:36 +0x34
net/http.readRequest(0x111f4270, 0x53f800, 0x10dc5400, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:793 +0x80
net/http.(

_conn).readRequest(0x10d33280, 0x6a0f90, 0x10e3bac0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:765 +0x3d4 net/http.(_conn).serve(0x10d33280,
0x6a0f90, 0x10e3bac0)
/usr/local/go/src/net/http/server.go:1532 +0x910
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x470

goroutine 3294 [chan receive]:
net.goLookupIPOrder(0x6a0fe0, 0x10a4ad80, 0x10e5c5e0, 0xf, 0x1, 0x0, 0x0,
0x0, 0x0, 0x0)
/usr/local/go/src/net/dnsclient_unix.go:492 +0x37c
net.lookupIP(0x6a0fe0, 0x10a4ad80, 0x10e5c5e0, 0xf, 0x0, 0x0, 0x0, 0x0,
0x0)
/usr/local/go/src/net/lookup_unix.go:76 +0xfc
net.glob..func11(0x6a0fe0, 0x10a4ad80, 0x53f7dc, 0x10e5c5e0, 0xf, 0x0,
0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/hook.go:19 +0x58
net.lookupIPContext.func1(0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/lookup.go:119 +0x78
internal/singleflight.(

_Group).doCall(0x6b6920, 0x10e84090, 0x10e5c5e0, 0xf, 0x10e5c6c0)
/usr/local/go/src/internal/singleflight/singleflight.go:93 +0x1c created by
internal/singleflight.(_Group).DoChan
/usr/local/go/src/internal/singleflight/singleflight.go:86 +0x37c

goroutine 3318 [runnable]:
main.shutterOpen()
/home/pi/printer/app/hardware-interface.go:217
created by main.printCalibration
/home/pi/printer/app/printing-calibration.go:38 +0x4f4

goroutine 3296 [runnable]:
net.runtime_pollWait(0x75213ca8, 0x72, 0x11158000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(

_pollDesc).wait(0x10a4b038, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34 net.(_pollDesc).waitRead(0x10a4b038,
0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(

_netFD).Read(0x10a4b000, 0x11158000, 0x200, 0x200, 0x0, 0x69f360,
0x10a10090) /usr/local/go/src/net/fd_unix.go:243 +0x1f0 net.(_conn).Read(0x10afc170,
0x11158000, 0x200, 0x200, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0xb8
net.dnsRoundTripUDP(0x69fdb8, 0x10afc170, 0x10a4afc0, 0x50d629ef, 0x0, 0x0)
/usr/local/go/src/net/dnsclient_unix.go:64 +0x18c
net.(*UDPConn).dnsRoundTrip(0x10afc170, 0x10a4afc0, 0x35064362, 0x0, 0x0)
/usr/local/go/src/net/dnsclient_unix.go:47 +0x44
net.exchange(0x6a0fe0, 0x10a4ae80, 0x10aecc80, 0xf, 0x10ab60d0, 0x10,
0x6a001c, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dnsclient_unix.go:164 +0x400
net.tryOneName(0x6a0fe0, 0x10a4ae80, 0x10bb4140, 0x10ab60d0, 0x10, 0x1c,
0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/dnsclient_unix.go:193 +0x418
net.goLookupIPOrder.func1(0x6a0fe0, 0x10a4ad80, 0x10bb4140, 0x10afc148,
0x10a4ae00, 0x110a001c)
/usr/local/go/src/net/dnsclient_unix.go:487 +0x4c
created by net.goLookupIPOrder
/usr/local/go/src/net/dnsclient_unix.go:489 +0x308

goroutine 3209 [IO wait]:
net.runtime_pollWait(0x752137f8, 0x72, 0x10c50000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(

_pollDesc).wait(0x10dee738, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34 net.(_pollDesc).waitRead(0x10dee738,
0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(

_netFD).Read(0x10dee700, 0x10c50000, 0x1000, 0x1000, 0x0, 0x69f360,
0x10a10090) /usr/local/go/src/net/fd_unix.go:243 +0x1f0 net.(_conn).Read(0x10ab4df8,
0x10c50000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0xb8
net/http.(

connReader).Read(0x10e37fc0, 0x10c50000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:586 +0x1d8 bufio.(

Reader).fill(0x10bb0270)
/usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(

_Reader).ReadSlice(0x10bb0270, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:330 +0x264 bufio.(_Reader).ReadLine(0x10bb0270,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:359 +0x60
net/textproto.(

_Reader).readLineSlice(0x10e87460, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:55 +0x58 net/textproto.(_Reader).ReadLine(0x10e87460,
0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:36 +0x34
net/http.readRequest(0x10bb0270, 0x53f800, 0x10e63780, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:793 +0x80
net/http.(

_conn).readRequest(0x10dee780, 0x6a0f90, 0x10be0140, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:765 +0x3d4 net/http.(_conn).serve(0x10dee780,
0x6a0f90, 0x10be0140)
/usr/local/go/src/net/http/server.go:1532 +0x910
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x470

goroutine 3317 [runnable]:
main.(

_StatusStruct).Save(0x6b7088) /home/pi/printer/app/db.go:430 created by
main.(_StatusStruct).SetCurrentHeightMm
/home/pi/printer/app/db.go:396 +0xac

goroutine 3295 [runnable]:
net.goLookupIPOrder.func1(0x6a0fe0, 0x10a4ad80, 0x10bb4140, 0x10afc148,
0x10a4ae00, 0x110a0001)
/usr/local/go/src/net/dnsclient_unix.go:486
created by net.goLookupIPOrder
/usr/local/go/src/net/dnsclient_unix.go:489 +0x308

goroutine 3293 [select]:
net.lookupIPContext(0x6a0fe0, 0x10a4ad80, 0x10e5c5e0, 0xf, 0x0, 0x0, 0x0,
0x0, 0x0)
/usr/local/go/src/net/lookup.go:122 +0x6d0
net.internetAddrList(0x6a0fe0, 0x10a4ad80, 0x4f362e, 0x3, 0x10e5c5e0,
0x12, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/ipsock.go:241 +0x4e0
net.resolveAddrList(0x6a0fe0, 0x10a4ad80, 0x4f3acc, 0x4, 0x4f362e, 0x3,
0x10e5c5e0, 0x12, 0x0, 0x0, ...)
/usr/local/go/src/net/dial.go:179 +0x624
net.(

_Dialer).DialContext(0x10a4a200, 0x6a0fe0, 0x10a4ad80, 0x4f362e, 0x3,
0x10e5c5e0, 0x12, 0x0, 0x0, 0x0, ...) /usr/local/go/src/net/dial.go:329
+0x474 net.(_Dialer).DialContext-fm(0x6a0fb8, 0x10a10188, 0x4f362e, 0x3,
0x10e5c5e0, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:43 +0x6c
net/http.(

_Transport).dial(0x10a800a0, 0x6a0fb8, 0x10a10188, 0x4f362e, 0x3,
0x10e5c5e0, 0x12, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/http/transport.go:821 +0x78 net/http.(_Transport).dialConn(0x10a800a0,
0x6a0fb8, 0x10a10188, 0x0, 0x4ff127, 0x4, 0x10e5c5e0, 0x12, 0x6b70f8, 0x0,
...)
/usr/local/go/src/net/http/transport.go:962 +0x17b8
net/http.(

_Transport).getConn.func4(0x10a800a0, 0x6a0fb8, 0x10a10188, 0x10e5c600,
0x10a4ab40) /usr/local/go/src/net/http/transport.go:880 +0x40 created by
net/http.(_Transport).getConn
/usr/local/go/src/net/http/transport.go:882 +0x34c

trap 0x0
error 0x0
oldmask 0x0
r0 0x0
r1 0x5f24
r2 0x6
r3 0x0
r4 0x76d9b094
r5 0x7d751460
r6 0x0
r7 0x10c
r8 0x1
r9 0x58204920
r10 0x111281e0
fp 0x698c94
ip 0x7d751920
sp 0x7d750ac0
lr 0x76c8af44
pc 0x76c8af70
cpsr 0x20000010
fault 0x0`


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17803 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA4HGfvGFcoT7V5HniGF0cWN4RO7iks5q67yDgaJpZM4KqEOq
.

@shahinv

This comment has been minimized.

shahinv commented Nov 5, 2016

pi@raspberrypi:~ $ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 6829
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 6829
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 5, 2016

Which kernel (sorry if this has been already answered)

On Sat, 5 Nov 2016, 19:09 shahinv notifications@github.com wrote:

pi@raspberrypi:~ $ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 6829
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 6829
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17803 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcAw7CnwWY357sEx-RF4cDamKdrbsSks5q7DnFgaJpZM4KqEOq
.

@shahinv

This comment has been minimized.

shahinv commented Nov 5, 2016

Thank you for the investigation.
4.4.21-v7+ (raspian jessie lite)
The issue reported from multiple sources possibly with different kernel versions.

Dave, I am not sure if the issue is isolated to 1.7, as far as I remember I had crashes albeit very rarely on 1.6 too not 100% sure if it was the same thread issue but reports were similar. Go 1.5/1.4 were very solid.

@quentinmit quentinmit added this to the Go1.8 milestone Nov 7, 2016

@quentinmit quentinmit changed the title from Changing process priority on ARM crashes the program to runtime: nice -20 causes thread exhaustion on ARM Nov 7, 2016

@quentinmit

This comment has been minimized.

Contributor

quentinmit commented Nov 7, 2016

@shahinv -20 is the highest priority on Linux; I suspect you are starving the other processes and kernel on the system, and your system calls are not completing in a timely manner. That said, I would expect to see those syscalls in the goroutine trace. Did you trim the backtrace above, or does it really skip from goroutine 36 to goroutine 3227?

@quentinmit

This comment has been minimized.

Contributor

quentinmit commented Nov 7, 2016

@shahinv Can you try to reproduce a crash with GOTRACEBACK=crash set in the environment?

That will cause the traceback to contain more information.

@shahinv

This comment has been minimized.

shahinv commented Nov 7, 2016

@quentinmit no I have not trimmed the panic message. I have attached another one for your reference.

Sure I will try reproduce that with GOTRACEBACK=crash. Will share the result shortly.

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x76d20f70 m=142

goroutine 0 [idle]:

goroutine 34 [syscall]:
runtime.notetsleepg(0x793034, 0x1bcf27f0, 0x0, 0x109ac500)
/usr/local/go/src/runtime/lock_futex.go:205 +0x4c fp=0x10918788 sp=0x10918770
runtime.timerproc()
/usr/local/go/src/runtime/time.go:209 +0x11c fp=0x109187dc sp=0x10918788
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:990 +0x4 fp=0x109187dc sp=0x109187dc
created by runtime.addtimerLocked
/usr/local/go/src/runtime/time.go:116 +0x1c4

goroutine 1 [IO wait, 2 minutes]:
net.runtime_pollWait(0x75a39fb8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0x109dbf78, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(_pollDesc).WaitRead(0x109dbf78, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(_netFD).accept(0x109dbf40, 0x0, 0x75a3a030, 0x10f007e0)
/usr/local/go/src/net/fd_unix.go:426 +0x21c
net.(_TCPListener).AcceptTCP(0x109b29c8, 0x109479e8, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4c
net/http.tcpKeepAliveListener.Accept(0x109b29c8, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2427 +0x3c
net/http.(_Server).Serve(0x10a68140, 0x75a3a160, 0x109b29c8, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2117 +0xfc
net/http.(_Server).ListenAndServe(0x10a68140, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2098 +0x124
net/http.ListenAndServe(0x10a0a858, 0x3, 0x75a3a0e0, 0x109d84b0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2195 +0x90
github.com/plimble/ace.(*Ace).Run(0x109d84b0, 0x10a0a858, 0x3)
/home/pi/go/src/github.com/plimble/ace/ace.go:75 +0x74
main.webInit()
/home/pi/printer/app/web.go:838 +0x34c4
main.main()
/home/pi/printer/app/server.go:68 +0x158

goroutine 17 [syscall, 1587 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:990 +0x4

goroutine 19 [sleep]:
time.Sleep(0x2a05f200, 0x1)
/usr/local/go/src/runtime/time.go:59 +0x104
main.faultDetection()
/home/pi/printer/app/hardware-interface.go:289 +0x3a0
created by main.hwInit
/home/pi/printer/app/hardware-interface.go:29 +0x20c

goroutine 7 [sleep]:
time.Sleep(0x7e11d600, 0x3)
/usr/local/go/src/runtime/time.go:59 +0x104
main.WifiStatus()
/home/pi/printer/app/wireless.go:104 +0x320
created by main.wifiConnect
/home/pi/printer/app/wireless.go:22 +0x7c

goroutine 23 [syscall]:
syscall.Syscall(0x3, 0x4, 0x11a83eec, 0x1, 0x471f80, 0x10f019e0, 0x549ab8)
/usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8
syscall.read(0x4, 0x11a83eec, 0x1, 0x1, 0x47a818, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_arm.go:783 +0x78
syscall.Read(0x4, 0x11a83eec, 0x1, 0x1, 0x10bc6a00, 0x0, 0x0)
/usr/local/go/src/syscall/syscall_unix.go:161 +0x4c
os.(_File).read(0x109ba2b8, 0x11a83eec, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/os/file_unix.go:228 +0x54
os.(_File).Read(0x109ba2b8, 0x11a83eec, 0x1, 0x1, 0x4, 0x0, 0x0)
/usr/local/go/src/os/file.go:95 +0x7c
github.com/tarm/serial.(_Port).Read(0x109ba2c0, 0x11a83eec, 0x1, 0x1, 0x11aecc00, 0x0, 0x0)
/home/pi/go/src/github.com/tarm/serial/serial_posix.go:129 +0x50
_/home/pi/printer/app/shield.(_Conf).Read(0x793090, 0x11a83eec, 0x1, 0x1, 0x0, 0x0, 0x0)
/home/pi/printer/app/shield/shield.go:90 +0x98
main.(_termStruct).readLine(0x793068, 0x0, 0x0, 0x0)
/home/pi/printer/app/term.go:54 +0x100
main.(_termStruct).Reader(0x793068)
/home/pi/printer/app/term.go:39 +0xd8
created by main.main
/home/pi/printer/app/server.go:52 +0xac

goroutine 24 [sleep]:
time.Sleep(0x540be400, 0x2)
/usr/local/go/src/runtime/time.go:59 +0x104
main.(*monitorStruct).watch(0x7a27d8)
/home/pi/printer/app/slice-monitor.go:83 +0x4d8
created by main.main
/home/pi/printer/app/server.go:53 +0xc8

goroutine 26 [sleep]:
time.Sleep(0x2a05f200, 0x1)
/usr/local/go/src/runtime/time.go:59 +0x104
_/home/pi/printer/app/hw.Monitor()
/home/pi/printer/app/hw/hw-monitor_linux.go:127 +0x7c
created by main.main
/home/pi/printer/app/server.go:55 +0xf0

goroutine 31 [sleep, 2 minutes]:
time.Sleep(0xf8475800, 0xd)
/usr/local/go/src/runtime/time.go:59 +0x104
main.updateControlPanel()
/home/pi/printer/app/server.go:167 +0x28
created by main.afterIP
/home/pi/printer/app/server.go:82 +0xe4

goroutine 7102 [IO wait]:
net.runtime_pollWait(0x75a39838, 0x72, 0x1577d000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0x1170cff8, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(_pollDesc).WaitRead(0x1170cff8, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(_netFD).Read(0x1170cfc0, 0x1577d000, 0x1000, 0x1000, 0x0, 0x75af9078, 0x1090e048)
/usr/local/go/src/net/fd_unix.go:250 +0x1c4
net.(_conn).Read(0x10f164c8, 0x1577d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xc8
net/http.(_connReader).Read(0x12890dc0, 0x1577d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:526 +0x1d8
bufio.(_Reader).fill(0x10e93cb0)
/usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(_Reader).ReadSlice(0x10e93cb0, 0x1025c70a, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:328 +0x264
bufio.(_Reader).ReadLine(0x10e93cb0, 0x0, 0x0, 0x0, 0x48200, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:357 +0x60
net/textproto.(_Reader).readLineSlice(0x142e7980, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:55 +0x58
net/textproto.(_Reader).ReadLine(0x142e7980, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:36 +0x34
net/http.readRequest(0x10e93cb0, 0x0, 0x173387e0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:721 +0x80
net/http.(_conn).readRequest(0x1170d040, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:705 +0x3d4
net/http.(_conn).serve(0x1170d040)
/usr/local/go/src/net/http/server.go:1425 +0x93c
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2137 +0x3bc

goroutine 7103 [IO wait]:
net.runtime_pollWait(0x75a39748, 0x72, 0x11914000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0x1170d0f8, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(_pollDesc).WaitRead(0x1170d0f8, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(_netFD).Read(0x1170d0c0, 0x11914000, 0x1000, 0x1000, 0x0, 0x75af9078, 0x1090e048)
/usr/local/go/src/net/fd_unix.go:250 +0x1c4
net.(_conn).Read(0x10f164d0, 0x11914000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xc8
net/http.(_connReader).Read(0x12889ac0, 0x11914000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:526 +0x1d8
bufio.(_Reader).fill(0x12870090)
/usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(_Reader).ReadSlice(0x12870090, 0x108fa70a, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:328 +0x264
bufio.(_Reader).ReadLine(0x12870090, 0x0, 0x0, 0x0, 0x1090c300, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:357 +0x60
net/textproto.(_Reader).readLineSlice(0x1720e0c0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:55 +0x58
net/textproto.(_Reader).ReadLine(0x1720e0c0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:36 +0x34
net/http.readRequest(0x12870090, 0x0, 0x109581c0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:721 +0x80
net/http.(_conn).readRequest(0x1170d140, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:705 +0x3d4
net/http.(_conn).serve(0x1170d140)
/usr/local/go/src/net/http/server.go:1425 +0x93c
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2137 +0x3bc

goroutine 5228 [runnable]:
time.Sleep(0xf4240, 0x0)
/usr/local/go/src/runtime/time.go:59 +0x104
main.waitForWaitMessage()
/home/pi/printer/app/hardware-interface.go:77 +0x3c
main.processInternalKeyword(0x12890180, 0x3, 0x3)
/home/pi/printer/app/gcode.go:64 +0xd14
main.gcodeExecute(0x11a20300, 0x17)
/home/pi/printer/app/gcode.go:40 +0x18c
main.gcodeSend(0x10979c30, 0xa4, 0x0, 0x0)
/home/pi/printer/app/gcode.go:27 +0xd8
main.printLayer(0xba, 0x24e, 0xba)
/home/pi/printer/app/printing.go:137 +0x130
main.startPlate(0x8, 0x1)
/home/pi/printer/app/printing.go:77 +0x368
created by main.printerStart
/home/pi/printer/app/web.go:615 +0x134

trap 0x0
error 0x0
oldmask 0x0
r0 0x0
r1 0x7e2d
r2 0x6
r3 0x0
r4 0x76e31094
r5 0x7a7e7460
r6 0x0
r7 0x10c
r8 0x1
r9 0x78fe7920
r10 0x14347790
fp 0x779518
ip 0x7a7e7920
sp 0x7a7e6b00
lr 0x76d20f44
pc 0x76d20f70
cpsr 0x20000010
fault 0x0

@shahinv

This comment has been minimized.

shahinv commented Nov 8, 2016

@quentinmit

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x76c44f70 m=145

goroutine 0 [idle]:

goroutine 18 [syscall]:
runtime.notetsleepg(0x691f0c, 0x9a1cfd, 0x0, 0x1)
/usr/local/go/src/runtime/lock_futex.go:205 +0x4c fp=0x10920788 sp=0x10920770
runtime.timerproc()
/usr/local/go/src/runtime/time.go:209 +0x11c fp=0x109207dc sp=0x10920788
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x109207dc sp=0x109207dc
created by runtime.addtimerLocked
/usr/local/go/src/runtime/time.go:116 +0x1c4

goroutine 1 [IO wait, 13 minutes]:
runtime.gopark(0x51cd78, 0x74080f8c, 0x4d2957, 0x7, 0x1b, 0x5)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10abbbe0 sp=0x10abbbcc
runtime.netpollblock(0x74080f78, 0x72, 0x10abbc00, 0x54a10)
/usr/local/go/src/runtime/netpoll.go:341 +0x178 fp=0x10abbc04 sp=0x10abbbe0
net.runtime_pollWait(0x74080f78, 0x72, 0x67a788)
/usr/local/go/src/runtime/netpoll.go:160 +0x60 fp=0x10abbc18 sp=0x10abbc04
net.(_pollDesc).wait(0x109d16f8, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34 fp=0x10abbc28 sp=0x10abbc18
net.(_pollDesc).waitRead(0x109d16f8, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30 fp=0x10abbc3c sp=0x10abbc28
net.(_netFD).accept(0x109d16c0, 0x0, 0x679d50, 0x10b50e30)
/usr/local/go/src/net/fd_unix.go:419 +0x21c fp=0x10abbcbc sp=0x10abbc3c
net.(_TCPListener).accept(0x109f0240, 0x2470d1ff, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x30 fp=0x10abbcd8 sp=0x10abbcbc
net.(_TCPListener).AcceptTCP(0x109f0240, 0xa23e0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock.go:209 +0x9c fp=0x10abbd00 sp=0x10abbcd8
github.com/labstack/echo/engine/standard.tcpKeepAliveListener.Accept(0x109f0240, 0x0, 0x0, 0x0, 0x0)
/home/pi/go/src/github.com/labstack/echo/engine/standard/server.go:199 +0x3c fp=0x10abbd1c sp=0x10abbd00
net/http.(_Server).Serve(0x10a4e050, 0x67bf00, 0x109f0240, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2273 +0x200 fp=0x10abbdcc sp=0x10abbd1c
github.com/labstack/echo/engine/standard.(_Server).Start(0x10a4e0a0, 0x0, 0x0)
/home/pi/go/src/github.com/labstack/echo/engine/standard/server.go:130 +0x3d4 fp=0x10abbe94 sp=0x10abbdcc
github.com/labstack/echo.(_Echo).Run(0x10995140, 0x67c6d0, 0x10a4e0a0, 0x0, 0x0)
/home/pi/go/src/github.com/labstack/echo/echo.go:581 +0x1b0 fp=0x10abbedc sp=0x10abbe94
main.webInit()
/home/pi/printer/app/web.go:975 +0x17dc fp=0x10abbf94 sp=0x10abbedc
main.main()
/home/pi/printer/app/server.go:70 +0x158 fp=0x10abbfa4 sp=0x10abbf94
runtime.main()
/usr/local/go/src/runtime/proc.go:183 +0x264 fp=0x10abbfcc sp=0x10abbfa4
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10abbfcc sp=0x10abbfcc

goroutine 17 [syscall, 584 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10930fd8 sp=0x10930fd8

goroutine 2 [force gc (idle), 3 minutes]:
runtime.gopark(0x51cd98, 0x691cf8, 0x4d693c, 0xf, 0x14, 0x1)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x1091e7a8 sp=0x1091e794
runtime.goparkunlock(0x691cf8, 0x4d693c, 0xf, 0x14, 0x1)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x1091e7c4 sp=0x1091e7a8
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:224 +0xa8 fp=0x1091e7dc sp=0x1091e7c4
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x1091e7dc sp=0x1091e7dc
created by runtime.init.3
/usr/local/go/src/runtime/proc.go:213 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0x51cd98, 0x691e28, 0x4d5844, 0xd, 0xa1314, 0x1)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x1091ef9c sp=0x1091ef88
runtime.goparkunlock(0x691e28, 0x4d5844, 0xd, 0x14, 0x1)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x1091efb8 sp=0x1091ef9c
runtime.bgsweep(0x1090e040)
/usr/local/go/src/runtime/mgcsweep.go:79 +0x13c fp=0x1091efd4 sp=0x1091efb8
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x1091efd4 sp=0x1091efd4
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:195 +0x4c

goroutine 4 [finalizer wait, 539 minutes]:
runtime.gopark(0x51cd98, 0x6a0f84, 0x4d61cb, 0xe, 0x14, 0x1)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x1091f788 sp=0x1091f774
runtime.goparkunlock(0x6a0f84, 0x4d61cb, 0xe, 0x14, 0x1)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x1091f7a4 sp=0x1091f788
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:158 +0xa4 fp=0x1091f7dc sp=0x1091f7a4
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x1091f7dc sp=0x1091f7dc
created by runtime.createfing
/usr/local/go/src/runtime/mfinal.go:139 +0x5c

goroutine 6 [sleep]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x176e13, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x1092ef70 sp=0x1092ef5c
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x692313, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x1092ef8c sp=0x1092ef70
time.Sleep(0x540be400, 0x2)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x1092efb0 sp=0x1092ef8c
main.(_StatusStruct).FindIP(0x692330)
/home/pi/printer/app/db.go:511 +0xe0 fp=0x1092efcc sp=0x1092efb0
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x1092efcc sp=0x1092efcc
created by main.(_StatusStruct).Load
/home/pi/printer/app/db.go:477 +0x58

goroutine 7 [sleep]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x13, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10920f50 sp=0x10920f3c
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x13, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x10920f6c sp=0x10920f50
time.Sleep(0x2a05f200, 0x1)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x10920f90 sp=0x10920f6c
main.faultDetection()
/home/pi/printer/app/hardware-interface.go:327 +0x3a4 fp=0x10920fdc sp=0x10920f90
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10920fdc sp=0x10920fdc
created by main.hwInit
/home/pi/printer/app/hardware-interface.go:33 +0x21c

goroutine 75 [GC worker (idle), 1 minutes]:
runtime.gopark(0x51ccd4, 0x10b50040, 0x4d6d86, 0x10, 0x14, 0x0)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10a3ff98 sp=0x10a3ff84
runtime.gcBgMarkWorker(0x10918000)
/usr/local/go/src/runtime/mgc.go:1420 +0xe0 fp=0x10a3ffd4 sp=0x10a3ff98
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10a3ffd4 sp=0x10a3ffd4
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x84

goroutine 12 [sleep]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x13, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10921ea4 sp=0x10921e90
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x10a28013, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x10921ec0 sp=0x10921ea4
time.Sleep(0x5f5e100, 0x0)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x10921ee4 sp=0x10921ec0
main.(_termStruct).readLine(0x691f60, 0x0, 0x0, 0x0)
/home/pi/printer/app/term.go:51 +0x7b0 fp=0x10921f8c sp=0x10921ee4
main.(_termStruct).Reader(0x691f60)
/home/pi/printer/app/term.go:39 +0xdc fp=0x10921fd4 sp=0x10921f8c
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10921fd4 sp=0x10921fd4
created by main.main
/home/pi/printer/app/server.go:54 +0xac

goroutine 13 [sleep]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x4d1313, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x11e53f28 sp=0x11e53f14
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x33e13, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x11e53f44 sp=0x11e53f28
time.Sleep(0x540be400, 0x2)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x11e53f68 sp=0x11e53f44
main.(*monitorStruct).watch(0x6a1040)
/home/pi/printer/app/slice-monitor.go:85 +0x548 fp=0x11e53fd4 sp=0x11e53f68
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x11e53fd4 sp=0x11e53fd4
created by main.main
/home/pi/printer/app/server.go:55 +0xc8

goroutine 73 [GC worker (idle), 1 minutes]:
runtime.gopark(0x51ccd4, 0x10b50030, 0x4d6d86, 0x10, 0x14, 0x0)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10a3c798 sp=0x10a3c784
runtime.gcBgMarkWorker(0x10916a00)
/usr/local/go/src/runtime/mgc.go:1420 +0xe0 fp=0x10a3c7d4 sp=0x10a3c798
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10a3c7d4 sp=0x10a3c7d4
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x84

goroutine 15 [sleep]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x10b50913, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10ad7f80 sp=0x10ad7f6c
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x13, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x10ad7f9c sp=0x10ad7f80
time.Sleep(0xa13b8600, 0x1)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x10ad7fc0 sp=0x10ad7f9c
projects/printer/app/hw.Monitor()
/home/pi/go/src/projects/printer/app/hw/hw-monitor_linux.go:38 +0x7c fp=0x10ad7fcc sp=0x10ad7fc0
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10ad7fcc sp=0x10ad7fcc
created by main.main
/home/pi/printer/app/server.go:57 +0xf0

goroutine 35 [IO wait]:
runtime.gopark(0x51cd78, 0x74080f14, 0x4d2957, 0x7, 0x1b, 0x5)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10ad18e4 sp=0x10ad18d0
runtime.netpollblock(0x74080f00, 0x72, 0x10ad1900, 0x54a10)
/usr/local/go/src/runtime/netpoll.go:341 +0x178 fp=0x10ad1908 sp=0x10ad18e4
net.runtime_pollWait(0x74080f00, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60 fp=0x10ad191c sp=0x10ad1908
net.(_pollDesc).wait(0x10a500b8, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x34 fp=0x10ad192c sp=0x10ad191c
net.(_pollDesc).waitRead(0x10a500b8, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x30 fp=0x10ad1940 sp=0x10ad192c
net.(_netFD).readFrom(0x10a50080, 0x10ad1a8c, 0x400, 0x400, 0x0, 0x0, 0x0, 0x67a788, 0x10910060)
/usr/local/go/src/net/fd_unix.go:270 +0x20c fp=0x10ad19b4 sp=0x10ad1940
net.(_UDPConn).readFrom(0x10a26210, 0x10ad1a8c, 0x400, 0x400, 0x551f780, 0x12c833fb, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:43 +0x54 fp=0x10ad1a1c sp=0x10ad19b4
net.(*UDPConn).ReadFromUDP(0x10a26210, 0x10ad1a8c, 0x400, 0x400, 0x10afe400, 0x192, 0x0, 0x0)
/usr/local/go/src/net/udpsock.go:85 +0xc8 fp=0x10ad1a58 sp=0x10ad1a1c
projects/printer/app/ssdp.Server(0x109c76ca, 0x2, 0x10979f4f, 0x8)
/home/pi/go/src/projects/printer/app/ssdp/ssdp.go:57 +0x1ec fp=0x10ad1fbc sp=0x10ad1a58
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10ad1fbc sp=0x10ad1fbc
created by main.afterIP
/home/pi/printer/app/server.go:82 +0x90

goroutine 37 [sleep]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x13, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10931f88 sp=0x10931f74
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x4d2d13, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x10931fa4 sp=0x10931f88
time.Sleep(0xf8475800, 0xd)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x10931fc8 sp=0x10931fa4
main.updateControlPanel()
/home/pi/printer/app/server.go:173 +0x28 fp=0x10931fd4 sp=0x10931fc8
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10931fd4 sp=0x10931fd4
created by main.afterIP
/home/pi/printer/app/server.go:84 +0xf8

goroutine 72 [GC worker (idle), 1 minutes]:
runtime.gopark(0x51ccd4, 0x10b50028, 0x4d6d86, 0x10, 0x14, 0x0)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x1091af98 sp=0x1091af84
runtime.gcBgMarkWorker(0x10916000)
/usr/local/go/src/runtime/mgc.go:1420 +0xe0 fp=0x1091afd4 sp=0x1091af98
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x1091afd4 sp=0x1091afd4
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x84

goroutine 69 [sleep]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x13, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x1091c738 sp=0x1091c724
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x13, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x1091c754 sp=0x1091c738
time.Sleep(0x540be400, 0x2)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x1091c778 sp=0x1091c754
main.WifiStatus()
/home/pi/printer/app/wireless.go:109 +0x320 fp=0x1091c7dc sp=0x1091c778
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x1091c7dc sp=0x1091c7dc
created by main.wifiConnect
/home/pi/printer/app/wireless.go:24 +0x7c

goroutine 74 [GC worker (idle), 1 minutes]:
runtime.gopark(0x51ccd4, 0x10b50038, 0x4d6d86, 0x10, 0x14, 0x0)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10a3df98 sp=0x10a3df84
runtime.gcBgMarkWorker(0x10917400)
/usr/local/go/src/runtime/mgc.go:1420 +0xe0 fp=0x10a3dfd4 sp=0x10a3df98
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10a3dfd4 sp=0x10a3dfd4
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x84

goroutine 2917 [runnable]:
runtime.gopark(0x51cd98, 0x691f00, 0x4d1f82, 0x5, 0x58216313, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x148 fp=0x10ad5c60 sp=0x10ad5c4c
runtime.goparkunlock(0x691f00, 0x4d1f82, 0x5, 0x405a8013, 0x2)
/usr/local/go/src/runtime/proc.go:265 +0x44 fp=0x10ad5c7c sp=0x10ad5c60
time.Sleep(0x3b8da4, 0x0)
/usr/local/go/src/runtime/time.go:59 +0xec fp=0x10ad5ca0 sp=0x10ad5c7c
projects/printer/app/hw.ZAxisMotor(0x0, 0x0, 0x0, 0x8ff36a, 0x0, 0x8ff36a, 0x0, 0xd5, 0xf0a01, 0x0, ...)
/home/pi/go/src/projects/printer/app/hw/gpio.go:123 +0xa80 fp=0x10ad5d7c sp=0x10ad5ca0
main.zAxisMove(0x0, 0x0, 0x0, 0x8ff36a, 0x0, 0x8ff36a, 0x0, 0xd5, 0xf0a01, 0x0, ...)
/home/pi/printer/app/hardware-interface.go:307 +0x78 fp=0x10ad5db8 sp=0x10ad5d7c
main.gpioInterface(0xd5, 0x101, 0x32, 0x0, 0x0, 0x2c2c8)
/home/pi/printer/app/pulsing.go:85 +0x5ac fp=0x10ad5eb8 sp=0x10ad5db8
main.rise(0xd5, 0x32, 0x0, 0x6a1000, 0x0)
/home/pi/printer/app/pulsing.go:37 +0x40 fp=0x10ad5ed4 sp=0x10ad5eb8
main.riseMicron(0xc8, 0x32, 0x0, 0x0, 0xa)
/home/pi/printer/app/pulsing.go:41 +0x94 fp=0x10ad5eec sp=0x10ad5ed4
main.printLayer(0x1d0, 0x44e, 0x1d1)
/home/pi/printer/app/printing.go:185 +0x4e0 fp=0x10ad5f54 sp=0x10ad5eec
main.startPlate(0x19, 0x190)
/home/pi/printer/app/printing.go:83 +0x368 fp=0x10ad5fc4 sp=0x10ad5f54
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x10ad5fc4 sp=0x10ad5fc4
created by main.printerStart
/home/pi/printer/app/web.go:730 +0x17c

trap 0x0
error 0x0
oldmask 0x0
r0 0x0
r1 0x4d5
r2 0x6
r3 0x0
r4 0x76d55094
r5 0x7bf0b460
r6 0x0
r7 0x10c
r8 0x1
r9 0x7af0b920
r10 0x13b573b0
fp 0x6742e8
ip 0x7bf0b920
sp 0x7bf0ab00
lr 0x76c44f44
pc 0x76c44f70
cpsr 0x20000010
fault 0x0


SIGQUIT: quit
PC=0xa8fa0 m=0

goroutine 0 [idle]:
runtime.futex(0x692c0c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55b98, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/runtime/sys_linux_arm.s:246 +0x1c fp=0x7ea1fb94 sp=0x7ea1fb94
runtime.futexsleep(0x692c0c, 0x0, 0xffffffff, 0xffffffff)
/usr/local/go/src/runtime/os_linux.go:45 +0x68 fp=0x7ea1fbc0 sp=0x7ea1fb94
runtime.notesleep(0x692c0c)
/usr/local/go/src/runtime/lock_futex.go:145 +0xa4 fp=0x7ea1fbd8 sp=0x7ea1fbc0
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1594 +0x100 fp=0x7ea1fbe8 sp=0x7ea1fbd8
runtime.findrunnable(0x10916a00, 0x0)
/usr/local/go/src/runtime/proc.go:2021 +0x758 fp=0x7ea1fc68 sp=0x7ea1fbe8
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2120 +0x26c fp=0x7ea1fc84 sp=0x7ea1fc68
runtime.park_m(0x11ca5950)
/usr/local/go/src/runtime/proc.go:2183 +0x158 fp=0x7ea1fc98 sp=0x7ea1fc84
runtime.mcall(0x692600)
/usr/local/go/src/runtime/asm_arm.s:183 +0x5c fp=0x7ea1fca0 sp=0x7ea1fc98
trap 0x0
error 0x0
oldmask 0x0
r0 0x692c0c
r1 0x0
r2 0x0
r3 0x0
r4 0x0
r5 0x0
r6 0xec1ba
r7 0xf0
r8 0xd248a
r9 0x0
r10 0x692680
fp 0x6a0fcc
ip 0x8
sp 0x7ea1fb94
lr 0x7409c
pc 0xa8fa0
cpsr 0xa0000010
fault 0x0


SIGQUIT: quit
PC=0xa8fa4 m=2

goroutine 0 [idle]:
runtime.futex(0x6924cc, 0x0, 0x0, 0x76a4bd74, 0x0, 0x0, 0x55cdc, 0x3c, 0x0, 0x76a4bd74, ...)
/usr/local/go/src/runtime/sys_linux_arm.s:247 +0x20 fp=0x76a4bd54 sp=0x76a4bd54
runtime.futexsleep(0x6924cc, 0x0, 0xf8475800, 0xd)
/usr/local/go/src/runtime/os_linux.go:62 +0xf8 fp=0x76a4bd80 sp=0x76a4bd54
runtime.notetsleep_internal(0x6924cc, 0xf8475800, 0xd, 0x109005a0)
/usr/local/go/src/runtime/lock_futex.go:174 +0x150 fp=0x76a4bda0 sp=0x76a4bd80
runtime.notetsleep(0x6924cc, 0xf8475800, 0xd, 0x1f01)
/usr/local/go/src/runtime/lock_futex.go:194 +0x68 fp=0x76a4bdb4 sp=0x76a4bda0
runtime.sysmon()
/usr/local/go/src/runtime/proc.go:3618 +0x688 fp=0x76a4be10 sp=0x76a4bdb4
runtime.mstart1()
/usr/local/go/src/runtime/proc.go:1126 +0x100 fp=0x76a4be20 sp=0x76a4be10
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1096 +0x84 fp=0x76a4be28 sp=0x76a4be20
trap 0x0
error 0x0
oldmask 0x0
r0 0xfffffffc
r1 0x0
r2 0x0
r3 0x76a4bd74
r4 0x0
r5 0x0
r6 0xffffffff
r7 0xf0
r8 0x0
r9 0x76efd6d0
r10 0x109005a0
fp 0x678ac4
ip 0x3c
sp 0x76a4bd54
lr 0x7412c
pc 0xa8fa4
cpsr 0x20000010
fault 0x0


SIGQUIT: quit
PC=0xa8fa0 m=3

goroutine 0 [idle]:
runtime.futex(0x109225ac, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55b98, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/runtime/sys_linux_arm.s:246 +0x1c fp=0x7624bd04 sp=0x7624bd04
runtime.futexsleep(0x109225ac, 0x0, 0xffffffff, 0xffffffff)
/usr/local/go/src/runtime/os_linux.go:45 +0x68 fp=0x7624bd30 sp=0x7624bd04
runtime.notesleep(0x109225ac)
/usr/local/go/src/runtime/lock_futex.go:145 +0xa4 fp=0x7624bd48 sp=0x7624bd30
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1594 +0x100 fp=0x7624bd58 sp=0x7624bd48
runtime.findrunnable(0x10916a00, 0x0)
/usr/local/go/src/runtime/proc.go:2021 +0x758 fp=0x7624bdd8 sp=0x7624bd58
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2120 +0x26c fp=0x7624bdf4 sp=0x7624bdd8
runtime.park_m(0x109014a0)
/usr/local/go/src/runtime/proc.go:2183 +0x158 fp=0x7624be08 sp=0x7624bdf4
runtime.mcall(0x10916000)
/usr/local/go/src/runtime/asm_arm.s:183 +0x5c fp=0x7624be10 sp=0x7624be08
trap 0x0
error 0x0
oldmask 0x0
r0 0x109225ac
r1 0x0
r2 0x0
r3 0x0
r4 0x0
r5 0x0
r6 0xec1ba
r7 0xf0
r8 0xd248a
r9 0x76efd6d0
r10 0x10900780
fp 0x6a0fcc
ip 0x8
sp 0x7624bd04
lr 0x7409c
pc 0xa8fa0
cpsr 0xa0000010
fault 0x0


SIGQUIT: quit
PC=0xa8fa0 m=5

goroutine 0 [idle]:
runtime.futex(0x1092282c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55b98, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/runtime/sys_linux_arm.s:246 +0x1c fp=0x758fed04 sp=0x758fed04
runtime.futexsleep(0x1092282c, 0x0, 0xffffffff, 0xffffffff)
/usr/local/go/src/runtime/os_linux.go:45 +0x68 fp=0x758fed30 sp=0x758fed04
runtime.notesleep(0x1092282c)
/usr/local/go/src/runtime/lock_futex.go:145 +0xa4 fp=0x758fed48 sp=0x758fed30
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1594 +0x100 fp=0x758fed58 sp=0x758fed48
runtime.findrunnable(0x10916a00, 0x0)
/usr/local/go/src/runtime/proc.go:2021 +0x758 fp=0x758fedd8 sp=0x758fed58
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2120 +0x26c fp=0x758fedf4 sp=0x758fedd8
runtime.park_m(0x11ca5950)
/usr/local/go/src/runtime/proc.go:2183 +0x158 fp=0x758fee08 sp=0x758fedf4
runtime.mcall(0x10916000)
/usr/local/go/src/runtime/asm_arm.s:183 +0x5c fp=0x758fee10 sp=0x758fee08
trap 0x0
error 0x0
oldmask 0x0
r0 0x1092282c
r1 0x0
r2 0x0
r3 0x0
r4 0x0
r5 0x0
r6 0xec1ba
r7 0xf0
r8 0xd248a
r9 0x76efd6d0
r10 0x10900b40
fp 0x6a0fcc
ip 0x8
sp 0x758fed04
lr 0x7409c
pc 0xa8fa0
cpsr 0xa0000010
fault 0x0


SIGQUIT: quit
PC=0xa8fa0 m=4

goroutine 0 [idle]:
runtime.futex(0x1094e0ac, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55b98, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/runtime/sys_linux_arm.s:246 +0x1c fp=0x750fed04 sp=0x750fed04
runtime.futexsleep(0x1094e0ac, 0x0, 0xffffffff, 0xffffffff)
/usr/local/go/src/runtime/os_linux.go:45 +0x68 fp=0x750fed30 sp=0x750fed04
runtime.notesleep(0x1094e0ac)
/usr/local/go/src/runtime/lock_futex.go:145 +0xa4 fp=0x750fed48 sp=0x750fed30
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1594 +0x100 fp=0x750fed58 sp=0x750fed48
runtime.findrunnable(0x10916000, 0x0)
/usr/local/go/src/runtime/proc.go:2021 +0x758 fp=0x750fedd8 sp=0x750fed58
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2120 +0x26c fp=0x750fedf4 sp=0x750fedd8
runtime.park_m(0x11ca5950)
/usr/local/go/src/runtime/proc.go:2183 +0x158 fp=0x750fee08 sp=0x750fedf4
runtime.mcall(0x10917400)
/usr/local/go/src/runtime/asm_arm.s:183 +0x5c fp=0x750fee10 sp=0x750fee08
trap 0x0
error 0x0
oldmask 0x0
r0 0x1094e0ac
r1 0x0
r2 0x0
r3 0x0
r4 0x0
r5 0x0
r6 0xec1ba
r7 0xf0
r8 0xd248a
r9 0x7624c920
r10 0x109500f0
fp 0x6a0fcc
ip 0x8
sp 0x750fed04
lr 0x7409c
pc 0xa8fa0
cpsr 0xa0000010
fault 0x0


@shahinv

This comment has been minimized.

shahinv commented Nov 25, 2016

@davecheney @quentinmit Is there anything which I can do to help investigation?

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 25, 2016

@shahinv is the source of your program available for me to try to reproduce the problem?

@shahinv

This comment has been minimized.

shahinv commented Nov 25, 2016

@davecheney Unfortunately the program is large (http://www.nanodlp.com) and it is hard to reproduce the issue (require couple of hours), I will try isolate the issue to share easier to debug source code ASAP.

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 25, 2016

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 25, 2016

@minux

This comment has been minimized.

Member

minux commented Nov 25, 2016

@shahinv

This comment has been minimized.

shahinv commented Nov 25, 2016

@davecheney I am checking the program for race frequently, as it slow down the program usually I test it for under an hour. If more could be helpful to detect race, please let me know.

@minux I have seen couple of completely different sets of behaviours as I have forced to change framework and rewrite codes to make sure none of them is the issue. Currently I have seen issue with two tight loops with sleep inside.

  1. The program collecting serial output from external devices, it is working fine. But to synchronize jobs I am searching output to see if given string is available or not to move to the next job. Sleep time and time waited correlated with crashes. Before each read and writes I was using Mutex lock. Even removing locks does not yield any difference. But increasing sleep time (100,000,000) or faster response from external device helps but never solve the issue.

Number of OS thread could increase up to 150, Also number of goroutines (runtime.NumGoroutine()) increases gradually also I have seen sporadic high cpu usage until time of crash, usually it crashes on around 800 goroutines.

Removing this loop, fix the issue. As a workaround instead of watching for string on loop I lock the main module using Mutex lock, code which read output from serial directly searching for string and unlock the main module.

  1. Second place which helped me solve the issue as first issue was not easily reproducible for me, I had loop similar to one below.

for !LayerExist(){
time.Sleep(100000000)
}

func LayerExist() bool {
_, err := os.Stat("file")
if err != nil {
return false
}
return true
}

Number of OS thread remains 9-15 (which is normal), but number of goroutines (runtime.NumGoroutine()) increases gradually also I have seen sporadic high cpu usage until time of crash, usually it crashes on around 800 goroutines.

Reproducing this one is much easier as making sure file does not exist and it will crashes the program sooner or later.
photo_2016-11-25_12-52-20

Removing nice completely fix the issues. Usual processor usage should not be higher than 0.5, IO usage also very low. Removing nice caused another issue, as the program directly generate pulses to move a stepper motor. Rate of uneven steps increased substantially. You can see what I am talking about on this video (During 40~50s) https://www.youtube.com/watch?v=uybLNu0zv28&list=PLP9pvG98FyFs8Jz131i5iXVoY0O9QIQyF&index=2

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 25, 2016

@shahinv

This comment has been minimized.

shahinv commented Nov 26, 2016

It uses (github.com/tarm/serial) to communicate with serial devices. I do not think data collection is the issue it somewhere which we check its result, when they off this check everything working fine.

Also the second case which I have described above happens with serial communication totally off. I have isolated issues and in the both case simple loop with wait inside causing the issue, unfortunately when I move these loops somewhere to reproduce the issue, it does not happen.

Lots of different SLA 3D printers being supported by the software, naturally it becomes complex.
Users could use different hardwares for the movements. RAMPS (through Serial), Trinamics drivers and etc (Serial or i2c), Directly generating pulse on RPi GPIO.
Even for a basic printer usually following functionality being used. Serial con to RAMPS, Serial con to a Projector, CGO call to DispmanX, Generating PWM (Through DMA) for shutter (right now I use pigpio), GPIO for external buttons, fault detection and etc.

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 26, 2016

@shahinv

This comment has been minimized.

shahinv commented Nov 26, 2016

Unfortunately extracted code does not cause the issue.
About nice -20, when generating pulses duration of pulses must be even to have smooth movement.
You can hear some strange sounds between 40~50s. It is sound of uneven steps. https://www.youtube.com/watch?v=uybLNu0zv28&list=PLP9pvG98FyFs8Jz131i5iXVoY0O9QIQyF&index=2

Having -20 helps to achieve better pulses. Even with nice -20 it is limited to 2KHz with golang.
http://www.nanodlp.com/forum/uploads/d71e78db4925cdb08f8635312226312e.png

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 27, 2016

@shahinv

This comment has been minimized.

shahinv commented Nov 28, 2016

@davecheney thank you for putting time into solving this issue.

I have found workarounds for the issues before filling bug report. But it would be awesome if the source of the problem could be solved even if not, this thread could help fellow programmers with same issue to avoid nicing.

As the issue hardly reported on go1.6, I will try go1.8 soon to see if new SSA helps.

@aclements

This comment has been minimized.

Member

aclements commented Dec 13, 2016

Getting too late for 1.8, especially since there's a workaround.

@aclements aclements added this to the Go1.9 milestone Dec 13, 2016

@aclements aclements removed this from the Go1.8 milestone Dec 13, 2016

@bradfitz bradfitz removed the WaitingForInfo label Jan 4, 2017

@aclements aclements modified the milestones: Go1.10, Go1.9 Jun 9, 2017

@rsc rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

@gopherbot gopherbot modified the milestones: Go1.11, Unplanned May 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment