Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

race condition in gopher-lua: fatal error: concurrent map writes #7

Closed
mischief opened this issue Oct 1, 2016 · 6 comments
Closed

race condition in gopher-lua: fatal error: concurrent map writes #7

mischief opened this issue Oct 1, 2016 · 6 comments

Comments

@mischief
Copy link
Contributor

mischief commented Oct 1, 2016

it seems like maybe the lua function "handle" is missing locking, or doesn't use a seperate lua instance from the state cache.

time="2016-10-01T09:03:34Z" level=info msg="Serving HTTP/2 on :443" 
time="2016-10-01T09:03:34Z" level=info msg="Serving HTTP on :80" 
time="2016-10-01T09:03:34Z" level=error msg="open /etc/algernon/cert.pem: no such file or directory" 
fatal error: concurrent map writes

goroutine 20 [running]:
runtime.throw(0x557a43505e40, 0x15)
    /usr/lib/go/src/runtime/panic.go:547 +0x90 fp=0xc8203fb398 sp=0xc8203fb380
runtime.mapassign1(0x557a432474c0, 0xc820159170, 0xc8203fb5f0, 0xc8203fb5e0)
    /usr/lib/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc8203fb440 sp=0xc8203fb398
github.com/yuin/gopher-lua.(*LState).setFieldString(0xc820077580, 0x7f83a8b460f8, 0xc82005b3e0, 0x557a43482ad0, 0x5, 0x7f83a8b46068, 0xc820406e40)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/yuin/gopher-lua/state.go:980 +0xb0e fp=0xc8203fb648 sp=0xc8203fb440
github.com/yuin/gopher-lua.(*LState).SetField(0xc820077580, 0x7f83a8b460f8, 0xc82005b3e0, 0x557a43482ad0, 0x5, 0x7f83a8b46068, 0xc820406e40)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/yuin/gopher-lua/state.go:1504 +0x5d fp=0xc8203fb688 sp=0xc8203fb648
github.com/yuin/gopher-lua.(*LState).SetGlobal(0xc820077580, 0x557a43482ad0, 0x5, 0x7f83a8b46068, 0xc820406e40)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/yuin/gopher-lua/state.go:1520 +0x7e fp=0xc8203fb6d8 sp=0xc8203fb688
main.exportBasicWeb(0x7f83a8b48260, 0xc82019add0, 0xc82042c700, 0xc820077580, 0x7ffc3001a24b, 0x1c, 0x0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/basic.go:129 +0x370 fp=0xc8203fb908 sp=0xc8203fb6d8
main.exportCommonFunctions(0x7f83a8b48260, 0xc82019add0, 0xc82042c700, 0x7ffc3001a24b, 0x1c, 0x7f83a8b41dd0, 0xc82005b2c0, 0xc820077580, 0xc8201c3b00, 0x0, ...)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/lua.go:247 +0x73 fp=0xc8203fb980 sp=0xc8203fb908
main.exportLuaHandlerFunctions.func1.1(0x7f83a8b48260, 0xc82019add0, 0xc82042c700)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/luahandler.go:23 +0xe4 fp=0xc8203fbae8 sp=0xc8203fb980
net/http.HandlerFunc.ServeHTTP(0xc820325380, 0x7f83a8b48260, 0xc82019add0, 0xc82042c700)
    /usr/lib/go/src/net/http/server.go:1618 +0x3a fp=0xc8203fbb08 sp=0xc8203fbae8
net/http.(*ServeMux).ServeHTTP(0xc820158bd0, 0x7f83a8b48260, 0xc82019add0, 0xc82042c700)
    /usr/lib/go/src/net/http/server.go:1910 +0x17d fp=0xc8203fbb60 sp=0xc8203fbb08
net/http.serverHandler.ServeHTTP(0xc82036c180, 0x7f83a8b48260, 0xc82019add0, 0xc82042c700)
    /usr/lib/go/src/net/http/server.go:2081 +0x19e fp=0xc8203fbbc0 sp=0xc8203fbb60
net/http.(*conn).serve(0xc82036cd80)
    /usr/lib/go/src/net/http/server.go:1472 +0xf2e fp=0xc8203fbf88 sp=0xc8203fbbc0
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8203fbf90 sp=0xc8203fbf88
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2137 +0x44e

goroutine 1 [chan send]:
main.serve(0xc820357960, 0xc820158bd0, 0xc820325440, 0xc8203253e0, 0x0, 0x0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/serve.go:184 +0x424
main.main()
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/main.go:354 +0x2df7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
    /usr/lib/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/lib/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [chan receive]:
github.com/bobappleyard/readline.handleSignals()
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/bobappleyard/readline/readline.go:301 +0x176
created by github.com/bobappleyard/readline.init.1
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/bobappleyard/readline/readline.go:314 +0x4a

goroutine 8 [chan receive]:
main.serve.func1(0xc8203254a0, 0xc820357960, 0xc820158bd0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/serve.go:121 +0x45
created by main.serve
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/serve.go:129 +0x101

goroutine 18 [select, locked to thread]:
runtime.gopark(0x557a435de028, 0xc82001e728, 0x557a43486180, 0x6, 0x18, 0x2)
    /usr/lib/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc82001e728, 0x0, 0x18)
    /usr/lib/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc82001e728)
    /usr/lib/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/lib/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 10 [IO wait]:
net.runtime_pollWait(0x7f83a8b48048, 0x72, 0x0)
    /usr/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820357bf0, 0x72, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820357bf0, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820357b90, 0x0, 0x7f83a8b48108, 0xc8203270e0)
    /usr/lib/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820026300, 0xc820037cd0, 0x0, 0x0)
    /usr/lib/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820026300, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc82036c180, 0x7f83a8b470c8, 0xc820026300, 0x0, 0x0)
    /usr/lib/go/src/net/http/server.go:2117 +0x129
github.com/tylerb/graceful.(*Server).Serve(0xc82036c200, 0x7f83a8b470c8, 0xc820026300, 0x0, 0x0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/tylerb/graceful/graceful.go:256 +0x53f
github.com/tylerb/graceful.(*Server).ListenAndServe(0xc82036c200, 0x0, 0x0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/tylerb/graceful/graceful.go:135 +0xdf
main.serve.func3(0xc820158bd0, 0xc820357960)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/serve.go:148 +0xa6
created by main.serve
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/serve.go:152 +0x3eb

goroutine 12 [select]:
github.com/tylerb/graceful.(*Server).manageConnections(0xc82036c200, 0xc820325560, 0xc8203255c0, 0xc820325620, 0xc820325680)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/tylerb/graceful/graceful.go:313 +0x4fb
created by github.com/tylerb/graceful.(*Server).Serve
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/tylerb/graceful/graceful.go:244 +0x2db

goroutine 13 [chan receive]:
github.com/tylerb/graceful.(*Server).handleInterrupt(0xc82036c200, 0xc8203256e0, 0xc820325740, 0x7f83a8b470c8, 0xc820026300)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/tylerb/graceful/graceful.go:350 +0x64
created by github.com/tylerb/graceful.(*Server).Serve
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/tylerb/graceful/graceful.go:252 +0x511

goroutine 37 [IO wait]:
net.runtime_pollWait(0x7f83a8b47e08, 0x72, 0xc820367000)
    /usr/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8204a2680, 0x72, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8204a2680, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8204a2620, 0xc820367000, 0x1000, 0x1000, 0x0, 0x7f83a8b39050, 0xc820018140)
    /usr/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8203ee138, 0xc820367000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/net.go:172 +0xe4
net/http.(*connReader).Read(0xc820423e00, 0xc820367000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/http/server.go:526 +0x196
bufio.(*Reader).fill(0xc8203257a0)
    /usr/lib/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc8203257a0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc8203257a0, 0x0, 0x0, 0x0, 0x557a43449c00, 0x0, 0x0)
    /usr/lib/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc82017dfb0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc82017dfb0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/textproto/reader.go:36 +0x40
net/http.readRequest(0xc8203257a0, 0x557a435dda00, 0xc82042c460, 0x0, 0x0)
    /usr/lib/go/src/net/http/request.go:721 +0xb6
net/http.(*conn).readRequest(0xc820391300, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/http/server.go:705 +0x359
net/http.(*conn).serve(0xc820391300)
    /usr/lib/go/src/net/http/server.go:1425 +0x947
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2137 +0x44e

goroutine 19 [runnable]:
github.com/yuin/gopher-lua.(*LState).setFieldString(0xc820077580, 0x7f83a8b460f8, 0xc82005b3e0, 0x557a434864a0, 0x5, 0x7f83a8b46068, 0xc82017a8c0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/yuin/gopher-lua/state.go:980 +0xb0e
github.com/yuin/gopher-lua.(*LState).SetField(0xc820077580, 0x7f83a8b460f8, 0xc82005b3e0, 0x557a434864a0, 0x5, 0x7f83a8b46068, 0xc82017a8c0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/yuin/gopher-lua/state.go:1504 +0x5d
github.com/yuin/gopher-lua.(*LState).SetGlobal(0xc820077580, 0x557a434864a0, 0x5, 0x7f83a8b46068, 0xc82017a8c0)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/yuin/gopher-lua/state.go:1520 +0x7e
main.exportBasicSystemFunctions(0xc820077580)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/basic.go:59 +0x6b3
main.exportCommonFunctions(0x7f83a8b48260, 0xc8200d2340, 0xc82042c540, 0x7ffc3001a24b, 0x1c, 0x7f83a8b41dd0, 0xc82005b2c0, 0xc820077580, 0xc8201c3b00, 0x0, ...)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/lua.go:250 +0x84
main.exportLuaHandlerFunctions.func1.1(0x7f83a8b48260, 0xc8200d2340, 0xc82042c540)
    /var/tmp/portage/www-servers/algernon-1.0/work/algernon-1.0/src/github.com/xyproto/algernon/luahandler.go:23 +0xe4
net/http.HandlerFunc.ServeHTTP(0xc820325380, 0x7f83a8b48260, 0xc8200d2340, 0xc82042c540)
    /usr/lib/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc820158bd0, 0x7f83a8b48260, 0xc8200d2340, 0xc82042c540)
    /usr/lib/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc82036c180, 0x7f83a8b48260, 0xc8200d2340, 0xc82042c540)
    /usr/lib/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820076100)
    /usr/lib/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2137 +0x44e
@xyproto
Copy link
Owner

xyproto commented Oct 9, 2016

Added commit e89ea7e that might perhaps fix this.

Please confirm that it works now. If not, please add steps to reproduce the issue and I'll look into it.

Thanks!

@mischief
Copy link
Contributor Author

@xyproto here is how i can produce this.

server.lua:

h = function()
    print("Hello, world!")
end

handle("/", h)
algernon --server --nolimit --addr=:3001 server.lua

using ab to do some concurrent requests:

ab -c 10 -n 10000 http://127.0.0.1:3001/

@xyproto
Copy link
Owner

xyproto commented Oct 12, 2016

Perfect, thanks! I'm able to reproduce the issue. Will fix.

xyproto added a commit that referenced this issue Oct 12, 2016
@xyproto
Add test case for issue #7
5046972
xyproto added a commit that referenced this issue Oct 12, 2016
@xyproto
Add test case for issue #7
31f32ac
xyproto added a commit that referenced this issue Oct 12, 2016
@xyproto
Add mutex for the Lua handler
a891d03 
Should solve issue #7
@xyproto
Copy link
Owner

xyproto commented Oct 12, 2016

Added a mutex. Using channels somehow might be more elegant, but this should fix the issue for now.

Please confirm that it now works as expected.

@mischief
Copy link
Contributor Author

yes, the runtime issue is fixed, but the mutex is unfortunate and will hinder throughput, but at least it works now. :-D

@xyproto xyproto mentioned this issue Oct 12, 2016
Closed
@xyproto xyproto closed this as completed Oct 12, 2016
@xyproto
Copy link
Owner

xyproto commented Oct 13, 2016

Moved the mutex so that it is only locking when exporting common Lua functions to the Lua state object, not when actually executing the Lua function. I think this should improve the speed, and all tests (and the benchmarking script) all pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xyproto @mischief