god_server panics due to invalid memory referencing on windows

[shivakumargn]

After changing god server's dir name from ip:port to ip_port, the server panics as below:

D:\sources\gopath\src\github.com\zond\god\god_server>god_server
panic: runtime error: invalid memory address or nil pointer dereference
panic: sync: unlock of unlocked mutex
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x49ce6b]

goroutine 1 [running]:
sync.(_Mutex).Unlock(0x12c87f60)
D:/sources/golang/src/pkg/sync/mutex.go:91 +0x79
sync.(_RWMutex).Unlock(0x12c87f60)
D:/sources/golang/src/pkg/sync/rwmutex.go:113 +0x77
sync/atomic.StoreUint64()
D:/sources/golang/src/pkg/sync/atomic/asm_386.s:147 +0xb
github.com/zond/god/dhash.func-+016(0x12d0cc40, 0x12d0cc60)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:104 +0x9b
github.com/zond/god/common.(_Ring).sendChanges(0x12c87f40, 0x12c5bc00, 0x10, 0x10)
D:/sources/gopath/src/github.com/zond/god/common/ring.go:103 +0x141
github.com/zond/god/common.(_Ring).Add(0x12c87f40, 0x12c5bb80, 0x10, 0x10, 0x12c5b270, ...)
D:/sources/gopath/src/github.com/zond/god/common/ring.go:182 +0x49f
github.com/zond/god/discord.(_Node).Start(0x12c5cf40, 0x0, 0x0)
D:/sources/gopath/src/github.com/zond/god/discord/node.go:208 +0x36b
github.com/zond/god/dhash.(_Node).Start(0x12c61840, 0x0, 0x0)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:198 +0xcd
github.com/zond/god/dhash.(*Node).MustStart(0x12c61840, 0xe)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:393 +0x28
main.main()
D:/sources/gopath/src/github.com/zond/god/god_server/god_server.go:48 +0x333

goroutine 2 [syscall]:

goroutine 17 [select]:
github.com/zond/god/persistence.(_Logger).record(0x12c5e390, 0x12d0d240)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:402 +0x414
created by github.com/zond/god/persistence.(_Logger).Record
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:382 +0x10d

goroutine 18 [syscall]:
syscall.Syscall6(0x7549d3ab, 0x5, 0xd0, 0x12c5bcb0, 0x12bf0a98, ...)
D:/sources/golang/src/pkg/runtime/zsyscall_windows_windows_386.c:97 +0x49
syscall.GetQueuedCompletionStatus(0xd0, 0x12c5bcb0, 0x12bf0a98, 0x12bf0a90, 0xffffffff, ...)
D:/sources/golang/src/pkg/syscall/zsyscall_windows_386.go:507 +0x7e
net.(*resultSrv).Run(0x12bf09a0)
D:/sources/golang/src/pkg/net/fd_windows.go:150 +0x11a
created by net.startServer
D:/sources/golang/src/pkg/net/fd_windows.go:285 +0xde

[zond]

This was really odd.

The line of code breaking is

atomic.StoreInt64(&result.lastReroute, time.Now().UnixNano())

which doesn't even use Mutexes.

Your Go version seems broken? What version are you running?

[shivakumargn]

I am on tip (currently go1.1rc2).

Problem is consistent with 32-bit build of Go on Win7.
With 64-bit build of Go there is no issue.

I rebuilt Go and tried with same result. Am sure there is no issue with Go version.

Stack trace showing Mutex is something I do not have an answer to right now. Does the sync/atomic achieve atomicity without "mutex" ?

[zond]

Have you tried it with 1.0.x?

Really strange that it happens in 32 but not 64 bits.

sync/atomic ought to work without mutexes, it should do its stuff using http://en.wikipedia.org/wiki/Compare-and-swap

[shivakumargn]

Tried with go1.0.3 (32bit), there is no crash. Seems to be a go1.1 issue. I will post on go-nuts list.

[shivakumargn]

With latest go1.1rc3 the panic stack is as below:

D:\sources\gopath\bin>god_server.exe
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x4a924b]

goroutine 1 [running]:
sync/atomic.StoreUint64()
D:/sources/golang/src/pkg/sync/atomic/asm_386.s:147 +0xb
github.com/zond/god/common.(_TimeLock).Lock(0x12b83900)
D:/sources/gopath/src/github.com/zond/god/common/time_lock.go:28 +0xba
github.com/zond/god/radix.(_Tree).Put(0x12b6ddb0, 0x12b6be20, 0xa, 0xa, 0x12b6be30, ...)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:591 +0x53
github.com/zond/god/radix.func-+006(0x12b6be20, 0xa, 0xa, 0x0, 0x0, ...)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:240 +0xd2
github.com/zond/god/persistence.(_logfile).play(0x12c2c3c0, 0x2716d4)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:90 +0x10b
github.com/zond/god/persistence.(_Logger).Play(0x12c1f030, 0x2716d4)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:237 +0xe6
github.com/zond/god/radix.(*Tree).Restore(0x12b6ddb0, 0x12b6b200)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:259 +0x59
github.com/zond/god/dhash.NewNodeDir(0x12b6b210, 0xe, 0x12b6b230, 0xe, 0x12b6b200, ...)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:110 +0x26b
main.main()
D:/sources/gopath/src/github.com/zond/god/god_server/god_server.go:29 +0x29d

[zond]

Great!

On Fri, May 10, 2013 at 5:13 AM, sgn notifications@github.com wrote:

Tried with go1.0.3 (32bit), there is no crash. Seems to be a go1.1 issue.
I will post on go-nuts list.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-17702273
.

[zond]

This was very odd as well.

The line in question

28 atomic.StoreInt64(&self.locktimes[self.index], time.Now().UnixNano())

is referencing the locktimes of self, which is an array (not a slice, so it
can't be un-initialized). Therefore, how can it ever be nil?

The fact that this does not panic at all in 1.0.3 also indicates a bug in
Go 1.1. Would you post that bug as well to the go-nuts list?

On Fri, May 10, 2013 at 5:23 AM, sgn notifications@github.com wrote:

With latest go1.1rc3 the panic stack is as below:

D:\sources\gopath\bin>god_server.exe

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x4a924b]

goroutine 1 [running]:

sync/atomic.StoreUint64()
D:/sources/golang/src/pkg/sync/atomic/asm_386.s:147 +0xb
github.com/zond/god/common.(_TimeLock).Lock(0x12b83900)
D:/sources/gopath/src/github.com/zond/god/common/time_lock.go:28 +0xba
github.com/zond/god/radix.(_Tree).Put(0x12b6ddb0, 0x12b6be20, 0xa, 0xa,
0x12b6be30, ...)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:591 +0x53
github.com/zond/god/radix.func-+006(0x12b6be20, 0xa, 0xa, 0x0, 0x0, ...)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:240 +0xd2
github.com/zond/god/persistence.(_logfile).play(0x12c2c3c0, 0x2716d4)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:90+0x10b
github.com/zond/god/persistence.(_Logger).Play(0x12c1f030, 0x2716d4)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:237+0xe6
github.com/zond/god/radix.(*Tree).Restore(0x12b6ddb0, 0x12b6b200)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:259 +0x59
github.com/zond/god/dhash.NewNodeDir(0x12b6b210, 0xe, 0x12b6b230, 0xe,
0x12b6b200, ...)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:110 +0x26b
main.main()
D:/sources/gopath/src/github.com/zond/god/god_server/god_server.go:29+0x29d

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-17702483
.

[zond]

Ah, I just read the lastest digest from go-nuts.

I will see if I can create a patch that solves the alignment issue.

On Fri, May 10, 2013 at 8:26 AM, Martin Bruse zondolfin@gmail.com wrote:

This was very odd as well.

The line in question

28 atomic.StoreInt64(&self.locktimes[self.index], time.Now().UnixNano())

is referencing the locktimes of self, which is an array (not a slice, so
it can't be un-initialized). Therefore, how can it ever be nil?

The fact that this does not panic at all in 1.0.3 also indicates a bug in
Go 1.1. Would you post that bug as well to the go-nuts list?

On Fri, May 10, 2013 at 5:23 AM, sgn notifications@github.com wrote:

With latest go1.1rc3 the panic stack is as below:

D:\sources\gopath\bin>god_server.exe

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x4a924b]

goroutine 1 [running]:

sync/atomic.StoreUint64()
D:/sources/golang/src/pkg/sync/atomic/asm_386.s:147 +0xb
github.com/zond/god/common.(_TimeLock).Lock(0x12b83900)
D:/sources/gopath/src/github.com/zond/god/common/time_lock.go:28 +0xba
github.com/zond/god/radix.(_Tree).Put(0x12b6ddb0, 0x12b6be20, 0xa, 0xa,
0x12b6be30, ...)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:591 +0x53
github.com/zond/god/radix.func-+006(0x12b6be20, 0xa, 0xa, 0x0, 0x0, ...)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:240 +0xd2
github.com/zond/god/persistence.(_logfile).play(0x12c2c3c0, 0x2716d4)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:90+0x10b
github.com/zond/god/persistence.(_Logger).Play(0x12c1f030, 0x2716d4)
D:/sources/gopath/src/github.com/zond/god/persistence/persistence.go:237+0xe6
github.com/zond/god/radix.(*Tree).Restore(0x12b6ddb0, 0x12b6b200)
D:/sources/gopath/src/github.com/zond/god/radix/tree.go:259 +0x59
github.com/zond/god/dhash.NewNodeDir(0x12b6b210, 0xe, 0x12b6b230, 0xe,
0x12b6b200, ...)
D:/sources/gopath/src/github.com/zond/god/dhash/dhash.go:110 +0x26b
main.main()
D:/sources/gopath/src/github.com/zond/god/god_server/god_server.go:29+0x29d

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-17702483
.

[zond]

So, see if that helps. I just moved the arrays and ints in question to be at the very top of the struct they are part of.

[shivakumargn]

Yes, works with this change.

[zond]

Perfect :)