fatal error: runtime: out of memory

[toast-uz]

Encountered out of memory on paho.mqtt.golang version July 14th, while no problem on version June 18th in my exactly same test environment making thousands of concurrent mqtt connections. As for vmstat, free memory decrease significantly only on version July 14th. Any degraded between June 18th - July 14th ?

------------------- error output
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x694e44, 0x16)
/usr/local/go/src/runtime/panic.go:596 +0x95
runtime.sysMap(0xc44fa70000, 0x110000, 0x0, 0x8543d8)
/usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0x83b8c0, 0x110000, 0x0)
/usr/local/go/src/runtime/malloc.go:428 +0x374
runtime.(*mheap).grow(0x83b8c0, 0x81, 0x0)
/usr/local/go/src/runtime/mheap.go:774 +0x62
runtime.(*mheap).allocSpanLocked(0x83b8c0, 0x81, 0x42c964)
/usr/local/go/src/runtime/mheap.go:678 +0x44f
runtime.(*mheap).alloc_m(0x83b8c0, 0x81, 0x100000000, 0x451f01)
/usr/local/go/src/runtime/mheap.go:562 +0xe2
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:627 +0x4b
runtime.systemstack(0x7fbddef87dd8)
/usr/local/go/src/runtime/asm_amd64.s:343 +0xab
runtime.(*mheap).alloc(0x83b8c0, 0x81, 0x10100000000, 0xc42001e000)
/usr/local/go/src/runtime/mheap.go:628 +0xa0
runtime.largeAlloc(0x100318, 0xc420001a01, 0x4)
/usr/local/go/src/runtime/malloc.go:795 +0x93
runtime.mallocgc.func1()
/usr/local/go/src/runtime/malloc.go:690 +0x3e
runtime.systemstack(0xc42001e000)
/usr/local/go/src/runtime/asm_amd64.s:327 +0x79
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1132

goroutine 9 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:281 fp=0xc4287c9d88 sp=0xc4287c9d80
runtime.mallocgc(0x100318, 0x686240, 0x69fc01, 0xc44ee82f00)
/usr/local/go/src/runtime/malloc.go:691 +0x930 fp=0xc4287c9e28 sp=0xc4287c9d88
runtime.newobject(0x686240, 0x0)
/usr/local/go/src/runtime/malloc.go:808 +0x38 fp=0xc4287c9e58 sp=0xc4287c9e28
github.com/eclipse/paho%2emqtt%2egolang.NewClient(0xc44f7e0500, 0x7, 0xc4288c9f98)
/home/centos/projects/src/src/github.com/eclipse/paho.mqtt.golang/client.go:93 +0x58 fp=0xc4288c9ed8 sp=0xc4287c9e58
main.Connect.func1(0xc42005a1e0, 0xc4200111c0, 0x3)

--------------- my code of connecting
func Connect() {
fmt.Println("Start Connecting.")
conns = make([]MQTT.Client, optmaxConns)
ch := make(chan int)
go Count(ch, 1000)
var wg sync.WaitGroup
for i := 0; i < optnumThreads; i++ {
wg.Add(1)
go func(i int) {
for j := 0; j <= optmaxConns / optnumThreads; j++ {
n := j * optnumThreads + i
if n >= optmaxConns {
break
}
opts := MQTT.NewClientOptions().AddBroker(optTarget)
opts.SetClientID(fmt.Sprintf("%s/%07d", optIDprefix, n))
conns[n] = MQTT.NewClient(opts)
if token := conns[n].Connect(); token.Wait() && token.Error() != nil {
fmt.Println("Error connect: conss", n)
if token := conns[n].Connect(); token.Wait() && token.Error() != nil {
panic(token.Error())
}
}
ch <- 1
}
wg.Done()
} (i)
}
wg.Wait()
ch <- 0 // release Count goroutine
fmt.Println("End Connecting.")
}

[alsm]

The only change I can see is that I preallocate a [65535]Token rather than using a map[uint16]Token I'm not sure how much space the new definition uses but it will definitely be more than the map. I did it to try and resolve an issue with nil lookups for message Ids, if you're opening thousands of connections (creating thousands of clients) I imagine this is related. I'm not sure I want to change it back, but I'm open to other suggestions for tracking the messageIDs in use.

[toast-uz]

FYI:
New version ate 2.3GB memory for 1k connections, while old version ate about 2.6GB memory for 50k connections (0.05GB for 1k). The new has about 50 times appetite than old.

[ghost]

@alsm , FYI:

You had done the change to preallocate [65535]Token along with the change to log missing messageId in #115 (in the quest to hopefully minimize/prevent it). I've been running builds with this change and I've seen the log entry (despite the preallocation) for messageIds > 0. So, IMO, you may revert back to map[uint16]Token.

[alsm]

As suggested given that this change didn't help with the missing tokens issue and caused the memory usage to balloon I have reverted the changes.