bug in concurrent map write with latest master

[vtolstov]

goroutine 114 [running]:

runtime.throw(0x13823a4, 0x26)

 /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc000393cc8 sp=0xc000393c98 pc=0x445332

runtime.mapiternext(0xc000393e30)

 /usr/local/go/src/runtime/map.go:853 +0x554 fp=0xc000393d48 sp=0xc000393cc8 pc=0x41e514

runtime.mapiterinit(0x116e0e0, 0xc00058c420, 0xc000393e30)

 /usr/local/go/src/runtime/map.go:843 +0x1c5 fp=0xc000393d68 sp=0xc000393d48 pc=0x41dec5

github.com/twmb/franz-go/pkg/kgo.(*groupConsumer).getUncommittedLocked(0xc00018e380, 0x1, 0x1366fb9)

 github.com/twmb/franz-go/pkg/kgo/consumer_group.go:1709 +0x495 fp=0xc000393f00 sp=0xc000393d68 pc=0xa8a335

github.com/twmb/franz-go/pkg/kgo.(*groupConsumer).loopCommit(0xc00018e380)

github.com/twmb/franz-go/pkg/kgo/consumer_group.go:1571 +0x291 fp=0xc000393fd8 sp=0xc000393f00 pc=0xa89251

runtime.goexit()

 /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000393fe0 sp=0xc000393fd8 pc=0x47d9c1

created by github.com/twmb/franz-go/pkg/kgo.(*consumer).initGroup

 github.com/twmb/franz-go/pkg/kgo/consumer_group.go:207 +0x708

[vtolstov]

i think that this related to latest changes for MarkCommitRecords

[vtolstov]

i'm check code and dont see any errors getUncommittedLocked always protected via g.mu.Lock/g.mu.Unlock

@twmb can you check?

[vtolstov]

another trace

fatal error: concurrent map writes

fatal error: concurrent map writes


 

goroutine 439 [running]:

runtime.throw(0x136e55b, 0x15)

 /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc000727db0 sp=0xc000727d80 pc=0x445332

runtime.mapassign_fast32(0x116e0e0, 0xc0001c4e10, 0xc00000000f, 0xc0001c05c8)

 /usr/local/go/src/runtime/map_fast32.go:176 +0x325 fp=0xc000727df0 sp=0xc000727db0 pc=0x41f725

github.com/twmb/franz-go/pkg/kgo.(*Client).MarkCommitRecords(0xc00016ea00, 0xc0063db020, 0x1, 0x1)

github.com/twmb/franz-go/pkg/kgo/consumer_group.go:1850 +0x151 fp=0xc000727ed0 sp=0xc000727df0 pc=0xa8a791

some.ru/service-platform/framework/framework-modules/broker-kgo.(*subscriber).handleFetches.func1(0x1, 0x0)

some.ru/service-platform/framework/framework-modules/broker-kgo/kgo.go:448 +0x2b3 fp=0xc000727f78 sp=0xc000727ed0 pc=0xadf493

golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0001c6de0, 0xc00491d580)

golang.org/x/sync/errgroup/errgroup.go:57 +0x59 fp=0xc000727fd0 sp=0xc000727f78 pc=0xada039

runtime.goexit()

 /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000727fd8 sp=0xc000727fd0 pc=0x47d9c1

created by golang.org/x/sync/errgroup.(*Group).Go

golang.org/x/sync/errgroup/errgroup.go:54 +0x66

[twmb]

Good find! I've merged your PR, that lock is critical, I'm surprised I missed this. I'll tag this directly to 1.0 quite soon.

[vtolstov]

thanks, can you wait 1-2 days? i'm profile my app to check differences with github.com/segmention/kafka-go performance
may be i can find some interesting stuff or have some questions, ok ?

[twmb]

Hopefully nothing surprising at this point, I've benchmarked differences myself with the bench example. I was planning to tag in an hour, I can wait until tomorrow or at the latest (hopefully), Monday.