Performance test occasionally fails with fatal error: concurrent map writes

[markmandel]

What happened:

Short version:

fatal error: concurrent map writes

goroutine 144634 [running]:
main.main.func1.1()
	/go/src/agones.dev/agones/test/load/allocation/runscenario/runscenario.go:126 +0xcf
created by main.main.func1 in goroutine 11
	/go/src/agones.dev/agones/test/load/allocation/runscenario/runscenario.go:123 +0x66d

goroutine 1 [semacquire, 48 minutes]:
sync.runtime_Semacquire(0xc000012db0?)
	/usr/local/go/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc0001d8d50?)
	/usr/local/go/src/sync/waitgroup.go:116 +0x48
main.main()
	/go/src/agones.dev/agones/test/load/allocation/runscenario/runscenario.go:137 +0x6f2

goroutine 6 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/local/go/src/runtime/time.go:195 +0x125
main.main.func1(0x0)
	/go/src/agones.dev/agones/test/load/allocation/runscenario/runscenario.go:131 +0x549
created by main.main in goroutine 1
	/go/src/agones.dev/agones/test/load/allocation/runscenario/runscenario.go:109 +0x10e9

See attached log for full details
log-275a3e64-934b-4512-80a4-28ff3ad733c8.txt

What you expected to happen:

The performance test should always pass

How to reproduce it (as minimally and precisely as possible):

Check the logs:
https://console.cloud.google.com/cloud-build/builds;region=global/275a3e64-934b-4512-80a4-28ff3ad733c8;step=6?e=13803378&mods=logs_tg_prod&project=agones-images

Anything else we need to know?:

Environment:

• Agones version: dev
• Kubernetes version (use kubectl version): whatever the perf cluster is.
• Cloud provider or hardware configuration: GKE
• Install method (yaml/helm):
• Troubleshooting guide log(s):
• Others:

[ashutosji]

Check the logs:
https://console.cloud.google.com/cloud-build/builds;region=global/275a3e64-934b-4512-80a4-28ff3ad733c8;step=6?e=13803378&mods=logs_tg_prod&project=agones-images

Error: Getting Permission denied for all resources.

[roberthbailey]

The error is coming from https://github.com/googleforgames/agones/blob/main/test/load/allocation/runscenario/runscenario.go#L126

Looking at the code, I'm curious how it ever works, since we have a gofunc per client and they are all writing to the same shared map.

[markmandel]

Check the logs:
https://console.cloud.google.com/cloud-build/builds;region=global/275a3e64-934b-4512-80a4-28ff3ad733c8;step=6?e=13803378&mods=logs_tg_prod&project=agones-images

Error: Getting Permission denied for all resources.

Sorry about that, probably not public, here's a copy
log-275a3e64-934b-4512-80a4-28ff3ad733c8.txt

Looking at the code, I'm curious how it ever works, since we have a gofunc per client and they are all writing to the same shared map.

I guess it doesn't manage to synchronously mutate the map too often.

Some quick ideas, not sure which ones are best:

• Some mutexes in the right places
• There are a few sync Maps that are thread safe out there, could go pick one (one with generics support?)
• Move stuff to channels?

[ashutosji]

I guess it doesn't manage to synchronously mutate the map too often.

Some quick ideas, not sure which ones are best:

• Some mutexes in the right places
• There are a few sync Maps that are thread safe out there, could go pick one (one with generics support?)
• Move stuff to channels?

I think placing mutex in the right places should work. I have created a small example for this use case: https://goplay.tools/snippet/N84X6jIZmIq

We can utilize the syncmap but syncmap is slow and has poor performance. Personally, I haven't done any benchmark testing but this is what i got from the internet.