Address engine data races detected by go test -race

[AaronFriel]

These races were detected via Go's race detection tool and are separated by commit.

cd pkg && GOWORK=off go test -race -p 1 --parallel 4 -v ./engine/...

[RobbieMcKinstry]

This LGTM! Can we adjust the Makefile to run the race detector as part of CI @AaronFriel? That way we don't let new races pop up in this section of the codebase.

[AaronFriel]

@RobbieMcKinstry The race detector dramatically slows down tests, so I think we should do that work as part of a CI overhaul when we have merge queues that allow us to move that later in the process so it's non-blocking for velocity. (Or we find a tailored subset of tests to run with it.)

More anecdata here: https://www.google.com/search?q=go+test+%22race%22+performance+slower+site:github.com&client=firefox-b-1-m

We have a quality bar epic, I'll add merge queues and race detection issues to it.

[RobbieMcKinstry]

I'm forgetting how this is implemented. IIRC it's doing runtime detection, which is why tests slow down. I never understood why the Go team didn't implement an interprocedural analysis for this feature instead.

Thanks for volunteering to add the issue to the quality bar epic. IMO its perfectly fine to add it later when we have more opportunity to optimize its execution around CI bandwidth constraints.

[AaronFriel]

@RobbieMcKinstry Created:

• Reduce PR blocking and keep trunk green with merge queues #10091
• Prevent data races from being released #10092