Go SDK: fix data race on Edge Worker's activeWorkloads map - #70188
Open
ColtenOuO wants to merge 1 commit into
Open
Go SDK: fix data race on Edge Worker's activeWorkloads map#70188ColtenOuO wants to merge 1 commit into
ColtenOuO wants to merge 1 commit into
Conversation
Every fetched job is dispatched via `go w.runWorkload(...)`, which reads and writes worker.activeWorkloads from that goroutine, while heartbeat() and the drain check in mainLoop() read it from a different goroutine -- all without synchronization. Since maxConcurrency defaults to 16, running more than one task at a time is the normal case, not an edge case. Reproduced with `go test -race` (4 distinct races reported) and, without -race, an actual unrecoverable `fatal error: concurrent map writes` that crashes the whole worker process, not just one task. Guard all reads/writes of activeWorkloads with a mutex.
ColtenOuO
requested review from
amoghrajesh,
ashb and
jason810496
as code owners
July 21, 2026 12:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
worker.activeWorkloads(a plainmap[uuid.UUID]bundlev1.ExecuteTaskWorkload)is read and written from multiple goroutines with no synchronization:
mainLoop()dispatches every fetched job viago w.runWorkload(ctx, ...)(a new goroutine per task), and
runWorkloadwrites to the map on startand deletes from it on completion.
heartbeat()and the drain check inmainLoop()readlen(w.activeWorkloads)from a separate goroutine.
maxConcurrencydefaults to 16, so running multiple tasks concurrently isthe normal operating mode of an Edge Worker, not an edge case. Any time two
tasks start/finish around the same moment a heartbeat or drain check reads
the map's length, this is a data race.
Reproduction
Confirmed two ways before fixing:
go test -race: 4 distinct data races reported, all pointing at thedocumented read/write sites (map assignment, map delete, and the
unsynchronized
len()reads).Without
-race, running the same concurrent access pattern repeatedlyreliably produces:
This is a Go runtime
fatal error, not apanic--recover()cannotcatch it, so it crashes the entire worker process (all in-flight tasks),
not just the one task being started/finished at that moment.
Changes
activeWorkloadsMu sync.Mutextoworker, and guard all fouraccess points (the write in
runWorkload, the delete in its deferredcleanup, and the two
len()reads inheartbeat()andmainLoop()'sdrain check).
Was generative AI tooling used to co-author this PR?