Description
In src/compute-plane-services/worker-utils/worker/large.go, multipartUpload
launches up to multipartUploadConcurrency part-upload goroutines. Each goroutine
wrote the shared function-scope err variable rather than a goroutine-local one:
var uploadResult *s3.UploadPartOutput
_, _, err = lo.AttemptWithDelay(3, 100*time.Millisecond, func(index int, duration time.Duration) error {
var err error
uploadResult, err = client.UploadPart(ctx, partInput)
return err
})
if err != nil {
return err
}
The err on the _, _, err = line is the multipartUpload function-scope err
(declared earlier and also reused by the main reader goroutine). Multiple upload
goroutines read and write it concurrently.
Impact
- Data race on
err (flagged by go test -race).
- If one part's upload fails (its local
uploadResult stays nil) but a peer
goroutine overwrites the shared err to nil before the failing goroutine
reaches its if err != nil check, the failing goroutine skips the return and
dereferences uploadResult.ETag on a nil pointer, panicking the worker.
- The mirror case makes a successful part inherit a peer's error and spuriously
abort the whole upload.
This is on the large-response upload path (responses over the multipart
threshold).
Definition of Done
- Each part upload uses a goroutine-local error and result.
- A unit test exercises the success and failure paths and a concurrent run under
the race detector, asserting a failing part does not clobber successful parts.
Description
In
src/compute-plane-services/worker-utils/worker/large.go,multipartUploadlaunches up to
multipartUploadConcurrencypart-upload goroutines. Each goroutinewrote the shared function-scope
errvariable rather than a goroutine-local one:The
erron the_, _, err =line is themultipartUploadfunction-scopeerr(declared earlier and also reused by the main reader goroutine). Multiple upload
goroutines read and write it concurrently.
Impact
err(flagged bygo test -race).uploadResultstays nil) but a peergoroutine overwrites the shared
errto nil before the failing goroutinereaches its
if err != nilcheck, the failing goroutine skips the return anddereferences
uploadResult.ETagon a nil pointer, panicking the worker.abort the whole upload.
This is on the large-response upload path (responses over the multipart
threshold).
Definition of Done
the race detector, asserting a failing part does not clobber successful parts.