fix(storage): fix read-write race in Writer.Write

[dragonsinth]

#6816

w.monitorCancel() -> w.CloseWithError() reads the values of w.opened and w.pw concurrently with the calling routine initializing them; if the incoming context is cancelled before or during initialization.

The fix is simply to deploy starting the monitor until all fields are initialized. The older open() code from previous releases did this in the right order:

https://github.com/googleapis/google-cloud-go/blob/storage/v1.24.0/storage/writer.go#L130-L134

[dragonsinth]

Note that the w.mu.Lock()/w.mu.Unlock() in monitorCancel() doesn't help because
a) it only guards w.err
b) openWriter() isn't holding the lock anyway

[tritone]

Thanks for catching this and for the proposed fix!

I think the spot you have put go monitorCancel() is too late. In the old code you linked, we launch monitorCancel after opening the pipe but before launching the go routine that actually sends a request. This reduces the likelihood that we would send a request after the context has already been cancelled. In this PR that logic happens inside the OpenWriter call on L193.

I think the answer is to pass monitorCancel in with the openWriterParams and then launch it from within the transport-specific clients and then have a callback to set Writer.pw. I can provide more direction on how to do this if you like, or implement it myself. I'd also like to add your repro as a test -- we have another test that checks for a similar condition here but this one passes because the cancellation happens after the first write I believe.

[dragonsinth]

I think the answer is to pass monitorCancel in with the openWriterParams and then launch it from within the transport-specific clients and then have a callback to set Writer.pw. I can provide more direction on how to do this if you like, or implement it myself. I'd also like to add your repro as a test -- we have another test that checks for a similar condition here but this one passes because the cancellation happens after the first write I believe.

Huh, I'm unfamiliar with the code underneath tc.OpenWriter as that jumps to an interface, but on the surface I wonder if it's really worth breaking through all those layers of abstraction?

I can think of at least two less-invasive ideas that have the effect of avoiding initiating the call:

1. Explicitly check context immediately before calling OpenWriter, e.g.:

	if err := w.ctx.Err(); err != nil {
		return err // short-circuit
	}
	w.pw, err = w.o.c.tc.OpenWriter(params, opts...)
	if err != nil {
		return err
	}
	w.opened = true
	go w.monitorCancel()

This will prevent the extra call in almost all cases.

2. Pass w.ctx down to OpenWriter and have it respect context internally; this would be the more standard way of doing things, but a more invasive change.

[tritone]

@dragonsinth Thanks for the suggestions, agreed that those would both be less complicated. I think your option (1) seems like a good idea; option (2) seems like a little more complexity if the pipe has already been opened and we have to do cleanup that currently occurs in monitorCancel. If you want to make that change to this PR I would be happy to approve.

[dragonsinth]

@dragonsinth Thanks for the suggestions, agreed that those would both be less complicated. I think your option (1) seems like a good idea; option (2) seems like a little more complexity if the pipe has already been opened and we have to do cleanup that currently occurs in monitorCancel. If you want to make that change to this PR I would be happy to approve.

Sweet, updated this PR.

[tritone]

Looks good to me. I'll add a test based on your example in the issue in a separate PR.

[dragonsinth]

Awesome, thanks @tritone