fix data race issue with calling wg.Add(1) after wg.Wait()

[alovak]

In certain situations, it is possible that the Close() method could be called at the same time as a callback function like ReadTimeoutHandler. This callback function could then call the Send method, which would attempt to increment the counter of a WaitGroup that has already had Wait called on it (inside Close). This scenario can lead to a data race, according to the data race detector.

To fix this issue, we should ensure that wg.Add(1) is not called after or during wg.Wait(). This is achieved by putting wg.Add(1) into the mutex that controls the closing state of the connection.

Some more context:

From the go documentation:

Note that calls with a positive delta that occur when the counter is zero must happen before a Wait.

Explanation from Rob Pike:

From the stack traces alone, this looks like a bug that showed up
today in some Google code. The issue is that there is no
happens-before relation between Waitgroup.Add and Waitgroup.Wait, so
your code does an Add after a Wait, the race detector will tell you.
You need to establish a synchronization to guarantee the Add does not
pass the Wait.

[alovak]

to reproduce the data race, I reduced the timeout here

[adamdecaf]

Can this get merged or tested more?

[alovak]

Yeah, going to test it with using test environment with real brand + test harness.