Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add synchronization to prevent race in broker #14

Merged
merged 3 commits into from Dec 5, 2019
Merged

Add synchronization to prevent race in broker #14

merged 3 commits into from Dec 5, 2019

Conversation

Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
@cohosh
Copy link
Owner

@cohosh cohosh commented Nov 22, 2019

There's a race condition in the broker where both the proxy and the
client processes try to pop/remove the same snowflake from the heap.
This patch adds synchronization to prevent simultaneous accesses to
snowflakes.

broker/broker.go Outdated
@@ -37,6 +38,8 @@ type BrokerContext struct {
// Map keeping track of snowflakeIDs required to match SDP answers from
// the second http POST.
idToSnowflake map[string]*Snowflake
// Synchronization for the
Copy link
Contributor

@NullHypothesis NullHypothesis Nov 27, 2019

For the what? 😉

Copy link
Owner Author

@cohosh cohosh Dec 2, 2019

Fixed :)

ctx.snowflakeLock.Lock()
numSnowflakes := ctx.snowflakes.Len()
ctx.snowflakeLock.Unlock()
if numSnowflakes <= 0 {
ctx.metrics.clientDeniedCount++
Copy link
Contributor

@NullHypothesis NullHypothesis Nov 27, 2019

Should this increment also be protected by a lock? Or is it impossible for two goroutines to execute it simultaneously?

Copy link
Owner Author

@cohosh cohosh Dec 2, 2019

Thanks for pointing this out. There is a chance for a race here if the counts are updated at the same time that the metrics are being logged (this happens once every 24 hours).

I added a different lock to the metrics struct to handle this in a new commit. How does it look?

cohosh added 3 commits Dec 5, 2019
There's a race condition in the broker where both the proxy and the
client processes try to pop/remove the same snowflake from the heap.
This patch adds synchronization to prevent simultaneous accesses to
snowflakes.
We had some data races in the broker that occur when proxies and clients
modify the heap/snowflake map at the same time. This test has a client
and proxy access the broker simultaneously to check for data races.
Added another lock to the metrics struct to synchronize accesses to the
broker stats. There's a possible race condition if stats are updated at
the same time they are being logged.
@cohosh cohosh merged commit 06298ee into master Dec 5, 2019
2 checks passed
@cohosh cohosh deleted the bug32576 branch Mar 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment