Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storagenode/orders: fix data race in settle #3042

Merged
merged 2 commits into from
Sep 13, 2019
Merged

Conversation

egonelbre
Copy link
Member

errList was being modified from two goroutines, use a separate errs.Group to capture errors. Unfortunately, errs.Group cannot be returned easily from group.Go, which would allow to make this bug less likely.

Please describe the tests:

  • Test 1:
  • Test 2:

Please describe the performance impact:

Code Review Checklist (to be filled out by reviewer)

  • Does the PR describe what changes are being made?
  • Does the PR describe why the changes are being made?
  • Does the code follow our style guide?
  • Does the code follow our testing guide?
  • Is the PR appropriately sized? (If it could be broken into smaller PRs it should be)
  • Does the new code have enough tests? (every PR should have tests or justification otherwise. Bug-fix PRs especially)
  • Does the new code have enough documentation that answers "how do I use it?" and "what does it do?"? (both source documentation and higher level, diagrams?)
  • Does any documentation need updating?
  • Do the database access patterns make sense?

@egonelbre egonelbre requested a review from a team September 13, 2019 07:34
@cla-bot cla-bot bot added the cla-signed label Sep 13, 2019
@ghost ghost requested review from Barterio and wthorp and removed request for a team September 13, 2019 07:34
@egonelbre egonelbre added the Request Code Review Code review requested label Sep 13, 2019
@egonelbre egonelbre added the Reviewer Can Merge If all checks have passed, non-owner can merge PR label Sep 13, 2019
@ifraixedes ifraixedes self-requested a review September 13, 2019 09:45
@@ -291,7 +290,7 @@ func (service *Service) settle(ctx context.Context, log *zap.Logger, satelliteID
zap.Error(err),
zap.Any("request", req),
)
errList.Add(err)
sendErrors.Add(err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't we use a buffered of error elements channel of len(orders) capacity and iterate over it after group.Wait() excluding nil element and accumulate them errList how it's done now?

Copy link
Member Author

@egonelbre egonelbre Sep 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? This sounds like a much more complicated approach than it's currently using.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't sendErrors.Add called concurrently? of if it's .Add is safe to be called concurrently?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sendErrors.Add is not called concurrently

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah, I saw the loop and the group in inverse order.
I've got confused with the "unfortunately" part of the initial message.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused now. why is sendErrors.Add not called concurrently?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is only goroutine using sendErrors at a time at all times. I.e. it's called from another goroutine, but there's only one of those.... so technically no concurrent calls.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it's only called inside of group.Go and it's read after the group ends.

I was mistaken because my brain lied to me making me see the for loop before group.Go so I thought that several goroutines created by several calls to group.Go were running, but the loop it's inside so there is only one goroutine.

@@ -291,7 +290,7 @@ func (service *Service) settle(ctx context.Context, log *zap.Logger, satelliteID
zap.Error(err),
zap.Any("request", req),
)
errList.Add(err)
sendErrors.Add(err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah, I saw the loop and the group in inverse order.
I've got confused with the "unfortunately" part of the initial message.

Copy link
Contributor

@VinozzZ VinozzZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@egonelbre egonelbre merged commit ca058e6 into master Sep 13, 2019
@egonelbre egonelbre deleted the ee/race-settle branch September 13, 2019 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed Request Code Review Code review requested Reviewer Can Merge If all checks have passed, non-owner can merge PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants