Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/deadlock due to TCS setresult behaviour #5055

Merged
merged 3 commits into from
Dec 27, 2022

Conversation

asdacap
Copy link
Contributor

@asdacap asdacap commented Dec 26, 2022

Fixes #5043

  • It turns out TaskCompletionSource.SetResult may run it's continuation (task that is waiting for its result) synchronously. See this.
  • So this means, when MSMS activate a forward sync feed via SyncDispatcher.UpdateState which locks _feedStateManipulation, it's TrySetResult will continue SyncDispatcher.Start from line 162 where it wait for dormantTaskLocal synchronously.
  • This execution which held _feedStateManipulation lock may continue further into MergeBlockDownloader.Dispatch and remains until it attempt to download something (anything async).
  • If however, SyncPeerAllocation.Replaced is triggered before any download is queue (likely during ChainLevelHelper.GetNextHeaders as it is IO bound), the replaced event execution will get blocked at SyncDispatcher.UpdateState as _feedStateManipulation is not released.
  • If the sync execution from TCS will continue without releasing the lock until sync dispatcher attempt to free the sync peer allocation, it will hang but it can't free because _allocationLock is held by the the peer upgrade event.
  • Running DoDispatch with Task.Run or setting TaskCreationOptions.RunContinuationsAsynchronously will work.

Changes:

  • Run DoDispatch with Task.Run.
  • Add TaskCreationOptions.RunContinuationsAsynchronously to both SyncDispatcher._dormantStateTask and Request.CompletionSource for good measure.

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Other (please describe):

Testing

Requires testing

  • Yes
  • No

In case you checked yes, did you write tests??

  • Yes
  • No

Comments about testing , should you have some (optional)

  • goerli can sync.
  • no clear indication of regression or improvement.

Further comments (optional)

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

@asdacap asdacap merged commit 0ffeb06 into master Dec 27, 2022
@asdacap asdacap deleted the fix/deadlock-due-to-tcs-setresult-behaviour branch December 27, 2022 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Goerli][OldHeaders][ForwardSync] Forward Sync not started and OldHeaders stucked on goerli sync
2 participants