Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove continuation resumption inside locks #2558

Merged

Conversation

rnro
Copy link
Contributor

@rnro rnro commented Oct 17, 2023

Motivation:

Whilst investigating deadlocking code elsewhere we discovered that resuming a continuation inside a lock can cause deadlocks.

The reason is that

  • withTaskCancellationHandler takes a sequence's underlying runtime lock then executes the cancellation handler.
  • The task cancellation handler then does work which requires taking the NIO-level lock
  • If a NIO method on a separate thread then takes the NIO-level lock and within it resumes a continuation e.g. in yield we will deadlock because the resumption attempts to obtain the underlying runtime lock.

Modifications:

Do not resume continuations within locks.

Result:

Fewer deadlock opportunities

Motivation:

Whilst investigating deadlocking code elsewhere we discovered that
resuming a continuation inside a lock can call deadlocks.

The reason is that
- `withTaskCancellationHandler` takes a sequence's
underlying runtime lock then executes the cancellation handler.
- The task cancellation handler then does work which requires taking the
  NIO-level lock
- If a NIO method on a separate thread then takes the NIO-level lock and
  within it resumes a continuation e.g. in `yield` we will deadlock
because the resumption attempts to obtain the underlying runtime lock.

Modifications:

Do not resume continuations within locks.

Result:

Fewer deadlock opportunities
@rnro rnro requested a review from FranzBusch October 17, 2023 14:47
Copy link
Member

@FranzBusch FranzBusch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

@FranzBusch FranzBusch merged commit c2da95c into apple:main Oct 17, 2023
8 checks passed
@rnro rnro deleted the remove_continuation_resumption_inside_locks branch October 17, 2023 15:41
@FranzBusch FranzBusch added the patch-version-bump-only For PRs that when merged will only cause a bump of the patch version, ie. 1.0.x -> 1.0.(x+1) label Oct 17, 2023
@johnfairh
Copy link

Nice one fixing this - I started sporadically hitting this today and was just figuring out which locks were involved and dreading writing up the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
patch-version-bump-only For PRs that when merged will only cause a bump of the patch version, ie. 1.0.x -> 1.0.(x+1)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants