Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: avoid deadlock in publisher and subscriber #1749

Merged
merged 3 commits into from Sep 24, 2021

Conversation

aman-bansal
Copy link
Contributor

@aman-bansal aman-bansal commented Sep 23, 2021

There is one scenario where badger could end up in a deadlock state. Complete goroutine dump can be found here https://drive.google.com/file/d/1nIrYlrbwlGtvk4WGDCzM0z996eEydaNI/view?usp=sharing

The issue is publisher sends out the messages over the subscriber channel and subscriber will process those message one at a time. Now subsriber channel size is 1000, so if the channel is completely filled publisher will wait indefinitly to send the message. And this is what happened. while processing the message, subscriber receive the error after 15 min and in the meantime publisher was waiting indefinitely for the channel to clear. On receiving the error subscriber asked the publisher to delete the subscriber and this turns into a deadlock state where one lock is acquired by the publisher to send the message and message processor of subscriber is waiting on that lock to delete the subscriber.

This PR tries to solve this issue by adding a atomic variable in subscriber so that publisher can stop sending out the new updates and thus making the deadlock to be released eventually.


This change is Reviewable

Copy link
Contributor

@NamanJain8 NamanJain8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. A couple of minor comments.

publisher_test.go Outdated Show resolved Hide resolved
publisher_test.go Show resolved Hide resolved
@aman-bansal aman-bansal merged commit 7677fcb into master Sep 24, 2021
3 of 5 checks passed
@aman-bansal aman-bansal deleted the aman/fix_pub_deadlock branch September 24, 2021 14:40
aman-bansal added a commit that referenced this pull request Sep 24, 2021
* fix: avoid deadlock in publisher and subscriber
aman-bansal added a commit that referenced this pull request Sep 24, 2021
* fix: avoid deadlock in publisher and subscriber
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants