-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-14131][SQL[STREAMING] Improved fix for avoiding potential deadlocks in HDFSMetadataLog #14292
Conversation
Test build #62645 has finished for PR 14292 at commit
|
test this |
Test build #62644 has finished for PR 14292 at commit
|
@tdas this change breaks the tests as they don't run in UninterruptibleThread |
Fixing it. |
LGTM. Pending tests. |
Test build #62687 has finished for PR 14292 at commit
|
Test build #3189 has finished for PR 14292 at commit
|
* potential dead-lock in Hadoop "Shell.runCommand" before 2.5.0 (HADOOP-10622). If the thread | ||
* running "Shell.runCommand" is interrupted, then the thread can get deadlocked. In our | ||
* case, `writeBatch` creates a file using HDFS API and calls "Shell.runCommand" to set the | ||
* file permissions, and can get deadlocked is the stream execution thread is stopped by |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/is/if
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
Test build #62835 has finished for PR 14292 at commit
|
Test build #3190 has finished for PR 14292 at commit
|
Tests have passed. Merging this to master and 2.0. Thanks for reviewing @zsxwing @jaceklaskowski |
…locks in HDFSMetadataLog ## What changes were proposed in this pull request? Current fix for deadlock disables interrupts in the StreamExecution which getting offsets for all sources, and when writing to any metadata log, to avoid potential deadlocks in HDFSMetadataLog(see JIRA for more details). However, disabling interrupts can have unintended consequences in other sources. So I am making the fix more narrow, by disabling interrupt it only in the HDFSMetadataLog. This is a narrower fix for something risky like disabling interrupt. ## How was this patch tested? Existing tests. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #14292 from tdas/SPARK-14131. (cherry picked from commit c979c8b) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
What changes were proposed in this pull request?
Current fix for deadlock disables interrupts in the StreamExecution which getting offsets for all sources, and when writing to any metadata log, to avoid potential deadlocks in HDFSMetadataLog(see JIRA for more details). However, disabling interrupts can have unintended consequences in other sources. So I am making the fix more narrow, by disabling interrupt it only in the HDFSMetadataLog. This is a narrower fix for something risky like disabling interrupt.
How was this patch tested?
Existing tests.