New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1442] Fix thread pool deadlock in GenomicRDD.pipe #1443

Merged
merged 1 commit into from Mar 21, 2017

Conversation

Projects
None yet
4 participants
@fnothaft
Member

fnothaft commented Mar 21, 2017

Resolves #1442. Eliminates use of thread pool in GenomicRDD and instead launches the InFormatterRunner (which extends Callable) as a thread and runs the OutFormatterRunner in the original thread that launched the piped subprocess.

Additionally, rewrote the AnySAMOutFormatter to improve performance.

With this, BWA-on-Cannoli works (https://github.com/heuermh/cannoli/pull/2).

[ADAM-1442] Fix thread pool deadlock in GenomicRDD.pipe
Resolves #1442. Eliminates use of thread pool in GenomicRDD and instead launches
the InFormatterRunner (which extends `Callable`) as a thread and runs the
OutFormatterRunner in the original thread that launched the piped subprocess.

Additionally, rewrote the `AnySAMOutFormatter` to improve performance.
@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Mar 21, 2017

Coverage Status

Coverage decreased (-0.3%) to 80.316% when pulling 5600cf6 on fnothaft:issues/1442-pipe-hang into b1fce67 on bigdatagenomics:master.

Coverage Status

Coverage decreased (-0.3%) to 80.316% when pulling 5600cf6 on fnothaft:issues/1442-pipe-hang into b1fce67 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Mar 21, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1889/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1443/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 27a25d8d1f707bfe451f787b310ef7f5689a5c5e # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1443/merge^{commit} # timeout=10Checking out Revision 27a25d8d1f707bfe451f787b310ef7f5689a5c5e (origin/pr/1443/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 27a25d8d1f707bfe451f787b310ef7f5689a5c5eFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1889/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1443/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 27a25d8d1f707bfe451f787b310ef7f5689a5c5e # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1443/merge^{commit} # timeout=10Checking out Revision 27a25d8d1f707bfe451f787b310ef7f5689a5c5e (origin/pr/1443/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 27a25d8d1f707bfe451f787b310ef7f5689a5c5eFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 21, 2017

Member

Jenkins, retest this please.

Member

fnothaft commented Mar 21, 2017

Jenkins, retest this please.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Mar 21, 2017

Coverage Status

Coverage decreased (-0.07%) to 80.504% when pulling 5600cf6 on fnothaft:issues/1442-pipe-hang into b1fce67 on bigdatagenomics:master.

coveralls commented Mar 21, 2017

Coverage Status

Coverage decreased (-0.07%) to 80.504% when pulling 5600cf6 on fnothaft:issues/1442-pipe-hang into b1fce67 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Mar 21, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1890/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1890/
Test PASSed.

@heuermh heuermh merged commit 95b2a86 into bigdatagenomics:master Mar 21, 2017

2 of 3 checks passed

coverage/coveralls Coverage decreased (-0.07%) to 80.504%
Details
codacy/pr Good work! A positive pull request.
Details
default Merged build finished.
Details
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 21, 2017

Member

Thank you, @fnothaft!

Member

heuermh commented Mar 21, 2017

Thank you, @fnothaft!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment