Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread pool will block inside of pipe command for streams too large to buffer #1442

Closed
fnothaft opened this issue Mar 21, 2017 · 0 comments
Closed
Assignees
Labels
bug
Milestone

Comments

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Mar 21, 2017

Sigh, this is a silly one on my behalf! Inside of the pipe command, we return the output iterator by getting the Future from a Callable which is submitted to a thread pool. This only works if the output stream is small enough that it can be buffered fully. On large inputs, this causes a deadlock.

@fnothaft fnothaft added the bug label Mar 21, 2017
@fnothaft fnothaft added this to the 0.22.0 milestone Mar 21, 2017
@fnothaft fnothaft self-assigned this Mar 21, 2017
fnothaft added a commit to fnothaft/adam that referenced this issue Mar 21, 2017
Resolves bigdatagenomics#1442. Eliminates use of thread pool in GenomicRDD and instead launches
the InFormatterRunner (which extends `Callable`) as a thread and runs the
OutFormatterRunner in the original thread that launched the piped subprocess.

Additionally, rewrote the `AnySAMOutFormatter` to improve performance.
heuermh added a commit that referenced this issue Mar 21, 2017
Resolves #1442. Eliminates use of thread pool in GenomicRDD and instead launches
the InFormatterRunner (which extends `Callable`) as a thread and runs the
OutFormatterRunner in the original thread that launched the piped subprocess.

Additionally, rewrote the `AnySAMOutFormatter` to improve performance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.