New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread pool will block inside of pipe command for streams too large to buffer #1442

Closed
fnothaft opened this Issue Mar 21, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@fnothaft
Member

fnothaft commented Mar 21, 2017

Sigh, this is a silly one on my behalf! Inside of the pipe command, we return the output iterator by getting the Future from a Callable which is submitted to a thread pool. This only works if the output stream is small enough that it can be buffered fully. On large inputs, this causes a deadlock.

@fnothaft fnothaft added the bug label Mar 21, 2017

@fnothaft fnothaft added this to the 0.22.0 milestone Mar 21, 2017

@fnothaft fnothaft self-assigned this Mar 21, 2017

fnothaft added a commit to fnothaft/adam that referenced this issue Mar 21, 2017

[ADAM-1442] Fix thread pool deadlock in GenomicRDD.pipe
Resolves #1442. Eliminates use of thread pool in GenomicRDD and instead launches
the InFormatterRunner (which extends `Callable`) as a thread and runs the
OutFormatterRunner in the original thread that launched the piped subprocess.

Additionally, rewrote the `AnySAMOutFormatter` to improve performance.

@heuermh heuermh closed this in #1443 Mar 21, 2017

heuermh added a commit that referenced this issue Mar 21, 2017

[ADAM-1442] Fix thread pool deadlock in GenomicRDD.pipe
Resolves #1442. Eliminates use of thread pool in GenomicRDD and instead launches
the InFormatterRunner (which extends `Callable`) as a thread and runs the
OutFormatterRunner in the original thread that launched the piped subprocess.

Additionally, rewrote the `AnySAMOutFormatter` to improve performance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment