Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Improve concurrency in multithreaded BGZF writer (bgzf_mt) #51
This PR includes two commits that significantly increase compression throughput with the multithreaded BGZF writer (bgzf_mt), improving the performance of samtools sort, merge, and view -b in multithreaded mode.
The first commit yields most of the performance benefit by continuing to ingest data while multithreaded compression proceeds in the background. This change does not complicate things very much and hopefully should be fairly uncontroversial.
The second commit, which yields a smaller improvement, is a bit riskier because it introduces writing to the file handle from different threads. It currently works out that the writes are serialized without needing an explicit mutex, but this could be a maintenance concern going forward.
Instead of periodically stalling the caller of bgzf_write to compress buffered data, run multithreaded compression on a 'background' workspace while continuing to ingest data into a 'foreground' workspace. When the foreground is full and the current round of background compression is complete, write out the results and swap the foreground & background. This significantly increases compression throughput with multiple cores under most circumstances, because the worker threads don't have to wait as long for the next batch of data to start compressing. The n_threads parameter to bgzf_mt now specifies the number of worker threads to spawn in addition to the main thread. Therefore, it's theoretically possible for the process to utilize n_threads+1 CPUs. However, in most applications the main thread is largely I/O-bound. Maintaining foreground & background workspaces doubles the memory usage associated with bgzf_mt, which was however modest to begin with (~16MB*n_threads, based on the suggested upper limit of n_sub_blks).
main thread. Prevents stalling the main thread on I/O in some circumstances. This change incurs some maintenance risk because it introduces use of hwrite from different threads (the main thread and one worker thread) without an explicit mutex. It does work out that only one thread is writing at any given time, modulo a new assert in bgzf_raw_write, and this will need to be maintained going forward.