Bgzip text mode (try 2) #1493
Merged
Bgzip text mode (try 2) #1493
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Mike Lin <dna@mlin.net> Compressing text now promotes alignment of BGZF blocks with the uncompressed text lines. BGZF blocks start at the beginning of an input line and end after some subsequent newline (except when the block's first line overflows the BGZF block size). This ensures it's possible to specify byte ranges of a BGZF file that decompress into complete text records -- useful for parallel processing and "slicing" from remote servers. To disable this feature and provide a way to produce identical output with 1.15 and earlier, the --binary option forces text data to be processed as if it were binary. The idea and initial implementation was Mike Lin's, with the current revised implementation by James Bonfield.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
An alternative to @mlin's #1369 for consideration.
The two implementations differ by:
At bgzf -l1 Mike's text mode added 6.2% CPU cycles, vs +0.4% for mine. However at level 6 these are considerably reduced as the Deflate overhead starts to dominate and these drop to 2.7% vs 0.0%. So maybe this isn't such an issue anyway. The main difference in CPU comes from searching backwards for the first newline vs searching forwards for the last newline. On long read technologies these two implementations start to converge a bit, but kgetline still has additional overhead with extra memcpys.
I also found the original implementation wasn't entirely foolproof when dealing with headers that start with lines over 64KB long. This is admittedly an extreme corner case, but it does mean the extra complexity of the new code may be doing something more anyway.