Could bgzip be made fastq-aware?

[jenzopr]

Hi all,

I'm currently exploring possibilities of bgzip'ed FASTQ files and noticed that FASTQ entries might get split somewhere in between when the block size is reached.
Would it be possible to allow splits into different blocks only when certain criteria are met (e.g. a new FASTQ entry starts, or easier: if the line_number can be divided by 4 without remainder)?

Best and thanks for a short notice,
Jens

[JoeRayn]

Why would you want this?

The random access to a bam file regions is acheived via the virtual file offset in a index file. Looks like #673 covers this.

from bam specification ( https://samtools.github.io/hts-specs/SAMv1.pdf ) :
4.1.1 Random access
BGZF files support random access through the BAM file index. To achieve this, the BAM file index uses virtual file offsets into the BGZF file. Each virtual file offset is an unsigned 64-bit integer, defined as: coffset<<16|uoffset, where coffset is an unsigned byte offset into the BGZF file to the beginning of a BGZF block, and uoffset is an unsigned byte offset into the uncompressed data stream represented by that BGZF block. Virtual file offsets can be compared, but subtraction between virtual file offsets and addition between a virtual offset and an integer are both disallowed.

I can think of a lot of reasons to want this. One would be: you have a seeker library into AWS S3 objects, and you want to be able to seek to a number of workers and pass the shards out to workers in your cluster.

[jkbonfield]

Not a direct solution to the above, but as a workaround you could convert your fastq to unmapped SAM (trivial) and then to CRAM. This then permits indexing and retrieval by Nth-Mth container using io_lib's cram_fitler tool. The resultant CRAM would then need converting back to FASTQ probably.

Alternatively even if it's just unmapped BAM then you'll get the benefits of automatically attempting to start a new sequence on a new bgzf boundary.

All messy, but the data is more compact and with better random access than permitted by fastq.gz.

It doesn't invalidate this request, but given locally we never use fastq if we can help it (as it's a dire format and unaligned SAM et al is preferable for many reasons) it's also not going to be high on the priority. [As is evidenced by the fact this issue was long forgotten apparently.]

I agree with above. Although, is there no way to feed an unmapped SAM into bwa?

[jenzopr]

I think we can close this. Thanks!