FastxFile silently prematurely truncates gzipped FASTQ

[jbloom]

test.fastq.gz
The attached *.fastq.gz file has 24520 entries.

But if I read them using FastxFile, it reports that there are only 114 entries:

>>>  len([_ for _ in pysam.FastxFile('test.fastq.gz')])
114

The problem appears to have something to do with the fact that the file is gzipped. If I gunzip it and then do the same, it correctly reports all the entries:

>>> len([_ for _ in pysam.FastxFile('test.fastq')])
24520

I'm not sure of the source of the bug, but I think FastxFile should either read all entries from the gzipped FASTQ or raise an error if there is some problem with the gzipped file.

This is with pysam version 0.15.4.

[jmarshall]

Your gzipped file contains ~200 gzip members — was it perhaps created by concatenating lots of individual .gz files?

You have encountered samtools/htslib#742: htslib 1.9 did not handle such multi-member plain gzipped files. This will be fixed in pysam when it moves from htslib 1.9 to 1.10 — or in the meantime you could apply PR #905 to your pysam yourself.

[jmarshall]

Duplicate of #738

[jmarshall]

This has been fixed since pysam 0.16.0 (when used with HTSlib 1.10 or later).