GIAB VCF import fails because htsjdk "repairs" header in vcf

[marksantcroos]

Hi folks,

In evaluating Hail to see whether it fits my use case (a variant frequency database) I ran into an issue with importing VCF files from GIAB. It turns out that these use type String for the PS ##FORMAT entry.

Subsequently, Hail fails to import these with the error:

is.hail.utils.HailException: HG001.vcf.gz:column 492: invalid character 'P' in integer literal

This is because of the default behaviour of htsjdk to "repair" these according to the VCF "standard".

htsjdk exposes codec.disableOnTheFlyModifications to toggle this behaviour which can be called from somewhere around https://github.com/hail-is/hail/blob/master/hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala#L1143.

Ideally I would like to expose this toggle also at the import_vcf method of Hail.
I'll create a PR to do so accordingly ASAP.

Comments/questions?

Thanks!

Regards,

Mark

[tpoterba]

Oh, we've been mad about this for a long time: #2822

If you can turn this behavior off, we're happy to accept that PR!

[tpoterba]

We're also pretty close to getting rid of htsjdk entirely, but this is a good immediate solution

[marksantcroos]

Ah, I missed #2822 in my search. That's why included as many strings that might match search queries in my ticket for future generations :-)

Anyway, the PR is there now. Please be gentle, these are my first lines of Scala ...