[hail] Rewrite VCF INFO Parser to not use htsjdk by chrisvittal · Pull Request #5828 · hail-is/hail

chrisvittal · 2019-04-09T01:04:22Z

We should now never call into htsjdk during line parsing, only during header parsing.

tpoterba · 2019-04-09T09:38:58Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

              rvb.start(t.physicalType)
              rvb.startStruct()
-              present = vcfLine.parseAddVariant(rvb, rgBc.map(_.value), contigRecoding, skipInvalidLoci)
+              present = vcfLine.parseAddVariant(rvb, rgBc.map(_.value), contigRecoding, hasRSID, skipInvalidLoci)


this feels like the wrong place for hasRSID - shouldn't it be handled in the same place as the filter/qual?

No. It doesn’t appear in the same place in a vcf line as filter and qual

Is the issue that you have to parse the ID before REF and ALT?

hail/src/main/scala/is/hail/io/vcf/HtsjdkRecordReader.scala

cseed · 2019-04-09T13:01:41Z

Any performance numbers?

chrisvittal · 2019-04-09T16:15:45Z

If there is an improvement, it is very small, running more tests now, but it looks like the new code is in the noise compared to other factors.

I believe rsid parsing is taking place in the correct spot. Other code was dead.

chrisvittal · 2019-04-09T18:05:33Z

More extensive cloud tests are showing speedups in the combiner pipeline compared to master, about 15-20 seconds per partition, but that adds up quickly at scale.

It was just a data class at that point.

jigold

I did a careful first pass. Giving you my feedback for now. Will look at it more carefully a second time since there's a lot of changes here.

jigold · 2019-04-09T16:17:02Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+      true
+    else {
+      val c = line(p)
+      c == '\t' || c == ';' || c == ','


When would a comma be the end of an array field?

I copied the name from endFormatArrayField, it ends an array element.

That's really confusing.

jigold · 2019-04-09T16:18:15Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

    }
  }

+  def endFilterArrayField(p: Int): Boolean = endInfoField


Is this for column 7 of the VCF? Piggy backing on the info field parsing?

They're the same, I wanted this name to make sense when parsing FILTERS.

jigold · 2019-04-09T16:21:24Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

    pos += 1 // tab
  }

+  def nextInfoField(): Unit = {


Do you need a hasNextInfoField?

No, the error handling could probably be cleaner, but if the assertion below fails then it's not a valid VCF. We carefully control when the next functions are called during parsing so we don't need to ask if there is a next field.

jigold · 2019-04-09T16:35:38Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+  def parseAddInfoInt(rvb: RegionValueBuilder) {
+    if (!infoFieldMissing()) {
+      rvb.setPresent()
+      rvb.addInt(parseInfoInt())


Is the default to set these to missing in the rvb? I feel like there should be an rvb.setMissing() here somewhere.

There is. I set everything to missing in parseAddInfo below.

jigold · 2019-04-09T16:37:15Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+  def parseAddInfoFloat(rvb: RegionValueBuilder) {
+    if (!infoFieldMissing()) {
+      rvb.setPresent()
+      rvb.addFloat(parseInfoString().toFloat)


What happens if these are parse errors? Will it be an incomprehensible error message? Is this what we do for parsing the genotype fields?

jigold · 2019-04-09T18:26:05Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    skipInfoField()
+
+    while (!endField()) {
+      nextInfoField()


Can you get rid of the code above and just keep this while loop?

No, The first key is special as it may be . to indicate that the whole INFO record is missing.

Sorry I meant this:

if (infoType.hasField(key)) { rvb.setFieldIndex(infoType.fieldIdx(key)) if (infoFlagFieldNames.contains(key)) rvb.addBoolean(true) else parseAddInfoField(rvb, infoType.fieldType(key)) } skipInfoField()

this is the alternative, thoughts?

var key = parseInfoKey() if (key == ".") { if (endField()) { rvb.endStruct() return } else parseError(s"invalid INFO key $key") } while (!endField()) { if (key == ".") { parseError(s"invalid INFO key $key") } if (infoType.hasField(key)) { rvb.setFieldIndex(infoType.fieldIdx(key)) if (infoFlagFieldNames.contains(key)) rvb.addBoolean(true) else parseAddInfoField(rvb, infoType.fieldType(key)) } skipInfoField() if (!endField()) { nextInfoField() key = parseInfoKey() } }

This seems better. So we just skip a field silently if it's not in the infoType?

jigold · 2019-04-09T18:31:27Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

              rvb.start(t.physicalType)
              rvb.startStruct()
-              present = vcfLine.parseAddVariant(rvb, rgBc.map(_.value), contigRecoding, skipInvalidLoci)
+              present = vcfLine.parseAddVariant(rvb, rgBc.map(_.value), contigRecoding, hasRSID, skipInvalidLoci)


Is the issue that you have to parse the ID before REF and ALT?

jigold · 2019-04-09T18:32:02Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

    arrayElementsRequired: Boolean,
    skipInvalidLoci: Boolean
  ): ContextRDD[RVDContext, RegionValue] = {
+    val hasRSID = t.isInstanceOf[TStruct] && t.asInstanceOf[TStruct].hasField("rsid")


Is this for the prune fields push down?

Yes, it is possible that the field won't be present in the final matrix and so we just need to skip over it.

jigold · 2019-04-09T18:32:47Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    if (c.hasQual) {
+      val qstr = l.parseString()
+      if (qstr == ".")
+        rvb.addDouble(-10.0)


Should this not be just set to missing?

htsjdk doesn't, it sets it to -10

That's annoying.

jigold · 2019-04-09T18:42:33Z

hail/src/main/scala/is/hail/io/vcf/HtsjdkRecordReader.scala

-      case (s: String, _: TFloat64) =>
-        val d = s match {
-          case "nan" => Double.NaN
-          case "-nan" => Double.NaN


I don't think your float parsing handles these correctly.

You are correct. Right now it handles it exactly as java would, to be properly VCF 4.3 compliant, we would need to handle, NaN, Inf, and -Inf. Which we don't even handle properly right now, just their lower case versions.

Can you make an issue then?

Futhermore, VCF 4.2 makes no claims about the string representation of floats at all. VCF is garbage.

chrisvittal

Left some replies.

chrisvittal · 2019-04-09T20:30:05Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+      else
+        parseAddInfoField(rvb, infoType.fieldType(key))
+    }
+    skipInfoField()


It mirrors skipFormatField

chrisvittal · 2019-04-09T20:32:24Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    skipInfoField()
+
+    while (!endField()) {
+      nextInfoField()


No, The first key is special as it may be . to indicate that the whole INFO record is missing.

chrisvittal · 2019-04-09T20:33:51Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

    arrayElementsRequired: Boolean,
    skipInvalidLoci: Boolean
  ): ContextRDD[RVDContext, RegionValue] = {
+    val hasRSID = t.isInstanceOf[TStruct] && t.asInstanceOf[TStruct].hasField("rsid")


Yes, it is possible that the field won't be present in the final matrix and so we just need to skip over it.

chrisvittal · 2019-04-09T20:34:00Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

              rvb.start(t.physicalType)
              rvb.startStruct()
-              present = vcfLine.parseAddVariant(rvb, rgBc.map(_.value), contigRecoding, skipInvalidLoci)
+              present = vcfLine.parseAddVariant(rvb, rgBc.map(_.value), contigRecoding, hasRSID, skipInvalidLoci)


chrisvittal · 2019-04-09T20:34:47Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    if (c.hasQual) {
+      val qstr = l.parseString()
+      if (qstr == ".")
+        rvb.addDouble(-10.0)


htsjdk doesn't, it sets it to -10

chrisvittal · 2019-04-09T21:13:03Z

hail/src/main/scala/is/hail/io/vcf/HtsjdkRecordReader.scala

-      case (s: String, _: TFloat64) =>
-        val d = s match {
-          case "nan" => Double.NaN
-          case "-nan" => Double.NaN


You are correct. Right now it handles it exactly as java would, to be properly VCF 4.3 compliant, we would need to handle, NaN, Inf, and -Inf. Which we don't even handle properly right now, just their lower case versions.

chrisvittal · 2019-04-09T21:44:06Z

@cseed

import hail as hl
hl.import_vcf(PATH, reference_genome='GRCh38').write('/tmp/vcfmt', overwrite=True)

On my laptop with a warmish filesystem cache takes 1 minute for this code and 1:20 for current master. That's actually pretty good.

jigold

Do we have tests that are complicated enough to fully test these changes? Maybe a complicated VEP signature?

jigold · 2019-04-10T20:00:29Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+      true
+    else {
+      val c = line(p)
+      c == '\t' || c == ';' || c == ','


That's really confusing.

jigold · 2019-04-10T20:01:59Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+  }
+
+  def parseStringInfoArrayElement() {
+    if (infoArrayFieldMissing()) {


jigold · 2019-04-10T20:03:46Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    skipInfoField()
+
+    while (!endField()) {
+      nextInfoField()


Sorry I meant this:

if (infoType.hasField(key)) { rvb.setFieldIndex(infoType.fieldIdx(key)) if (infoFlagFieldNames.contains(key)) rvb.addBoolean(true) else parseAddInfoField(rvb, infoType.fieldType(key)) } skipInfoField()

jigold · 2019-04-10T20:12:04Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    if (c.hasQual) {
+      val qstr = l.parseString()
+      if (qstr == ".")
+        rvb.addDouble(-10.0)


That's annoying.

jigold · 2019-04-10T20:16:31Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

  }
 }
+
+class BufferedLineIterator(bit: BufferedIterator[String]) extends htsjdk.tribble.readers.LineIterator {


Can you move this to the top of the file?

jigold · 2019-04-10T20:17:45Z

hail/src/main/scala/is/hail/io/vcf/HtsjdkRecordReader.scala

-      case (s: String, _: TFloat64) =>
-        val d = s match {
-          case "nan" => Double.NaN
-          case "-nan" => Double.NaN


Can you make an issue then?

jigold · 2019-04-10T20:23:48Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    if (endInfoArrayField())
+      parseError("empty integer")
+    var mul = 1
+    if (line(pos) == '-') {


Please reply to discussion of key parsing. Otherwise, I think this is good. I'll work on tests.

The MatrixTables checked in here were the result of using the old parser to read vcfs and write matrix tables.

chrisvittal · 2019-04-11T18:31:19Z

I added some old vs. new tests.

jigold · 2019-04-11T19:44:14Z

hail/python/test/hail/methods/test_impex.py


+    def test_vcf_parser_golden_master(self):
+        # the three matrix tables referenced here were generated using the old VCF parser
+        # parser


ok, I still feel that knowing how these files were generated if this test ever fails will be useful.

jigold · 2019-04-11T19:48:26Z

hail/src/test/resources/ex.vcf

+##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth">
+##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality">
+##FORMAT=<ID=CNL,Number=.,Type=Integer>
+#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	NA00001	NA00002	NA00003


Does this cover all possible field types? For backwards compatibility add the fields with nans etc. What about adding an example with DB=0? I see arrays of doubles? Do we also need arrays of strings? Can we also get a completely missing INFO field .? Just trying to make sure we got all possible code paths.

jigold · 2019-04-11T19:52:05Z

hail/src/main/scala/is/hail/io/vcf/LoadVCF.scala

+    skipInfoField()
+
+    while (!endField()) {
+      nextInfoField()


This seems better. So we just skip a field silently if it's not in the infoType?

jigold

This looks pretty good. See comments.

Done

chrisvittal changed the title ~~Better vcf parsing~~ [hail] Rewrite VCF INFO Parser to not use htsjdk Apr 9, 2019

chrisvittal requested a review from jigold April 9, 2019 01:04

chrisvittal assigned jigold Apr 9, 2019

tpoterba previously requested changes Apr 9, 2019

View reviewed changes

chrisvittal added 7 commits April 9, 2019 11:22

Move rsid parsing to parseAddVariant

77addeb

Add qual parsing to parseLine/out of HtsjdkRecordReader

3c207c6

Add filter parsing to VCFLine

1749b42

Add info parsing to VCFline

4a3062e

Remove unused parameter

4efed48

Fix issues, remove codec decoding

3ebd600

Remove dead code

a89928f

chrisvittal force-pushed the better-vcf-parsing branch from 3928789 to a89928f Compare April 9, 2019 15:22

Remove HtsjdkRecordReader

3b28700

It was just a data class at that point.

jigold suggested changes Apr 9, 2019

View reviewed changes

Fix json serialization in python for field rename

d745c78

chrisvittal commented Apr 9, 2019

View reviewed changes

INFO floats are always doubles

a25f765

jigold previously requested changes Apr 10, 2019

View reviewed changes

chrisvittal added 2 commits April 10, 2019 16:48

Move BufferedLineIterator

ad336ab

s/ArrayField/ArrayElement/g

fda182b

chrisvittal added 2 commits April 11, 2019 14:26

Add flag=0 checking.

7e9fa30

Refactor common code, add old vs new output checking

c7d1a63

The MatrixTables checked in here were the result of using the old parser to read vcfs and write matrix tables.

chrisvittal force-pushed the better-vcf-parsing branch from af5ac2f to c7d1a63 Compare April 11, 2019 18:30

Re-add string to double logic from old parser

4b760b4

jigold reviewed Apr 11, 2019

View reviewed changes

jigold previously requested changes Apr 11, 2019

View reviewed changes

Add some info line cases

9fbc048

jigold approved these changes Apr 12, 2019

View reviewed changes

danking merged commit 2571917 into hail-is:master Apr 13, 2019

Conversation

chrisvittal commented Apr 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisvittal Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cseed commented Apr 9, 2019

Uh oh!

chrisvittal commented Apr 9, 2019

Uh oh!

chrisvittal commented Apr 9, 2019

Uh oh!

jigold left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisvittal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisvittal Apr 9, 2019 •

edited

Loading