error in annotation filtering for t2d vcf

[jbloom22]

Tim, when I run the following command I get the exception below.

~/hail/build/install/hail/bin/hail import -i ~/t2d/GoT2D.first10k.vcf filtervariants --keep -c "true" count

I get the following exception. "[-80" is not in the original vcf, so Cotton thinks it may be because htsjdk parses the info, then you converts them back to Strings, and then reparses them, so you might have trouble eating your own output. I'll share the vcf with you. I have no trouble filtering based on sample and interval lists, or doing qc or linreg.

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost): java.lang.NumberFormatException: For input string: "[-80"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.parseInt(Integer.java:615)
    at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
    at scala.collection.immutable.StringOps.toInt(StringOps.scala:30)
    at org.broadinstitute.hail.methods.AnnotationValueString$$anonfun$toArrayInt$extension$1.apply(Filter.scala:18)
    at org.broadinstitute.hail.methods.AnnotationValueString$$anonfun$toArrayInt$extension$1.apply(Filter.scala:18)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
    at org.broadinstitute.hail.methods.AnnotationValueString$.toArrayInt$extension(Filter.scala:18)
    at __wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c.__wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c$$anonfun$wrapper$1$__vaClass$1$__info$$anonfun$3.apply(<no source file>:11)
    at __wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c.__wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c$$anonfun$wrapper$1$__vaClass$1$__info$$anonfun$3.apply(<no source file>:11)
    at scala.Option.map(Option.scala:146)
    at __wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c.__wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c$$anonfun$wrapper$1$__vaClass$1$__info.<init>(<no source file>:11)
    at __wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c.__wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c$$anonfun$wrapper$1$__vaClass$1.<init>(<no source file>:23)
    at __wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c.__wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c$$anonfun$wrapper$1.apply(<no source file>:32)
    at __wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c.__wrapper$1$c1aef7d48806473ea6c7fc7ceb61989c$$anonfun$wrapper$1.apply(<no source file>:2)
    at org.broadinstitute.hail.methods.FilterVariantCondition.apply(Filter.scala:613)
    at org.broadinstitute.hail.driver.FilterVariants$$anonfun$2.apply(FilterVariants.scala:45)
    at org.broadinstitute.hail.driver.FilterVariants$$anonfun$2.apply(FilterVariants.scala:45)
    at org.broadinstitute.hail.variant.VariantSampleMatrix$$anonfun$5.apply(VariantSampleMatrix.scala:151)
    at org.broadinstitute.hail.variant.VariantSampleMatrix$$anonfun$5.apply(VariantSampleMatrix.scala:151)
    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:415)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1626)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1099)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1099)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
    at org.apache.spark.scheduler.Task.run(Task.scala:70)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1273)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1264)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1263)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1263)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1457)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1418)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

[tpoterba]

This was caused by a problem in my use of hstjdk, but also by htsjdk being inconsistent in a really bad way. If an info field has number "A", it gets dropped as a string: here I am printing the class name and value of "AC" in the 1st line of sample.vcf java.lang.String 89.
If the number is "2" like in the T2D vcf, htsjdk puts it in an array list, like CIEND in the T2D VCF: java.util.ArrayList [-80, 73]

[tpoterba]

Fixed in dffe287