[hail][wip] Local ld prune ptype

[akotlar]

Tests not passing, not sure why yet. Errors are MatchError.

Stacked on #6421, will see if I can unwind

> Task :test
Running test: Test method bitPackedVectorCorrectWhenOffsetNotZero(is.hail.methods.LocalLDPruneSuite)

Gradle suite > Gradle test > is.hail.methods.LocalLDPruneSuite.bitPackedVectorCorrectWhenOffsetNotZero PASSED
Running test: Test method testBitPackUnpack(is.hail.methods.LocalLDPruneSuite)

Gradle suite > Gradle test > is.hail.methods.LocalLDPruneSuite.testBitPackUnpack FAILED
    scala.MatchError at LocalLDPruneSuite.scala:222
Running test: Test method testIsLocallyUncorrelated(is.hail.methods.LocalLDPruneSuite)

Gradle suite > Gradle test > is.hail.methods.LocalLDPruneSuite.testIsLocallyUncorrelated FAILED
    org.apache.spark.SparkException at LocalLDPruneSuite.scala:214
        Caused by: scala.MatchError
Running test: Test method testR2(is.hail.methods.LocalLDPruneSuite)

Gradle suite > Gradle test > is.hail.methods.LocalLDPruneSuite.testR2 FAILED
    scala.MatchError at LocalLDPruneSuite.scala:244
Running test: Test method testRandom(is.hail.methods.LocalLDPruneSuite)

Gradle suite > Gradle test > is.hail.methods.LocalLDPruneSuite.testRandom FAILED
    java.lang.AssertionError at LocalLDPruneSuite.scala:323
        Caused by: scala.MatchError at LocalLDPruneSuite.scala:323

cc @tpoterba

[tpoterba]

@akotlar can you rebase and remove stacked tag? Let's get this in

[akotlar]

Yep, would like to issue the infer ptype wip PR first, targeting today

[akotlar]

rebased. one merge change in object TableValue apply I wasn't sure about (a constructor deleted). tests likely failing. will follow up on both tonight.

[akotlar]

working on it now

[akotlar]

Still not satisfied with my understanding of the landscape (namely why my changes in LDPruneSuite prior to last commit were failing), but passes

[tpoterba]

small fixes

[tpoterba]

should be bpvType I think?

[tpoterba]

also the canonicalRVDType above should be RVDType(bpvType, Array("locus", "alleles"))

[akotlar]

I don't yet understand this section of code well, but making the following changes results in an error in the r2 test

(added rvb.addFields(bpvType, rv, fieldIndicesToAdd) , rvdLP.mapPartitions( RVDType(bpvType, Array("locus", "alleles")) )

 val fieldIndicesToAdd = Array("locus", "alleles", "mean", "centered_length_rec")
      .map(field => bpvType.fieldIdx(field))
    val sitesOnly = rvdLP.mapPartitions(
      RVDType(bpvType, Array("locus", "alleles"))
    )({ it =>
      val region = Region()
      val rvb = new RegionValueBuilder(region)
      val newRV = RegionValue(region)

      it.map { rv =>
        region.clear()
        rvb.set(region)
        rvb.start(bpvType)
        rvb.startStruct()
        rvb.addFields(bpvType, rv, fieldIndicesToAdd)
        rvb.endStruct()
        newRV.setOffset(rvb.end())
        newRV
      }
    })

Gradle suite > Gradle test > is.hail.methods.LocalLDPruneSuite.testIsLocallyUncorrelated FAILED
    java.lang.IllegalArgumentException at LocalLDPruneSuite.scala:355
Running test: Test method testR2(is.hail.methods.LocalLDPruneSuite)

Tired, not thinking clearly

[akotlar]

Will finish digging through tomorrow:

java.lang.IllegalArgumentException: requirement failed
	at scala.Predef$.require(Predef.scala:212)
	at is.hail.expr.ir.TableValue.<init>(TableValue.scala:45)
	at is.hail.methods.LocalLDPrune.execute(LocalLDPrune.scala:368)
	at is.hail.methods.LocalLDPrune$$anonfun$apply$3.apply(LocalLDPrune.scala:286)
	at is.hail.methods.LocalLDPrune$$anonfun$apply$3.apply(LocalLDPrune.scala:284)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:8)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:7)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.annotations.Region$.scoped(Region.scala:11)
	at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:7)
	at is.hail.methods.LocalLDPrune$.apply(LocalLDPrune.scala:284)
	at is.hail.methods.LocalLDPruneSuite.testIsLocallyUncorrelated(LocalLDPruneSuite.scala:355)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess

[akotlar]

With this issue I think this PR is ok?

addressed?

[tpoterba]

needs a bump