GloWGR scaling improvements

[karenfeng]

What changes are proposed in this pull request?

• Pushes validation that each row of variants contains the same number of genotype values lazily into the block transformer.
• Disable automatic broadcast merges that are likely to OOM, such asCaused by: org.apache.spark.sql.execution.OutOfMemorySparkException: Size of broadcasted table far exceeds estimates and exceeds limit of spark.driver.maxResultSize=4294967296. You can disable broadcasts for this query using set spark.sql.autoBroadcastJoinThreshold=-1
• Adds logging during transform_loco for the per-chromosome predictions.
• Adds documentation advising against performing multiallelic splitting in the same query as matrix blocking, related to Document disabling whole-stage codegen during GloWGR data prep #280.
• Adds documentation regarding scalability limits. PyArrow indexes its elements with integers; as a result, the size of the vector buffer is limited to the maximum value of an integer (2,147,483,647). Vectors are allocated in sizes that are powers of two; the largest power-of-two below this limit is 1,073,741,824. Each vector buffer contains a data buffer and a validity buffer. The size of the data buffer of 8-byte floats is (# elements * 8) and the size of the validity buffer is (# elements + 63) >> 6) << 3. Therefore, the maximum number of elements is 132,152,839. The bottleneck of elements in each sample block/label pair is determined by the following equation: (# alphas) * (# SNPs/ # SNPs per block) * (# samples / # sample blocks). The floats are stored in an array of length (# samples / # sample blocks) in each row. There are (# SNPs/ # SNPs per block) * (# alphas) rows.

How is this patch tested?

• Unit tests
• Integration tests
• Manual tests

[codecov]

Codecov Report

Merging #282 into master will decrease coverage by 0.12%.
The diff coverage is 96.15%.

@@            Coverage Diff             @@
##           master     #282      +/-   ##
==========================================
- Coverage   93.76%   93.64%   -0.13%     
==========================================
  Files          94       92       -2     
  Lines        4681     4403     -278     
  Branches      447      400      -47     
==========================================
- Hits         4389     4123     -266     
+ Misses        292      280      -12     

Impacted Files	Coverage Δ
...o/projectglow/sql/expressions/VariantQcExprs.scala	88.12% <95.45%> (+1.16%)	⬆️
...ckvariantsandsamples/VariantSampleBlockMaker.scala	100.00% <100.00%> (ø)
...in/scala/io/projectglow/vcf/TabixIndexHelper.scala	83.85% <0.00%> (-0.49%)	⬇️
.../main/scala/io/projectglow/vcf/VCFFileFormat.scala	97.33% <0.00%> (-0.17%)	⬇️
...low/vcf/InternalRowToVariantContextConverter.scala	93.08% <0.00%> (-0.10%)	⬇️
...low/vcf/VariantContextToInternalRowConverter.scala	96.50% <0.00%> (-0.02%)	⬇️
...rojectglow/vcf/VCFLineToInternalRowConverter.scala
...e/src/main/scala/io/projectglow/sql/GlowConf.scala

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e18444f...1d299f0. Read the comment docs.

[henrydavidge]

What does the error message look like here?

[karenfeng]

It looks like this: java.lang.RuntimeException: '(size(array_repeat(0.0, input[4, int, true]), true) = 17190)' is not true!

[henrydavidge]

Hm, that's an inscrutable error message. If we give the column an informative name, does the alias appear in the error message?

[karenfeng]

The column is already called values; maybe I can re-throw with a more helpful message.

[karenfeng]

Actually, I'm not sure if we can re-throw in this case as the issue doesn't manifest until the DataFrame is materialized.

[henrydavidge]

Sorry, had a few more comments.

[henrydavidge]

Hm, actually I think it might be better to exclude this from the Python / Scala APIs. I think the use case is pretty narrow, and since this will be in the next release of Spark, I don't think there's much upside to exposing it in our public API.

You can set the exclude_python option in the YAML file. We should add an analogous exclude_scala flag. I can take care of that if you don't have time.

[henrydavidge]

How about "At least one row has an inconsistent number of values (expected x). Please verify that each row contains the same number of values."

[henrydavidge]

Nice! LGTM