[query] GVS team reports an OOM

[danking]

What happened?

https://github.com/broadinstitute/gatk/pull/8525/files#diff-f9dec862e55411345f08073fe67e2399a3b23877139fa41e1dc0460c23225dd5

Running this script OOMs in 0.2.123 and 0.2.124 but does not in 0.2.120, 0.2.121, and 0.2.122.

Version

0.2.124

Relevant log output

No response

[danking]

#13748

[danking]

The root cause of large memory usage is #13748 but we should be deploying Hail in a manner that has enough RAM (in this case, 4GiB is plenty). I'm following up with GVS to determine whether the JVM indeed has enough RAM.

[danking]

Dataproc submission bash script: https://gist.github.com/mcovarr/06eaecad849e979d608adf43e2118f5a
Python script: https://github.com/broadinstitute/gatk/blob/ah_var_store/scripts/variantstore/wdl/extract/filter_VDS_and_shard_by_contig.py

A few things 0.2.123 introduced:

• new gradle (which has caused other dataproc issues)
• SemanticHash

My guess is the combination of semantic hash and large JSON literals is pushing us past the 50GiB limit.

[danking]

0.2.125 was released with a fix, but it also contained a critical correctness bug. We will try again with 0.2.126.

[danking]

GVS team confirmed this is no longer an issue.