[vds/combiner] Better calculation of ref_block_max_length #14178
Conversation
Use the patch in place function of `store_ref_block_max_length` to compute `ref_block_max_length` rather than computing it on a zip join pipeline, causing that zip join pipeline to be executed twice.
| ) | ||
|
|
||
| vds.write(self._output_path) | ||
| hl.vds.store_ref_block_max_length(self._output_path) |
There was a problem hiding this comment.
I agree we shouldn't do this pass twice if not necessary, but is it not possible to compute this as we go so that we can just store this information in the globals? This feels like a hack that is pointing at some limitation of the Hail model.
There was a problem hiding this comment.
It depends on the precise behavior of _localize=False in aggregate_entries if that causes one pass, then we're good, otherwise this is better as computing this from a full vds is better than redoing the join twice.
There was a problem hiding this comment.
OK, let's do the not terrible thing but I think this reveals a deficiency in the Hail Query language. Following up in Zulip: https://hail.zulipchat.com/#narrow/stream/123011-Hail-Query-Dev/topic/write.20.2B.20aggregate
Use the patch in place function of
store_ref_block_max_lengthto computeref_block_max_lengthrather than computing it on a zip join pipeline, causing that zip join pipeline to be executed twice.