[query] when sending large literals from Python to Scala, Query should use a compact binary format

[danking]

What happened?

We encode literals from Python as JSON. This dramatically inflates the encoding of hl.tarray(hl.tstruct(...)) because the struct fields are repeated. We should instead use EncodedLiteral. There are some subtleties:

1. We need to implement an encoder in Python. We currently have a decoder (_from_encoding and _convert_from_encoding) which implements {"name":"StreamBufferSpec"}. We should write an encoder for that.
2. Our IR is plaintext, so we'll need to encode this binary encoding as base64 so it may be included in plaintext.
3. EncodedLiteral is not currently parsable (it throws UnsupportedOperationException in ir/Parser.scala). As mentioned in (2), we'll need to implement parsing.

For (3), just represent the AbstractTypedCodecSpec in terms of a virtual type and a buffer spec (assume the buffer spec is StreamBufferSpec). When parsing, we can construct an EncodedLiteral with the base64-decoded value and a codec spec we construct like this:

val codec = TypedCodecSpec(
    EType.fromTypeAllOptional(virtualType),
    virtualType,
    BufferSpec.parseOrDefault(bufferSpecString)
)

For (1), it should be simple enough to invert the logic from _convert_from_encoding to encode rather than decode. Write tests that round-trip in Python for lots of types.

---

See also #13748

Version

0.2.142

Relevant log output

No response

[danking]

GVS team confirms their pipeline containing interval literals went from >50 GB (crashing at that point) to less than 11GB! 👏