Skip to content

v0.8.10

Choose a tag to compare

View all tags
@rustyconover rustyconover released this 29 Jun 19:27
· 219 commits to main since this release
v0.8.10
d52b654

Performance: scalar UDF constants can now be carried through as typed Arrow scalars.

  • ConstParam annotated with a pyarrow scalar class — Annotated[pa.Int64Scalar, ConstParam(...)] — now delivers the typed pa.Scalar straight to compute() instead of converting it to a Python value. The constant can be passed directly into pyarrow.compute with no per-call scalar re-inference, which is ~40–50x slower on Windows than a typed scalar and dominates per-batch cost for scalar UDFs. Python-type annotations (int/str/float/…) are unchanged, so existing workers are unaffected.
  • New vgi.arguments.SCALAR_CLASS_TO_DATATYPE (scalar class → Arrow type), used to infer the Arrow type and to mark which constants are delivered as scalars.
  • Example fixture MultiplyFunction updated to the idiomatic fast pattern.
  • CI: boot the http integration worker from the staging dir so copy_from/copy_to tests resolve their relative paths (fixes the http-lane Integration suite).
Assets 2
Loading