Skip to content

Implement return_field() for imported Rust UDFs #5251

Description

@andygrove

Follow-up from review of #4459 (thread).

ImportedCScalarUdf::return_type already calls the kernel through the C ABI, decodes an FFI_ArrowSchema into a full Field, and then throws everything but the DataType away. The planner separately hardcodes the output field as nullable:

let return_field = Arc::new(Field::new(&call.name, kernel_return_type, true));

So a kernel that reports a non-nullable output, or attaches field metadata, has both discarded.

Implementing ScalarUDFImpl::return_field instead would carry the kernel's own Field through to the plan, and would let the planner stop fabricating one.

Worth checking before doing it:

  • Spark UDF results are nullable in Spark's own schema, so promising DataFusion a non-nullable field needs to not trip a schema comparison at the Comet boundary or in the shuffle writer.
  • The declared-vs-actual return type check in planner.rs deliberately erases nested nullability, because Spark carries containsNull in the type while the delivered array normalizes children to nullable. Whatever this does with top-level nullability should be consistent with that.
  • A test for a kernel returning a non-nullable field, since nothing exercises that today.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions