minor: make HigherOrderSignature less error-prone

[gstvg]

Which issue does this PR close?

Follow up of #21679

Rationale for this change

As noted by @LiaCastaneda in #21679 (comment), the higher-order signature can be made less error prone by removing the need to set the coerce_values_for_lambdas field when coerce_values_for_lambdas should be called

What changes are included in this PR?

Remove HigherOrderSignature.coerce_values_for_lambdas/with_coerce_values_for_lambdas and modify HigherOrderUDF::coerce_values_for_lambdas return from Result<Vec<DataType>> to Result<Option<Vec<DataType>>>, and it's default implementation which now returns Ok(None) instead of an error

Are these changes tested?

Existing test cover the change

Are there any user-facing changes?

To unreleased items only

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-expr v53.1.0 (current)
       Built [  24.563s] (current)
     Parsing datafusion-expr v53.1.0 (current)
      Parsed [   0.065s] (current)
    Building datafusion-expr v53.1.0 (baseline)
       Built [  24.374s] (baseline)
     Parsing datafusion-expr v53.1.0 (baseline)
      Parsed [   0.066s] (baseline)
    Checking datafusion-expr v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   1.465s] 222 checks: 219 pass, 2 fail, 1 warn, 30 skip

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/inherent_method_missing.ron

Failed in:
  HigherOrderSignature::with_coerce_values_for_lambdas, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/5fb0c109749870b1b5b4e5eb915614403ca13208/datafusion/expr/src/higher_order_function.rs:130

--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---

Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/struct_pub_field_missing.ron

Failed in:
  field coerce_values_for_lambdas of struct HigherOrderSignature, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/5fb0c109749870b1b5b4e5eb915614403ca13208/datafusion/expr/src/higher_order_function.rs:78

--- warning partial_ord_struct_fields_reordered: struct fields reordered in #[derive(PartialOrd)] struct ---

Description:
A public struct that derives PartialOrd had its fields reordered. #[derive(PartialOrd)] uses the field order to set the struct's ordering behavior, so this change may break downstream code that relies on the previous order.
        ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/partial_ord_struct_fields_reordered.ron

Failed in:
  HigherOrderSignature.lambda_parameters_max_iterations moved from position 4 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/higher_order_function.rs:80

     Summary semver requires new major version: 2 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [  51.823s] datafusion-expr

[kumarUjjawal]

Should this be updated now?

[gstvg]

Yes, thank you 030c2fb

[kosiew]

@gstvg

Thanks for the cleanup here. I did not find any blocking issues, just a couple of small doc/example updates that would make the new API shape clearer.

[kosiew]

Small docs nit: this still describes lambda output coercion as being defined by the signature, but this PR now makes coerce_values_for_lambdas opt in by returning Some(...).

Could you update this and the sibling comment above value_fields_with_higher_order_udf to mention that the method is called and may return None? That would keep the public API docs aligned with the new invariant.

[gstvg]

Thanks @kosiew, missed that one ba11072
Just to make sure, I realized the old value_fields_with_higher_order_udf docs were imprecise, it should have been "Note this does not invokes coerce_values_for_lambdas even if requested by the function signature", so even with the new API, value_fields_with_higher_order_udf still doesn't call coerce_values_for_lambdas because it's used when lambdas output field aren't know yet

[kosiew]

This still asserts a bare Vec<DataType>, but coerce_values_for_lambdas now returns Result<Option<Vec<DataType>>>.

Could you change the expected value to Some(vec![...]) so readers copying the example see the current API shape?

[gstvg]

Also at ba11072