fix: propagate field metadata through coalesce, nvl, and nvl2

[james-willis]

Which issue does this PR close?

• Part of CASE expressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860.

Rationale for this change

COALESCE, NVL/IFNULL, and NVL2 drop the field metadata of their arguments when computing their return field: return_field_from_args builds the output Field from the argument data types only. For columns carrying Arrow extension types (ARROW:extension:name / ARROW:extension:metadata — e.g. arrow.uuid, geoarrow.wkb), the planned schema of any expression containing these functions silently loses the extension-type identity, and metadata-aware UDFs that dispatch on their argument Fields fail to plan over them (see #24860 for a full reproducer).

This is one of several independent metadata drop sites listed in #24860. It is the plan-time schema fix for the conditional functions; it does not by itself change execution-time behavior, because these functions simplify into CASE, which has its own drop sites (also #24860) that I intend to address in follow-up PRs.

What changes are included in this PR?

• A small unanimous_metadata helper in datafusion/functions/src/utils.rs: returns the metadata shared by every argument field that can contribute a value to the result. Fields with a Null data type (untyped NULL literals) are ignored; if the remaining fields disagree, the result carries no metadata rather than claiming a type identity that only some inputs have.
• coalesce: return_field_from_args now attaches the unanimous metadata of its arguments. Return type and nullability computation are unchanged. NVL/IFNULL delegate to coalesce and are fixed by the same change.
• nvl2: same, over the second and third arguments only — the first argument is only tested for NULL and never contributes a value, so its metadata is excluded.

What is the testing strategy for this PR?

Unit tests in coalesce.rs and nvl2.rs directly exercise return_field_from_args: metadata propagated when the value arguments agree, dropped when they disagree, untyped NULL arguments not blocking propagation, NVL2's test argument excluded, and nullability unchanged.

There is deliberately no sqllogictest: arrow_metadata() reads its argument field at execution time, and at execution these functions have been simplified into CASE, whose own metadata drops (#24860) still lose the metadata en route. End-to-end SLT coverage lands with the CASE fix. (This ordering is safe: the optimizer's schema invariant intentionally ignores metadata — assert_expected_schema / logically_equivalent_names_and_types — so a plan whose coalesce carries metadata that the simplified CASE does not does not error; it just degrades to today's behavior until the CASE fix lands.)

Are there any user-facing changes?

The planned schema (e.g. DataFrame::schema()) of expressions containing COALESCE/NVL/IFNULL/NVL2 over metadata-bearing columns now preserves that metadata. No API changes.

---

cc @paleolimbot — this is the first slice of #24860, in the same family as your #22112; you'd mentioned on #21984 you could help shepherd this class of metadata fixes. This is my first DataFusion PR, so CI will need a committer to trigger it.

[james-willis]

In this PR, if the branches of the coalesce, etc don't agree on the metadata, we drop extension metadata defensively.

My opinion is that this should actually be a planning time error unless all rows will hit only one branch but wanted to take the more conservative approach for now. I want to solicit opinions from the reviewers on what the desried behavior is here.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.62%. Comparing base (fefc227) to head (b48d6ba).

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24865    +/-   ##
========================================
  Coverage   81.61%   81.62%            
========================================
  Files        1123     1123            
  Lines      409392   409506   +114     
  Branches   409392   409506   +114     
========================================
+ Hits       334143   334245   +102     
- Misses      55630    55637     +7     
- Partials    19619    19624     +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[paleolimbot]

Thank you for this fix! I'm not a committer so I can't help with the CI but I'll review from a metadata perspective.

I think the family of functions that are affected here are ones that have >1 "compatible" (castable to a common type) inputs. While there are some extension aware type systems that can handle this (notably: vctrs via vec_ptype2() and vec_cast()), datafusion has no mechanism for that and so probably the best we can do for now is to propagate agressively and rely on engines to write analyzer or optimizer rules to enforce other behaviour.

One other place I've seen this pattern is combining values in a values literal clause. I would personally love if the logic were the same there but that might insert too much scope here.

datafusion/datafusion/expr/src/logical_plan/builder.rs

Lines 309 to 320 in d02bd78

	let metadata = value.metadata(&schema)?;
	if let Some(ref cm) = common_metadata {
	if &metadata != cm {
	return plan_err!(
	"Inconsistent metadata across values list at row {i} column {j}. Was {:?} but found {:?}",
	cm,
	metadata
	);
	}
	} else {
	common_metadata = Some(metadata.clone());
	}

I am curious what @timsaucer would expect to happen to metadata in CASE WHEN since that is a metadata usage I'm less familiar with!

[paleolimbot]

I think this may be too strict for some practical things that can happen here. Notably, extension types that use JSON for parameters can have non-equal JSON bytes, and unrelated field metadata that arrived unbeknownst to a user (e.g., was present in an Arrow file or embedded Arrow schema in a Parquet file) might not agree and could break existing (and completely valid) queries.

This should probably be a very lenient combination (possibly just merging all metadata keys or returning the first encountered metadata)...engines can insert extra constraints here if they would like to but if the return field drops the extension name, extension aware functions will fail (like ours did!).

As an example, we would probably write an optimizer rule that checks for equal SedonaTypes for a case when or ifelse; however, if the return type of the case when doesn't carry extension metadata, it will fail before the optimizer rule can run.

This may also be more appropriate next to some related utilities:

datafusion/datafusion/common/src/metadata.rs

Lines 75 to 86 in d02bd78

	/// Assert equality of data types where one or both sides may have field metadata
	///
	/// This currently compares absent metadata (e.g., one side was a DataType) and
	/// empty metadata (e.g., one side was a field where the field had no metadata)
	/// as equal and uses byte-for-byte comparison for the keys and values of the
	/// fields, even though this is potentially too strict for some cases (e.g.,
	/// extension types where extension metadata is represented by JSON, or cases
	/// where field metadata is orthogonal to the interpretation of the data type).
	///
	/// Returns a planning error with suitably formatted type representations if
	/// actual and expected do not compare to equal.
	pub fn check_metadata_with_storage_equal(

[james-willis]

Thanks Dewey! This is good to consider. I will allow some more time for any other folks to chime in if they would like.