Pushdown compare with NaN broken

[a10y]

We implement our compare binary comparison operation using the arrow_ord kernels. This uses the f32/f64 total_cmp, which includes NaN as >= all positive normal numbers.

This causes inconsistent behavior when pushing down a comparison operation against NaN to a file column that contains NaN.

Directly evaluating the following comparison results in [false, false, true, true]:

PrimitiveArray([1.0, 2.0, NaN, NaN]) >= f32::NAN

However, this is not the case when the filter is pushed down into a scan, because our min/max stats do not consider NaN. The pruning expression that gets pushed down is

$.max < f32::NAN --> 2.0 < f32::NAN = true

which causes the whole thing to get pruned and the result becomes [false, false, false, false]

vortex/vortex-array/src/arrays/primitive/compute/min_max.rs

Lines 43 to 48 in bba0e63

	fn compute_min_max<'a, T>(iter: impl Iterator<Item = &'a T>, dtype: &DType) -> Option<MinMaxResult>
	where
	T: Into<ScalarValue> + NativePType,
	{
	// `total_compare` function provides a total ordering (even for NaN values).
	// However, we exclude NaNs from min max as they're not useful for any purpose where min/max would be used

We need to either

1. Make the stats contain NaN, and preserve the total_cmp ordering in pushdown
2. Have a fallback compare kernel for float arrays with NaNCount > 0 that masks out the NaNs to false

[lwwmanning]

Prefer 2, I think. NaN is generally awful, let's avoid inflicting it on stats & stats consumers

[a10y]

Actually while I'm here, I suppose +/- Infinity also fall into the same bucket here

EDIT Actually infinity is included in the stats calculation, so it's just NaNs really. Maybe +/- 0.0 are still a problem though

[lwwmanning]

Yeah, I think +/- Infinity are well-behaved. +/- 0.0 should also be fine... the only case I can think of where we'd care is if we were doing a signed-to-unsigned conversion, but floats are always signed.

[a10y]

Poking a bit more at the behavior of other systems, DataFusion and DuckDB both seem to support filter comparisons against NaN. I had assumed that attempting to compare against NaN would always return false but apparently not.

So actually, the thing we need to do here is not mask out NaNs, but actually fix the pruning behavior to consult the NaNCount stat in addition to min/max at pruning time.