feat: add RunEndEncoded variant to ScalarValue enum

[xsa-dev]

Which issue does this PR close?

• Closes Missing ScalarValue variant for RunEndEncoded #18563

Rationale for this change

This PR addresses the missing RunEndEncoded variant in the ScalarValue enum, which is essential for proper type coercion and operations involving RunEndEncoded arrays. RunEndEncoded (REE) is an Arrow data type for run-length encoding of repeated values, and without proper ScalarValue support, operations involving REE arrays would fail to work correctly.

The implementation follows the existing patterns in the DataFusion codebase and ensures compatibility with the Arrow specification for RunEndEncoded arrays.

What changes are included in this PR?

1. Added RunEndEncoded variant to ScalarValue enum in datafusion/common/src/scalar/mod.rs

   • New variant: RunEndEncoded(Arc<RunEndEncodedScalar>)
   • Positioned correctly within the enum definition

2. Implemented RunEndEncodedScalar struct with required trait implementations:

   • PartialEq and Eq for value equality comparisons
   • Hash for hash-based operations
   • PartialOrd for ordering operations
   • Debug and Clone for debugging and copying

3. Updated existing trait implementations to handle the new variant:

   • PartialEq for ScalarValue
   • PartialOrd for ScalarValue
   • Hash for ScalarValue

4. Added support for null values in try_new_null() method

5. Comprehensive test coverage with unit tests covering:

   • Value equality and inequality
   • Hash consistency
   • Partial ordering
   • Null value handling
   • Type safety

Are these changes tested?

Yes, this PR includes comprehensive test coverage:

• Unit tests for RunEndEncodedScalar covering all trait implementations
• Integration tests for ScalarValue enum operations with the new variant
• Null value tests to ensure proper handling of null RunEndEncoded scalars
• Edge case tests for various value types and comparison scenarios

All tests pass successfully and the implementation maintains compatibility with existing functionality.

Are there any user-facing changes?

Yes, this PR introduces user-facing changes by extending the public API:

New Public API:

• ScalarValue::RunEndEncoded(Arc<RunEndEncodedScalar>) - New enum variant
• RunEndEncodedScalar struct with public run_ends and values fields

User Impact:

• ✅ Positive: Users can now work with RunEndEncoded scalars in DataFusion queries
• ✅ Backward Compatible: Existing code continues to work unchanged
• ✅ Type Safe: Proper type support for REE operations

No Breaking Changes:

• All existing APIs remain unchanged
• No modifications to public method signatures
• No changes to existing behavior

The changes follow DataFusion's API evolution guidelines and are fully backward compatible.

[Jefffrey]

I don't think it makes sense for the RunEndEncoded scalar to hold a reference to arrays; scalars are meant to be single values, and run end encoding is similar to dictionary encoding in that it is a way to encode another array (e.g. we can have Int32 array, or we could represent the same exact array (*without nulls) as a RunEndEncoded Int32 array)

I suggest reading the documentation on run end encoding to get a better understanding of what it actually represents:

• https://arrow.apache.org/docs/format/Columnar.html#run-end-encoded-layout
• https://docs.rs/arrow/latest/arrow/array/struct.RunArray.html