Add Arrow Decimal Array Support - #189
Merged
Merged
Conversation
) * Wire DecimalArray into NumericArray, Array, and Scalar enums with From impls and tests (#182) Phase 2 of EPC86. Adds Decimal32/64/128 variants gated behind the `decimal` feature flag to NumericArray, Array, and Scalar enums. Cascades match arms across all dispatch sites, adds typed accessors (dec32/try_dec32, dec64/try_dec64, dec128/try_dec128), widening conversions between decimal widths, and liberal From impls for DecimalArray -> NumericArray -> Array. * Implement decimal conversion arms in NumericArray try_ accessors The try_i32/i64/u32/u64/f32/f64/bool/str methods and cow_into_f64 previously returned errors for decimal variants. Now they perform scale-aware conversions consistent with every other numeric type.
… SuperArrayV, and Consolidate (#187) - Add DecimalAVT type alias for decimal view tuples - Add Consolidate impl for Vec<DecimalAVT> with precision/scale propagation - Wire gather_decimal! macro in Vec<ArrayVT> consolidation (replaces panics) - Add Decimal32/64/128 arms to Array::inner, inner_mut, inner_check, inner_check_mut - Import DecimalArray in array.rs for the inner method type resolution - Tests for ArrayV windowing, NumericArrayV decimal views, SuperArray chunked append with type validation, SuperArrayV consolidation across chunks, null mask propagation, and Value flow-through
…on + fix null mask drop on broadcasting (#185) * Add decimal arithmetic kernels with checked overflow and auto-promotion * Propagate null mask through broadcast for all array types
* Add Decimal32 and Decimal64 round-trip tests for arrow-rs Polars only supports i128-backed decimals so Decimal32/64 round-trips are not valid through that path without widening first. * Add decimal support to pyo3 and minarrow-py Python bridges pyo3: - Add `decimal` feature flag forwarding minarrow/decimal - Map Decimal32/64/128 ArrowType variants to pa.decimal128() in arrow_type_to_pyarrow (PyArrow only supports 128-bit decimal) - Zero-copy export/import through existing Arrow C Data Interface FFI - Document decimal type mappings in the FFI module - Add Decimal128 roundtrip tests (array, high precision, RecordBatch) minarrow-py: - Add `decimal` feature flag - Add Decimal32/64/128 variants to PyArrowType enum with From impls - Add DType::Decimal variant with Numeric group membership - Add precision/scale properties on PyArray (None for non-decimal) - Handle decimal dtype strings: decimal128(P,S), decimal64(P,S), decimal32(P,S), decimal(P,S) alias for Decimal128 - Construct DecimalArray from Python int sequences with dtype param - Return Python decimal.Decimal values from element access - Scale-aware formatting in repr via format_decimal_string - 23 pytest tests covering construction, inspection, element access, display, slicing, ArrowType, and PyArrow round-trip * Make precision and scale report values for all numeric types Precision and scale are properties of numeric data, not decimal-specific. Integer types report their digit capacity with scale 0, float types report their significant-digit capacity, and decimal types report their configured precision and scale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Decimal32, Decimal64, and Decimal128 array support
This feature adds fixed-point Arrow-compliant decimal arrays for exact numeric values where floating-point approximation is not acceptable (for e.g., monetary, accounting, and high-precision numeric columns).