Skip to content

perf: Cast entire Date32 array to Date64 on 1st failure#21948

Open
huymq1710 wants to merge 4 commits intoapache:mainfrom
huymq1710:cast-entire-date32-array
Open

perf: Cast entire Date32 array to Date64 on 1st failure#21948
huymq1710 wants to merge 4 commits intoapache:mainfrom
huymq1710:cast-entire-date32-array

Conversation

@huymq1710
Copy link
Copy Markdown

@huymq1710 huymq1710 commented Apr 30, 2026

Which issue does this PR close?

Rationale for this change

// When `Date32` formatting fails (due to time specifiers),
// the code called cast individually for each row instead of converting the entire array
let date64_value = cast(&data_array.slice(idx, 1), &Date64)?;

What changes are included in this PR?

Cast the entire Date32 array to Date64 once on first failure, instead of per-row

cast(data_array.as_ref(), &Date64

Are these changes tested?

Yes

  • Add 2 benchmarks:
    • Date32 + datetime patterns (all rows trigger the fallback)
    • Date32 + mixed patterns (roughly half do)
  ┌─────────────────────────────────────┬───────────────────┬───────────┐
  │              Benchmark              │      Format       │  Change   │
  ├─────────────────────────────────────┼───────────────────┼───────────┤
  │ array_date_only_patterns_1000       │ array, date-only  │ No change │
  ├─────────────────────────────────────┼───────────────────┼───────────┤
  │ array_datetime_patterns_1000        │ array, datetime   │ -3.21%    │
  ├─────────────────────────────────────┼───────────────────┼───────────┤
  │ array_mixed_patterns_1000           │ array, mixed      │ +1.41%    │
  ├─────────────────────────────────────┼───────────────────┼───────────┤
  │ scalar_date_only_pattern_1000       │ scalar, date-only │ No change │
  ├─────────────────────────────────────┼───────────────────┼───────────┤
  │ scalar_datetime_pattern_1000        │ scalar, datetime  │ No change │
  ├─────────────────────────────────────┼───────────────────┼───────────┤
  │ array_date32_datetime_patterns_1000 │ array, datetime   │ -20.24%   │
  ├─────────────────────────────────────┼───────────────────┼───────────┤
  │ array_date32_mixed_patterns_1000    │ array, mixed      │ -15.74%   │
  └─────────────────────────────────────┴───────────────────┴───────────┘
Detail
cargo bench --bench to_char          
   Compiling datafusion-functions v53.1.0 (/Users/qmac/Documents/GitHub/datafusion/datafusion/functions)
    Finished `bench` profile [optimized] target(s) in 1m 10s
     Running benches/to_char.rs (target/release/deps/to_char-d2acea4b7a3e2fba)
Gnuplot not found, using plotters backend
Benchmarking to_char_array_date_only_patterns_1000: Warming up for 3.00Benchmarking to_char_array_date_only_patterns_1000: Collecting 100 sampto_char_array_date_only_patterns_1000
                        time:   [104.01 µs 104.37 µs 104.74 µs]
                        change: [−0.5692% +0.3244% +1.0788%] (p = 0.47 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild

Benchmarking to_char_array_datetime_patterns_1000: Warming up for 3.000Benchmarking to_char_array_datetime_patterns_1000: Collecting 100 samplto_char_array_datetime_patterns_1000
                        time:   [165.96 µs 166.43 µs 166.94 µs]
                        change: [−4.1660% −3.2055% −2.3626%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  1 (1.00%) high severe

Benchmarking to_char_array_mixed_patterns_1000: Collecting 100 samples to_char_array_mixed_patterns_1000
                        time:   [139.58 µs 140.02 µs 140.49 µs]
                        change: [+1.0116% +1.4122% +1.8178%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild

Benchmarking to_char_scalar_date_only_pattern_1000: Warming up for 3.00Benchmarking to_char_scalar_date_only_pattern_1000: Collecting 100 sampto_char_scalar_date_only_pattern_1000
                        time:   [60.073 µs 63.184 µs 66.309 µs]
                        change: [−4.0228% +0.6839% +6.1581%] (p = 0.79 > 0.05)
                        No change in performance detected.

Benchmarking to_char_scalar_datetime_pattern_1000: Warming up for 3.000Benchmarking to_char_scalar_datetime_pattern_1000: Collecting 100 samplto_char_scalar_datetime_pattern_1000
                        time:   [117.65 µs 124.76 µs 131.84 µs]
                        change: [−3.3009% +2.6889% +8.6148%] (p = 0.37 > 0.05)
                        No change in performance detected.
Found 30 outliers among 100 measurements (30.00%)
  16 (16.00%) low mild
  14 (14.00%) high mild

Benchmarking to_char_array_date32_datetime_patterns_1000: Warming up foBenchmarking to_char_array_date32_datetime_patterns_1000: Collecting 10to_char_array_date32_datetime_patterns_1000
                        time:   [289.67 µs 290.77 µs 291.88 µs]
                        change: [−20.661% −20.240% −19.863%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking to_char_array_date32_mixed_patterns_1000: Warming up for 3Benchmarking to_char_array_date32_mixed_patterns_1000: Collecting 100 sto_char_array_date32_mixed_patterns_1000
                        time:   [194.47 µs 195.28 µs 196.11 µs]
                        change: [−16.230% −15.738% −15.285%] (p = 0.00 < 0.05)
                        Performance has improved.

Are there any user-facing changes?

No

@github-actions github-actions Bot added the functions Changes to functions implementation label Apr 30, 2026
@huymq1710 huymq1710 changed the title Cast entire date32 array perf: Cast entire Date32 array to Date64 on 1st failure Apr 30, 2026
Comment thread datafusion/functions/benches/to_char.rs Outdated
Comment thread datafusion/functions/src/datetime/to_char.rs
Comment thread datafusion/functions/src/datetime/to_char.rs
@huymq1710
Copy link
Copy Markdown
Author

Thank you. I updated

@kumarUjjawal kumarUjjawal requested a review from alamb May 2, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize to_char for array conversions

2 participants