perf: optimize `date_trunc` by andygrove · Pull Request #23542 · apache/datafusion

andygrove · 2026-07-14T01:18:14Z

Which issue does this PR close?

N/A

Rationale for this change

Optimize existing expression.

What changes are included in this PR?

Replaced the per-row chrono NaiveDateTime round trip in the untimezoned coarse path of date_trunc (week/month/quarter/year) with integer civil-calendar arithmetic, cutting several allocationless-but-validating chrono field setters down to a few integer ops per row.

Are these changes tested?

Existing tests.

Benchmark (criterion):

date_trunc_month_nanos_1000: 48.452% faster (base 19164ns -> cand 9878ns)
date_trunc_week_nanos_1000: 76.045% faster (base 23337ns -> cand 5590ns)
date_trunc_month_second_1000: 47.688% faster (base 17376ns -> cand 9090ns)
date_trunc_quarter_nanos_1000: 32.493% faster (base 22005ns -> cand 14854ns)
date_trunc_year_nanos_1000: 39.449% faster (base 20460ns -> cand 12388ns)
date_trunc_minute_1000: 1.2% faster (base 654ns -> cand 647ns)

Full criterion output:

date_trunc_minute_1000  time:   [645.50 ns 652.66 ns 659.52 ns]
                        change: [−2.0837% −1.1996% −0.2678%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

date_trunc_month_second_1000
                        time:   [9.0984 µs 9.1139 µs 9.1298 µs]
                        change: [−47.773% −47.688% −47.597%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

date_trunc_week_nanos_1000
                        time:   [5.5882 µs 5.6103 µs 5.6308 µs]
                        change: [−76.154% −76.045% −75.951%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low mild

date_trunc_month_nanos_1000
                        time:   [9.8640 µs 9.8831 µs 9.9059 µs]
                        change: [−48.576% −48.452% −48.309%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

date_trunc_quarter_nanos_1000
                        time:   [14.849 µs 14.857 µs 14.869 µs]
                        change: [−32.666% −32.493% −32.339%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 16 outliers among 100 measurements (16.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  7 (7.00%) high mild
  6 (6.00%) high severe

date_trunc_year_nanos_1000
                        time:   [12.385 µs 12.391 µs 12.398 µs]
                        change: [−39.498% −39.449% −39.405%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
  3 (3.00%) high severe

Are there any user-facing changes?

gabotechs · 2026-07-14T13:20:46Z

run benchmarks date_trunc

adriangbot · 2026-07-14T13:23:37Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4969613851-1050-wp77d 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing auto-opt/date_trunc-datafusion-20260713-184055 (bce729e) to 3a29d6b (merge-base) diff using: date_trunc
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-07-14T13:27:58Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                            HEAD                                   auto-opt_date_trunc-datafusion-20260713-184055
-----                            ----                                   ----------------------------------------------
date_trunc_minute_1000           1.00      3.1±0.00µs        ? ?/sec    1.00      3.1±0.00µs        ? ?/sec
date_trunc_month_nanos_1000                                             1.00     11.8±0.03µs        ? ?/sec
date_trunc_month_second_1000                                            1.00     16.3±0.01µs        ? ?/sec
date_trunc_quarter_nanos_1000                                           1.00     17.2±0.02µs        ? ?/sec
date_trunc_week_nanos_1000                                              1.00      6.1±0.01µs        ? ?/sec
date_trunc_year_nanos_1000                                              1.00     14.1±0.01µs        ? ?/sec

Resource Usage

date_trunc — base (merge-base)

Metric	Value
Wall time	105.0s
Peak memory	5.6 MiB
Avg memory	522.5 KiB
CPU user	7.7s
CPU sys	0.0s
Peak spill	0 B

date_trunc — branch

Metric	Value
Wall time	145.0s
Peak memory	26.8 MiB
Avg memory	8.0 MiB
CPU user	65.6s
CPU sys	0.1s
Peak spill	0 B

File an issue against this benchmark runner

gabotechs · 2026-07-14T13:39:12Z

🤔 As main does not have the benchmarks, the bot cannot perform the comparison. I run the benches myself and can replicate locally, so it's all good 👍

Benchmarking date_trunc_minute_1000: Warming up forBenchmarking date_trunc_minute_1000: Collecting 100date_trunc_minute_1000  time:   [694.47 ns 699.78 ns 704.89 ns]
                        change: [−6.8407% −6.2167% −5.5712%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking date_trunc_month_second_1000: Warming Benchmarking date_trunc_month_second_1000: Collecting 100 samples in estimated 5.0138 s (500k iteratioBenchmarking date_trunc_month_second_1000: Analyzindate_trunc_month_second_1000
                        time:   [9.8196 µs 9.8537 µs 9.8944 µs]
                        change: [−68.766% −66.918% −65.032%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking date_trunc_week_nanos_1000: Warming upBenchmarking date_trunc_week_nanos_1000: Collecting 100 samples in estimated 5.0005 s (869k iterationsdate_trunc_week_nanos_1000
                        time:   [5.7474 µs 5.7796 µs 5.8186 µs]
                        change: [−78.538% −78.362% −78.193%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking date_trunc_month_nanos_1000: Warming uBenchmarking date_trunc_month_nanos_1000: Collecting 100 samples in estimated 5.0166 s (465k iterationdate_trunc_month_nanos_1000
                        time:   [10.598 µs 10.634 µs 10.676 µs]
                        change: [−49.751% −49.509% −49.275%] (p = 0.00 < 0.05)
                        Performance has improved.

date_trunc_quarter_nanos_1000
                        time:   [16.065 µs 16.086 µs 16.111 µs]
                        change: [−32.441% −32.229% −32.017%] (p = 0.00 < 0.05)
                        Performance has improved.

date_trunc_year_nanos_1000
                        time:   [13.355 µs 13.404 µs 13.462 µs]
                        change: [−40.818% −40.571% −40.316%] (p = 0.00 < 0.05)
                        Performance has improved.

gabotechs

👍 pretty smart!

gabotechs · 2026-07-14T13:35:29Z

+fn civil_from_days(days: i64) -> (i64, i64, i64) {
+    let z = days + DAYS_EPOCH_SHIFT;
+    let era = z.div_euclid(DAYS_PER_ERA);
+    let day_of_era = z.rem_euclid(DAYS_PER_ERA);
+    let year_of_era = (day_of_era - day_of_era / 1460 + day_of_era / 36524
+        - day_of_era / 146_096)
+        / 365;
+    let day_of_year =
+        day_of_era - (365 * year_of_era + year_of_era / 4 - year_of_era / 100);
+    // Month index with March as 0, so that the leap day falls at the end of the year.
+    let month_index = (5 * day_of_year + 2) / 153;
+    let day = day_of_year - (153 * month_index + 2) / 5 + 1;
+    let month = if month_index < 10 {
+        month_index + 3
    } else {
-        _date_trunc_coarse::<NaiveDateTime>(granularity, None)?;
-        Ok(None)
-    }
+        month_index - 9
+    };
+    let year = year_of_era + era * 400 + i64::from(month <= 2);
+    (year, month, day)
 }



This math looks correct. Is there any link we can put in the comment so that people and robots can have a reference in case this code needs some maintenance?

For example, I found these:

https://howardhinnant.github.io/date_algorithms.html#days_from_civil

https://howardhinnant.github.io/date_algorithms.html#civil_from_days

…dar helpers

gabotechs · 2026-07-14T15:00:36Z

Nice, thanks @andygrove!

andygrove · 2026-07-14T15:55:39Z

Thanks for the review @gabotechs!

perf: optimize date_trunc in datafusion-functions

bce729e

github-actions Bot added the functions Changes to functions implementation label Jul 14, 2026

andygrove changed the title ~~perf: optimize date_trunc in datafusion-functions~~ perf: optimize date_trunc Jul 14, 2026

andygrove marked this pull request as ready for review July 14, 2026 09:12

gabotechs approved these changes Jul 14, 2026

View reviewed changes

docs: cite Howard Hinnant's date algorithms in date_trunc civil calen…

3f50771

…dar helpers

gabotechs added this pull request to the merge queue Jul 14, 2026

Merged via the queue into apache:main with commit d194e31 Jul 14, 2026
37 checks passed

andygrove deleted the auto-opt/date_trunc-datafusion-20260713-184055 branch July 14, 2026 15:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf: optimize `date_trunc`#23542

perf: optimize `date_trunc`#23542
gabotechs merged 2 commits into
apache:mainfrom
andygrove:auto-opt/date_trunc-datafusion-20260713-184055

andygrove commented Jul 14, 2026 •

edited

Loading

Uh oh!

gabotechs commented Jul 14, 2026

Uh oh!

adriangbot commented Jul 14, 2026

Uh oh!

adriangbot commented Jul 14, 2026

Uh oh!

gabotechs commented Jul 14, 2026

Uh oh!

gabotechs left a comment

Uh oh!

gabotechs Jul 14, 2026 •

edited

Loading

Uh oh!

gabotechs commented Jul 14, 2026

Uh oh!

Uh oh!

andygrove commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

andygrove commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

gabotechs commented Jul 14, 2026

Uh oh!

adriangbot commented Jul 14, 2026

Uh oh!

adriangbot commented Jul 14, 2026

Uh oh!

gabotechs commented Jul 14, 2026

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

gabotechs Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabotechs commented Jul 14, 2026

Uh oh!

Uh oh!

andygrove commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andygrove commented Jul 14, 2026 •

edited

Loading

gabotechs Jul 14, 2026 •

edited

Loading