Skip to content

[PoC] Implementation of PARQUET-2249: Introduce IEEE 754 total order (no NaN counts)#9669

Draft
etseidl wants to merge 2 commits intoapache:mainfrom
etseidl:total_order_221
Draft

[PoC] Implementation of PARQUET-2249: Introduce IEEE 754 total order (no NaN counts)#9669
etseidl wants to merge 2 commits intoapache:mainfrom
etseidl:total_order_221

Conversation

@etseidl
Copy link
Copy Markdown
Contributor

@etseidl etseidl commented Apr 6, 2026

Which issue does this PR close?

PoC for apache/parquet-format#221

Rationale for this change

Submitting PR for benchmarking

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added the parquet Changes to the parquet crate label Apr 6, 2026
@etseidl
Copy link
Copy Markdown
Contributor Author

etseidl commented Apr 6, 2026

run benchmark arrow_writer

@adriangbot
Copy link
Copy Markdown

adriangbot commented Apr 6, 2026

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4195327421-914-56nxj 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu) 
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing total_order_221 (1b83e01) to 65ad652 (merge-base) diff
BENCH_NAME=arrow_writer
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

@adriangbot
Copy link
Copy Markdown

adriangbot commented Apr 6, 2026

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu) 
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details 

group                                     main                                   total_order_221
-----                                     ----                                   ---------------
bool/bloom_filter                         1.00     13.8±0.10ms    18.1 MB/sec    1.00     13.8±0.05ms    18.1 MB/sec
bool/default                              1.01     11.8±0.09ms    21.2 MB/sec    1.00     11.7±0.04ms    21.3 MB/sec
bool/parquet_2                            1.03     15.2±0.10ms    16.4 MB/sec    1.00     14.7±0.05ms    17.0 MB/sec
bool/zstd                                 1.00     12.3±0.09ms    20.3 MB/sec    1.00     12.3±0.05ms    20.4 MB/sec
bool/zstd_parquet_2                       1.03     15.6±0.09ms    16.0 MB/sec    1.00     15.2±0.05ms    16.4 MB/sec
bool_non_null/bloom_filter                1.00      6.8±0.03ms    18.4 MB/sec    1.02      6.9±0.02ms    18.1 MB/sec
bool_non_null/default                     1.00      4.1±0.02ms    30.2 MB/sec    1.02      4.2±0.02ms    29.6 MB/sec
bool_non_null/parquet_2                   1.06      8.6±0.03ms    14.5 MB/sec    1.00      8.1±0.03ms    15.4 MB/sec
bool_non_null/zstd                        1.00      4.5±0.02ms    27.8 MB/sec    1.02      4.6±0.02ms    27.4 MB/sec
bool_non_null/zstd_parquet_2              1.05      9.0±0.02ms    13.9 MB/sec    1.00      8.5±0.03ms    14.7 MB/sec
float_with_nans/bloom_filter              1.08     94.9±0.39ms   147.6 MB/sec    1.00     87.6±0.67ms   159.9 MB/sec
float_with_nans/default                   1.10     75.5±0.31ms   185.4 MB/sec    1.00     68.9±0.19ms   203.2 MB/sec
float_with_nans/parquet_2                 1.08     98.4±0.36ms   142.2 MB/sec    1.00     90.8±0.29ms   154.2 MB/sec
float_with_nans/zstd                      1.07    113.9±0.22ms   122.9 MB/sec    1.00    106.9±0.24ms   131.0 MB/sec
float_with_nans/zstd_parquet_2            1.05    136.0±0.33ms   103.0 MB/sec    1.00    129.3±1.12ms   108.2 MB/sec
list_primitive/bloom_filter               1.00    320.1±2.07ms  1703.7 MB/sec    1.24   395.4±10.65ms  1379.1 MB/sec
list_primitive/default                    1.00    240.2±3.10ms     2.2 GB/sec    1.35    323.4±8.03ms  1686.6 MB/sec
list_primitive/parquet_2                  1.00    276.7±1.10ms  1971.0 MB/sec    1.22    336.8±4.20ms  1619.0 MB/sec
list_primitive/zstd                       1.00    492.5±5.01ms  1107.4 MB/sec    1.10   540.6±16.31ms  1008.8 MB/sec
list_primitive/zstd_parquet_2             1.00    480.8±0.46ms  1134.2 MB/sec    1.14    546.6±2.19ms   997.7 MB/sec
list_primitive_non_null/bloom_filter      1.00   438.6±12.63ms  1240.9 MB/sec    1.10   483.7±32.59ms  1125.2 MB/sec
list_primitive_non_null/default           1.00   310.7±10.57ms  1751.9 MB/sec    1.14   353.8±29.73ms  1538.5 MB/sec
list_primitive_non_null/parquet_2         1.00   304.2±10.36ms  1789.0 MB/sec    1.27    386.3±6.84ms  1408.8 MB/sec
list_primitive_non_null/zstd              1.00   726.6±15.54ms   749.0 MB/sec    1.03   751.2±25.65ms   724.5 MB/sec
list_primitive_non_null/zstd_parquet_2    1.00    713.2±1.35ms   763.1 MB/sec    1.06    757.9±1.36ms   718.1 MB/sec
primitive/bloom_filter                    1.02    160.4±0.66ms   279.9 MB/sec    1.00    157.0±0.60ms   285.9 MB/sec
primitive/default                         1.02    127.1±0.55ms   353.1 MB/sec    1.00    124.9±0.42ms   359.2 MB/sec
primitive/parquet_2                       1.01    141.7±0.89ms   316.7 MB/sec    1.00    140.0±0.41ms   320.4 MB/sec
primitive/zstd                            1.01    156.8±0.96ms   286.2 MB/sec    1.00    154.7±0.45ms   290.1 MB/sec
primitive/zstd_parquet_2                  1.01    175.2±0.55ms   256.1 MB/sec    1.00    173.8±0.43ms   258.2 MB/sec
primitive_non_null/bloom_filter           1.03    113.4±0.39ms   388.0 MB/sec    1.00    110.1±0.53ms   399.5 MB/sec
primitive_non_null/default                1.01     69.8±0.30ms   630.6 MB/sec    1.00     69.3±0.38ms   634.6 MB/sec
primitive_non_null/parquet_2              1.01     92.3±0.38ms   477.0 MB/sec    1.00     91.2±0.30ms   482.4 MB/sec
primitive_non_null/zstd                   1.08    107.5±0.31ms   409.4 MB/sec    1.00    100.0±0.35ms   440.2 MB/sec
primitive_non_null/zstd_parquet_2         1.01    133.1±2.16ms   330.5 MB/sec    1.00    131.2±1.78ms   335.3 MB/sec
string/bloom_filter                       1.11   227.0±24.41ms     2.3 GB/sec    1.00    204.7±2.06ms     2.5 GB/sec
string/default                            1.00   136.1±23.15ms     3.8 GB/sec    1.06   144.2±23.26ms     3.6 GB/sec
string/parquet_2                          1.00    169.8±1.23ms     3.0 GB/sec    1.09    185.9±4.85ms     2.8 GB/sec
string/zstd                               1.01   458.0±18.83ms  1144.6 MB/sec    1.00   454.7±18.37ms  1152.8 MB/sec
string/zstd_parquet_2                     1.00    406.5±9.28ms  1289.7 MB/sec    1.03   417.0±35.04ms  1257.2 MB/sec
string_and_binary_view/bloom_filter       1.01     67.0±0.26ms   481.5 MB/sec    1.00     66.2±0.33ms   487.4 MB/sec
string_and_binary_view/default            1.01     50.4±0.23ms   639.8 MB/sec    1.00     50.0±0.17ms   645.2 MB/sec
string_and_binary_view/parquet_2          1.00     61.0±0.22ms   528.4 MB/sec    1.00     60.9±0.18ms   529.6 MB/sec
string_and_binary_view/zstd               1.00     86.8±0.25ms   371.4 MB/sec    1.00     86.9±0.26ms   371.1 MB/sec
string_and_binary_view/zstd_parquet_2     1.00     74.9±0.27ms   430.5 MB/sec    1.00     75.0±0.27ms   429.9 MB/sec
string_dictionary/bloom_filter            1.00    121.5±1.50ms     2.1 GB/sec    1.17    141.7±1.74ms  1864.3 MB/sec
string_dictionary/default                 1.00     77.5±1.29ms     3.3 GB/sec    1.23     95.0±5.99ms     2.7 GB/sec
string_dictionary/parquet_2               1.00     78.8±0.33ms     3.3 GB/sec    1.26     99.4±0.62ms     2.6 GB/sec
string_dictionary/zstd                    1.00    231.1±1.60ms  1142.9 MB/sec    1.10    253.8±1.56ms  1040.7 MB/sec
string_dictionary/zstd_parquet_2          1.00    211.3±0.38ms  1250.1 MB/sec    1.11    233.9±0.62ms  1129.3 MB/sec
string_non_null/bloom_filter              1.04   255.0±10.37ms     2.0 GB/sec    1.00   244.1±15.23ms     2.1 GB/sec
string_non_null/default                   1.00   134.2±12.02ms     3.8 GB/sec    1.00   134.3±10.99ms     3.8 GB/sec
string_non_null/parquet_2                 1.00    142.2±6.55ms     3.6 GB/sec    1.02    145.4±0.38ms     3.5 GB/sec
string_non_null/zstd                      1.00    537.5±3.27ms   974.9 MB/sec    1.02    546.9±2.51ms   958.1 MB/sec
string_non_null/zstd_parquet_2            1.00    503.1±0.62ms  1041.5 MB/sec    1.00    503.1±0.74ms  1041.5 MB/sec

Resource Usage

base (merge-base)

Metric Value
Wall time 1332.1s
Peak memory 4.9 GiB
Avg memory 4.6 GiB
CPU user 1256.7s
CPU sys 75.1s
Peak spill 0 B

branch

Metric Value
Wall time 1393.5s
Peak memory 4.9 GiB
Avg memory 4.6 GiB
CPU user 1245.1s
CPU sys 148.3s
Peak spill 0 B

File an issue against this benchmark runner

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

parquet Changes to the parquet crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@etseidl @adriangbot