[TESTING] Test thrift-remodel branch with datafusion

[alamb]

Which issue does this PR close?

• part of Test thrift-remodel branch with DataFusion arrow-rs#8513
• part of Use custom thrift decoder to improve speed of parsing parquet metadata arrow-rs#5854

Rationale for this change

This PR is not meant to be merged -- I am using it to test the proposed
thrift-remodel branch with DataFusion. The thrift-remodel branch has
a brand new, much faster, thrift metadata decoder.

Specifically, this PR is to run the DataFusion test suite with the thrift-remodel branch
of arrow-rs to see if there are any issues before we merge it.

What changes are included in this PR?

Are these changes tested?

Changes

• starts from [WIP] Upgrade to arrow/parquet 57.0.0 #17888
• upgrade datafusion to include thrift-remdel Use custom thrift parser for parquet metadata (phase 1 of Thrift remodel) arrow-rs#8530
• update tests

Things that would help upgrade

1. Add PrimitiveColumnIndex::len() (can fake with min_values.len())

Are there any user-facing changes?

[alamb]

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1016-gcp #17~24.04.1-Ubuntu SMP Wed Sep 3 01:55:36 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_thrift_remodel (082a22c) to 307f5c3 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

[alamb]

🤖: Benchmark completed

Details

Comparing HEAD and alamb_test_thrift_remodel
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_test_thrift_remodel ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0     │  2700.35 ms │                2710.11 ms │ no change │
│ QQuery 1     │  1314.50 ms │                1314.08 ms │ no change │
│ QQuery 2     │  2377.78 ms │                2478.90 ms │ no change │
│ QQuery 3     │  1200.18 ms │                1212.18 ms │ no change │
│ QQuery 4     │  2234.25 ms │                2231.08 ms │ no change │
│ QQuery 5     │ 27699.83 ms │               27833.86 ms │ no change │
│ QQuery 6     │  4152.07 ms │                4171.78 ms │ no change │
│ QQuery 7     │  3540.75 ms │                3465.12 ms │ no change │
└──────────────┴─────────────┴───────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 45219.71ms │
│ Total Time (alamb_test_thrift_remodel)   │ 45417.10ms │
│ Average Time (HEAD)                      │  5652.46ms │
│ Average Time (alamb_test_thrift_remodel) │  5677.14ms │
│ Queries Faster                           │          0 │
│ Queries Slower                           │          0 │
│ Queries with No Change                   │          8 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_test_thrift_remodel ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.17 ms │                   2.25 ms │     no change │
│ QQuery 1     │    49.20 ms │                  47.38 ms │     no change │
│ QQuery 2     │   138.30 ms │                 131.98 ms │     no change │
│ QQuery 3     │   159.35 ms │                 168.28 ms │  1.06x slower │
│ QQuery 4     │  1045.43 ms │                1091.99 ms │     no change │
│ QQuery 5     │  1492.46 ms │                1554.38 ms │     no change │
│ QQuery 6     │     2.13 ms │                   2.12 ms │     no change │
│ QQuery 7     │    53.21 ms │                  53.66 ms │     no change │
│ QQuery 8     │  1434.21 ms │                1473.44 ms │     no change │
│ QQuery 9     │  1838.70 ms │                1845.95 ms │     no change │
│ QQuery 10    │   382.38 ms │                 396.02 ms │     no change │
│ QQuery 11    │   436.88 ms │                 454.99 ms │     no change │
│ QQuery 12    │  1361.72 ms │                1388.66 ms │     no change │
│ QQuery 13    │  2083.61 ms │                2158.60 ms │     no change │
│ QQuery 14    │  1256.48 ms │                1284.43 ms │     no change │
│ QQuery 15    │  1195.69 ms │                1255.00 ms │     no change │
│ QQuery 16    │  2618.19 ms │                2700.84 ms │     no change │
│ QQuery 17    │  2604.96 ms │                2683.48 ms │     no change │
│ QQuery 18    │  5721.59 ms │                5017.27 ms │ +1.14x faster │
│ QQuery 19    │   128.61 ms │                 129.04 ms │     no change │
│ QQuery 20    │  2070.96 ms │                1962.33 ms │ +1.06x faster │
│ QQuery 21    │  2328.83 ms │                2307.45 ms │     no change │
│ QQuery 22    │  4036.89 ms │                3974.83 ms │     no change │
│ QQuery 23    │ 12711.99 ms │               12806.91 ms │     no change │
│ QQuery 24    │   213.13 ms │                 219.26 ms │     no change │
│ QQuery 25    │   514.80 ms │                 521.07 ms │     no change │
│ QQuery 26    │   231.62 ms │                 224.10 ms │     no change │
│ QQuery 27    │  2894.93 ms │                2896.02 ms │     no change │
│ QQuery 28    │ 24837.90 ms │               23176.07 ms │ +1.07x faster │
│ QQuery 29    │   995.76 ms │                 979.67 ms │     no change │
│ QQuery 30    │  1325.19 ms │                1342.25 ms │     no change │
│ QQuery 31    │  1338.85 ms │                1350.07 ms │     no change │
│ QQuery 32    │  4830.25 ms │                4421.62 ms │ +1.09x faster │
│ QQuery 33    │  6069.62 ms │                5566.52 ms │ +1.09x faster │
│ QQuery 34    │  6306.35 ms │                6039.25 ms │     no change │
│ QQuery 35    │  2036.06 ms │                2010.36 ms │     no change │
│ QQuery 36    │   123.25 ms │                 120.80 ms │     no change │
│ QQuery 37    │    53.81 ms │                  53.18 ms │     no change │
│ QQuery 38    │   119.12 ms │                 119.42 ms │     no change │
│ QQuery 39    │   198.46 ms │                 195.11 ms │     no change │
│ QQuery 40    │    45.41 ms │                  44.21 ms │     no change │
│ QQuery 41    │    40.96 ms │                  38.79 ms │ +1.06x faster │
│ QQuery 42    │    32.94 ms │                  32.91 ms │     no change │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 97362.32ms │
│ Total Time (alamb_test_thrift_remodel)   │ 94241.96ms │
│ Average Time (HEAD)                      │  2264.24ms │
│ Average Time (alamb_test_thrift_remodel) │  2191.67ms │
│ Queries Faster                           │          6 │
│ Queries Slower                           │          1 │
│ Queries with No Change                   │         36 │
│ Queries with Failure                     │          0 │
└──────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_test_thrift_remodel ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 166.71 ms │                 172.26 ms │     no change │
│ QQuery 2     │  26.13 ms │                  26.14 ms │     no change │
│ QQuery 3     │  42.16 ms │                  40.55 ms │     no change │
│ QQuery 4     │  28.04 ms │                  28.23 ms │     no change │
│ QQuery 5     │  79.19 ms │                  75.26 ms │     no change │
│ QQuery 6     │  20.21 ms │                  19.24 ms │     no change │
│ QQuery 7     │ 222.58 ms │                 220.24 ms │     no change │
│ QQuery 8     │  35.17 ms │                  31.38 ms │ +1.12x faster │
│ QQuery 9     │ 107.54 ms │                 104.46 ms │     no change │
│ QQuery 10    │  60.03 ms │                  60.34 ms │     no change │
│ QQuery 11    │  16.97 ms │                  17.73 ms │     no change │
│ QQuery 12    │  51.69 ms │                  51.34 ms │     no change │
│ QQuery 13    │  48.33 ms │                  46.48 ms │     no change │
│ QQuery 14    │  13.83 ms │                  13.78 ms │     no change │
│ QQuery 15    │  24.72 ms │                  23.89 ms │     no change │
│ QQuery 16    │  24.83 ms │                  24.42 ms │     no change │
│ QQuery 17    │ 149.17 ms │                 146.79 ms │     no change │
│ QQuery 18    │ 317.56 ms │                 323.60 ms │     no change │
│ QQuery 19    │  36.65 ms │                  36.25 ms │     no change │
│ QQuery 20    │  48.92 ms │                  49.23 ms │     no change │
│ QQuery 21    │ 333.92 ms │                 320.94 ms │     no change │
│ QQuery 22    │  20.79 ms │                  20.16 ms │     no change │
└──────────────┴───────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                        ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                        │ 1875.14ms │
│ Total Time (alamb_test_thrift_remodel)   │ 1852.73ms │
│ Average Time (HEAD)                      │   85.23ms │
│ Average Time (alamb_test_thrift_remodel) │   84.21ms │
│ Queries Faster                           │         1 │
│ Queries Slower                           │         0 │
│ Queries with No Change                   │        21 │
│ Queries with Failure                     │         0 │
└──────────────────────────────────────────┴───────────┘

[alamb]

My assessment is that the thrift-remodel branch is ready to go, so closing down this PR: #17916 (comment)