Repository navigation
New Features
- Spark 4.2 support: Added build and runtime compatibility for Apache Spark 4.2.
- Memory-bounded native joins: Added hybrid hash-join fallback for large build sides and expanded residual-condition support across SortMergeJoin, BroadcastHashJoin, ShuffledHashJoin, and BroadcastNestedLoopJoin.
- Expanded Iceberg native scans: Added runtime filters,
_partitionmetadata, identity partition values, changelog pruning, decimal residual pruning, and improved schema-evolution support. - Native expressions and functions: Added native support for additional array and datetime functions, including
array_contains,array_distinct,array_union,arrays_overlap,datediff,date_add,date_sub,add_months,next_day,trunc, andcoalesce. - Flink operator fusion: Added source–Calc fusion into a single native plan and native support for
UNIX_TIMESTAMP. - Native conversion diagnostics: Added operator conversion status, fallback reasons, AQE snapshots, effective configuration, and live SQL metrics to the Auron UI.
Improvements
- Memory management: Introduced coordinated native spilling with configurable watermarks and bounded spill concurrency, while honoring Spark executor memory-overhead settings.
- Execution performance: Optimized primitive sorting and Top-N, array/map explode, batch serialization, IPC compression, broadcast serialization, and aggregate processing.
- Scan performance: Added bounded parallel reading for small files and improved Iceberg partition-size estimation, pruning, and scan metrics.
- Runtime stability: Made native initialization retryable, improved spill error handling, and cleaned up native Arrow shuffle files when shuffles are unregistered.
- Build and packaging: Switched the native engine to stable Rust, added Docker Compose support, fixed ARM Docker builds, and improved Maven option handling.
- Dependency maintenance: Upgraded DataFusion from 49.0.0 to 55.2.0 and Arrow/Parquet from 55.2.0 to 59.2.0, with reproducible vendor-and-patch management for customized dependencies.
Bug Fixes
- Fixed Spark semantic differences involving float ordering, float-to-decimal casts, negative square roots, datetime functions, nullable structs, and string casts.
- Fixed aggregate correctness issues involving filtered aggregates, sliced batches, accumulator rows, and placeholder evaluation.
- Fixed join correctness issues involving residual conditions, complex NULL keys, row counts, build-side fallback, and forced ShuffledHashJoin.
- Fixed SortExec schema mapping, Top-K, WindowGroupLimit, union partition mapping, NativeCollectLimit, and zero-column shuffle behavior.
- Fixed native JSON, Protocol Buffers, Arrow FFI, LZ4_HADOOP Parquet, ORC predicate pushdown, and Parquet statistics handling.
- Fixed Iceberg changelog projection, nested schema evolution, metadata columns, partition metrics, scan locality, and unsupported-data-type fallback behavior.
- Fixed Spark UDF synchronization, shuffle cleanup, spill-file handling, and native environment initialization failures.
Notable Changes
- Apache top-level project release: Removed the
incubatingdesignation and updated project and release infrastructure following Auron’s graduation. - Safer large joins: Hybrid hash-join fallback is enabled by default to reduce memory pressure and avoid expensive full sort-merge fallback for large build sides.
- Stable Rust toolchain: Native-engine builds no longer require a pinned nightly Rust toolchain.
- Datetime conversion default: Native datetime-function conversion is disabled by default. Set
spark.auron.enableNativeDateTimeFunctions=trueto restore the previous behavior. - String-cast behavior:
spark.auron.cast.trimStringnow defaults tofalse. Set it totrueif applications rely on automatic whitespace trimming before casts. - Spark 4.2 requirements: Spark 4.2 builds require JDK 17 or later and Scala 2.13. Iceberg integration is not currently supported with Spark 4.2.
What's Changed
- Bump bytesize from 2.3.1 to 2.4.2 by @dependabot[bot] in #2382
- [AURON #2373] support Docker Compose in build script by @zhaohehuhu in #2374
- [AURON #2375] Fix Iceberg changelog scan field-id projection by @lyne7-sc in #2376
- Bump bytes from 1.11.1 to 1.12.1 by @dependabot[bot] in #2383
- Bump prost-reflect from 0.16.4 to 0.16.5 by @dependabot[bot] in #2384
- Bump regex from 1.12.3 to 1.13.0 by @dependabot[bot] in #2385
- [AURON #2265] Add AI-assisted development guidance by @ShreyeshArangath in #2387
- [AURON #2378] Support runtime filters in native Iceberg scan by @lyne7-sc in #2379
- Bump regex from 1.13.0 to 1.13.1 by @dependabot[bot] in #2389
- Bump lz4_flex from 0.13.1 to 0.14.0 by @dependabot[bot] in #2388
- [AURON #2396] [RELEASE] Bump the development version to 9.0.0-SNAPSHOT by @slfan1989 in #2397
- [AURON #1865] Merge native operators: fuse Flink source and Calc into one native plan by @weiqingy in #2327
- [AURON #2352] Lock spills before new spiller by @wForget in #2350
- Bump serde from 1.0.228 to 1.0.229 by @dependabot[bot] in #2403
- [AURON #2398] Fix Maven -D argument handling in auron-build.sh by @slfan1989 in #2399
- Bump futures-util from 0.3.32 to 0.3.33 by @dependabot[bot] in #2401
- Bump async-trait from 0.1.89 to 0.1.91 by @dependabot[bot] in #2400
- Bump tokio from 1.52.3 to 1.53.0 by @dependabot[bot] in #2402
- [AURON #2372] Fix Docker build on ARM hosts by @zhaohehuhu in #2371
- Bump actions/labeler from 6 to 7 by @dependabot[bot] in #2410
- Bump futures from 0.3.32 to 0.3.33 by @dependabot[bot] in #2411
- Bump tokio from 1.53.0 to 1.53.1 by @dependabot[bot] in #2412
- [AURON #2413] fix(ci): adapt labeler.yml to actions/labeler v7 by @weiqingy in #2414
- [AURON #2320] Fix PB deserialization bug & improve PB parsing performance by @Tartarus0zm in #2339
- [AURON #2417] log rss write size metrics by @cxzl25 in #2418
- [AURON #1891] Implement randn() function by @robreeves in #1938
- Bump base64 from 0.22.1 to 0.23.0 by @dependabot[bot] in #2420
- [AURON #2427] Report fallback reason for unsupported Iceberg scan data types by @weimingdiit in #2428
- [AURON #2425] Support Iceberg STARTS_WITH pruning in native scan by @weimingdiit in #2426
- [AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build example. by @slfan1989 in #2408
- [AURON #2329] Test and document multi-consumer source-Calc fan-out safety by @weiqingy in #2415
- [AURON #2432] Vendor third-party dependencies via setup-vendors.sh with patch series by @diliulou in #2433
- Bump actions/stale from 10 to 11 by @dependabot[bot] in #2436
- [AURON #2440] Consolidate Maven properties and format code by @cxzl25 in #2441
- [AURON #2439] Add git to yum install by @cxzl25 in #2442
- [AURON #2419] RowNullChecker may mis-detect null join keys for complex Arrow row types by @weimingdiit in #2424
- [AURON #2429] Add mixed-format fallback coverage for Iceberg changelog native scan by @weimingdiit in #2430
- [AURON #2446] Flush buffered data based on row count by @zuston in #2447
- [AURON #2422] Make RSS spill memory triggers configurable by @cxzl25 in #2423
- [AURON #1863] Support native Flink UNIX_TIMESTAMP: native scalar function by @weiqingy in #2409
- [AURON #2434] Support full-data-file Iceberg changelog deletes by @weimingdiit in #2435
- [AURON #2449] Add fallback coverage for unsupported Paimon scan data type by @weimingdiit in #2450
- [AURON #2453] Fix filtered aggregation for groups with no matching rows by @lyne7-sc in #2454
- [AURON #1863] Support native Flink UNIX_TIMESTAMP: converter integration by @weiqingy in #2448
- [AURON #1863] Support the zero-argument Flink UNIX_TIMESTAMP natively by @weiqingy in #2465
- [AURON #2463] Fix native Top-K with only sort keys by @lyne7-sc in #2464
- [AURON #2437] Support inner residual join conditions in native broadcast hash join by @weimingdiit in #2438
- [AURON #2451] Add Paimon COW multi-commit correctness coverage by @weimingdiit in #2452
- [AURON #2457] Fix per-partition job submission in NativeCollectLimit by @lyne7-sc in #2458
- [AURON #2431] Prune Iceberg changelog tasks by metadata predicates by @goutamadwant in #2456
- [AURON #1902] Reuse extension function implementations by @goutamadwant in #2467
- [AURON #2455] Run UNIX_TIMESTAMP natively for fixed-offset session time zones by @weiqingy in #2466
- [AURON #2468] Support binary sort keys in native range shuffle by @weimingdiit in #2469
- [AURON #2472] Fix nullable struct field null propagation by @lyne7-sc in #2473
- [AURON #2392] Support Spark 4.2 by @XorSum in #2404
- [AURON #2405] Add primitive fast path for single-column integer sort by @yew1eb in #2406
- [AURON #2369] fix incorrect ORC predicate pushdown with OR by @Flyangz in #2370
- [AURON #2482] handle optional sort indices for zero-column batches by @lyne7-sc in #2483
- [AURON #2474] Fix make_date null and ANSI semantics by @Sigma-Ma in #2479
- [AURON #2444] Add build branch and build reversion info by @zhaohehuhu in #2445
- [AURON #2308] Fix native shuffle exchange NPE in shuffle write tasks by @Sigma-Ma in #2487
- [AURON #2480] Optimize array and map explode using offset ranges by @lyne7-sc in #2481
- [AURON #2485] Support last_day function natively by @Sigma-Ma in #2486
- [AURON #2493] Docs: update GitHub repo About description by @diliulou in #2493
- [AURON #2495] Docs: update README description by @diliulou in #2495
- [AURON #2488] Support dayofyear function natively by @Sigma-Ma in #2489
- [AURON #2503] Rename blaze to auron by @Deegue in #2504
- [AURON #2475] Fix float-to-decimal precision loss by @Sigma-Ma in #2496
- [AURON #2501] Support datediff function natively by @Sigma-Ma in #2502
- [AURON #2499] perf: avoid identity take in GenerateExec by @lyne7-sc in #2500
- [AURON #2497] Fix SortExec projected schema and sort key mapping by @lyne7-sc in #2498
- [AURON #2351] Reduce native broadcast task serialization size by @Sigma-Ma in #2506
- [AURON #2513] Optimize spill error handling and add an IOExceptiion for spill file missing by @Deegue in #2514
- [AURON #2518] Cache maximum accumulator indices to avoid repeated scans by @lyne7-sc in #2519
- [AURON #2509] Support date_add and date_sub functions natively by @Sigma-Ma in #2512
- [AURON #2307] Add @transient to large fields and use NativePartition to reduce serialization overhead by @Deegue in #2515
- [AURON #2511] Prune full-data-file Iceberg changelog delete tasks by @Sigma-Ma in #2523
- [AURON #2530] Fix incorrect results in partial merge aggregation with sliced batches by @Deegue in #2531
- [AURON #1743] Support ORC reader batch size by @Sigma-Ma in #2525
- [AURON #2534] Fix native broadcast nested loop join diff by @Deegue in #2535
- [AURON #2536] Add shared CPU join condition evaluation by @Deegue in #2537
- [AURON #2529] Support add_months function natively by @Sigma-Ma in #2533
- [AURON #2527] Populate identity partition values in native Iceberg scans by @lyne7-sc in #2528
- [AURON #2524] Support unix_date and date_from_unix_date natively by @Sigma-Ma in #2526
- [AURON #2521] Fix NULL predicate handling in aggregate filters by @lyne7-sc in #2522
- [AURON #2538] Support and extend residual conditions in native sort-merge joins by @Deegue in #2543
- [AURON #2547] Use TLP dist repositories in release script by @richox in #2548
- [AURON #2549] update GitHub repo and README by @diliulou in #2549
- [AURON #2550] docs: remove Incubating references after graduation by @diliulou in #2550
- [AURON #2539] Support residual conditions in native broadcast hash joins by @Deegue in #2551
- [AURON #2540] Support residual conditions in native shuffled hash joins by @Deegue in #2552
- [AURON #2541] Support residual conditions in native broadcast nested loop joins by @Deegue in #2553
- [AURON #2554] Support shared native memory manager and spill coordinator by @Deegue in #2556
- [AURON #2544] Support next_day function natively by @Sigma-Ma in #2546
- [AURON #2510] Add non-UTC timestamp coverage for native datediff by @Sigma-Ma in #2520
- [AURON #2555] Support coordinated spilling for native host memory by @Deegue in #2557
- [AURON #2559] Add native conversion diagnostics to Auron SQL UI. by @slfan1989 in #2560
- [AURON #2562] Document native conversion diagnostics in the Spark UI. by @slfan1989 in #2563
- [AURON #2567] Honor Spark memory overhead factors when calculating the native budget by @robreeves in #2568
- [AURON #2118] make native environment initialization retriable after failure by @XorSum in #2151
- [AURON #2564] Fix union input partition mapping during plan rewrites by @Deegue in #2565
- [AURON #2577] Prevent stale Celeborn PID files from blocking CI startup by @lyne7-sc in #2578
- [AURON #2569] Fix WindowGroupLimit fusion with multiple window expressions by @lyne7-sc in #2570
- [AURON #2558] Support Iceberg _partition metadata in native scans by @Sigma-Ma in #2566
- [AURON #2561] Support trunc function natively by @Sigma-Ma in #2571
- [AURON #1919] Clean up Clippy code style/redundancy lints (Part 2) by @Sigma-Ma in #2585
- [AURON #2579] Fix zero-column round-robin shuffle presort by @squanch-c in #2580
- [AURON #2572] Support array_contains natively by @Sigma-Ma in #2573
- [AURON #2593] Fix missing locality in NativeRDD by @Deegue in #2594
- [AURON #2584] Implement native support for array_distinct by @Sigma-Ma in #2592
- [AURON #2517] Align native float ordering with Spark by @Sigma-Ma in #2545
- [AURON #2588] Preserve literal parameters during expression fallback by @lyne7-sc in #2589
- [AURON #2608] Fix freezing of selected UDAF accumulator rows by @Deegue in #2616
- [AURON #2609] Simplify native Parquet execution-plan output by @Deegue in #2617
- [AURON #2610] Reuse LZ4 and ZSTD codec contexts across IPC blocks by @Deegue in #2618
- [AURON #2581] Extract supported conjuncts for Iceberg changelog task … by @Sigma-Ma in #2583
- [AURON #2611] Preserve unknown Parquet null counts by @Deegue in #2622
- [AURON #2612] Optimize batch serialization with SIMD byte transposition by @Deegue in #2623
- [AURON #2603] Fix SMJ fallback timing metrics under output backpressure by @lyne7-sc in #2604
- [AURON #2596] Support decimal residual predicates in native Iceberg Parquet scan pruning by @Sigma-Ma in #2619
- [AURON #2646] Hide the Auron tab in SHS when Auron is disabled by @robreeves in #2647
- [AURON #2632] Fix shared row indices for aggregates with different FILTER predicates by @lyne7-sc in #2633
- [AURON #2621] Restore Parquet row-group statistics pruning metrics by @Sigma-Ma in #2626
- [AURON #2620] Support Spark's array_union natively by @Sigma-Ma in #2627
- [AURON #2615] Separate explicit spills from coordinator admission by @Deegue in #2644
- [AURON #2613] Support building the native engine with stable Rust by @Deegue in #2641
- [AURON #2651] Support parallel reading for tables with small files by @richox in #2652
- [AURON #2614] Add hybrid hash-join fallback for large build sides by @Deegue in #2643
- [AURON #2653] Preserve native join row counts and simplify hybrid execution by @Deegue in #2655
- [AURON #2654] Upgrade DataFusion to 55.2.0 and Arrow to 59.2.0 by @Deegue in #2656
- [AURON #2664] Display available live SQL metrics in the Auron UI by @lyne7-sc in #2665
- [AURON #2666] Synchronize batch evaluation in Spark UDF wrapper by @lihao712 in #2667
- [AURON #2668] Fix xxhash64 with scalar arguments in datafusion-spark by @lihao712 in #2669
- [AURON #2670] Evaluate coalesce natively in datafusion by @lihao712 in #2671
- [AURON #2673] Default auron.cast.trimString to false by @lihao712 in #2674
- [AURON #2687] Fix forceShuffledHashJoin functionality by @richox in #2688
- [AURON #2681] Remove arrow shuffle files when a shuffle is unregistered by @lihao712 in #2682
- [AURON #2685] Handle null scalar input in get_parsed_json_object by @lihao712 in #2686
- [AURON #2678] Normalize sliced Arrow arrays before exporting to Java by @Deegue in #2694
- [AURON #2679] Share batch coalescing between execution streams and IPC readers by @Deegue in #2693
- [AURON #2662] Classify adaptive shuffle readers as wrappers in Auron UI by @robreeves in #2663
- [AURON #2676] Add a switch for native datetime function conversion by @lihao712 in #2677
- [AURON #2683] Return execution errors for aggregate placeholder evaluation by @Deegue in #2701
- [AURON #2680] Reuse compatible datafusion-spark extension functions by @Deegue in #2700
- [AURON #2672] Fix native JSON scalar batch handling by @squanch-c in #2692
- [AURON #2629] Keep native Iceberg scans for unaffected projections after nested schema evolution by @Sigma-Ma in #2658
- [AURON #2576] Support arrays_overlap natively by @Sigma-Ma in #2582
- [AURON #2684] Return NaN for negative sqrt inputs by @lyne7-sc in #2697
- [AURON #2635] Fix LZ4_HADOOP parquet framing interop with parquet-java by @COLDMANYU in #2638
- [AURON #2659] Count distinct data files in native Iceberg snapshot scan metrics by @Sigma-Ma in #2696
- [AURON #2705] Fix Clippy failure in arrays_overlap test by @Sigma-Ma in #2706
- [AURON #2707] Align struct comparison with Spark for max/min aggregations by @richox in #2708
- [AURON #2689] Use scan-range lengths when estimating native Iceberg partition sizes by @Sigma-Ma in #2699
- [AURON #2716] Bump version to 9.0.0 by @weimingdiit in #2717
New Contributors
- @zhaohehuhu made their first contribution in #2374
- @goutamadwant made their first contribution in #2456
- @squanch-c made their first contribution in #2580
- @COLDMANYU made their first contribution in #2638
Full Changelog: v8.0.0...v9.0.0