Skip to content

v9.0.0

Latest

Choose a tag to compare

@weimingdiit weimingdiit released this 19 Nov 03:44
· 11 commits to master since this release

New Features

  • Spark 4.2 support: Added build and runtime compatibility for Apache Spark 4.2.
  • Memory-bounded native joins: Added hybrid hash-join fallback for large build sides and expanded residual-condition support across SortMergeJoin, BroadcastHashJoin, ShuffledHashJoin, and BroadcastNestedLoopJoin.
  • Expanded Iceberg native scans: Added runtime filters, _partition metadata, identity partition values, changelog pruning, decimal residual pruning, and improved schema-evolution support.
  • Native expressions and functions: Added native support for additional array and datetime functions, including array_contains, array_distinct, array_union, arrays_overlap, datediff, date_add, date_sub, add_months, next_day, trunc, and coalesce.
  • Flink operator fusion: Added source–Calc fusion into a single native plan and native support for UNIX_TIMESTAMP.
  • Native conversion diagnostics: Added operator conversion status, fallback reasons, AQE snapshots, effective configuration, and live SQL metrics to the Auron UI.

Improvements

  • Memory management: Introduced coordinated native spilling with configurable watermarks and bounded spill concurrency, while honoring Spark executor memory-overhead settings.
  • Execution performance: Optimized primitive sorting and Top-N, array/map explode, batch serialization, IPC compression, broadcast serialization, and aggregate processing.
  • Scan performance: Added bounded parallel reading for small files and improved Iceberg partition-size estimation, pruning, and scan metrics.
  • Runtime stability: Made native initialization retryable, improved spill error handling, and cleaned up native Arrow shuffle files when shuffles are unregistered.
  • Build and packaging: Switched the native engine to stable Rust, added Docker Compose support, fixed ARM Docker builds, and improved Maven option handling.
  • Dependency maintenance: Upgraded DataFusion from 49.0.0 to 55.2.0 and Arrow/Parquet from 55.2.0 to 59.2.0, with reproducible vendor-and-patch management for customized dependencies.

Bug Fixes

  • Fixed Spark semantic differences involving float ordering, float-to-decimal casts, negative square roots, datetime functions, nullable structs, and string casts.
  • Fixed aggregate correctness issues involving filtered aggregates, sliced batches, accumulator rows, and placeholder evaluation.
  • Fixed join correctness issues involving residual conditions, complex NULL keys, row counts, build-side fallback, and forced ShuffledHashJoin.
  • Fixed SortExec schema mapping, Top-K, WindowGroupLimit, union partition mapping, NativeCollectLimit, and zero-column shuffle behavior.
  • Fixed native JSON, Protocol Buffers, Arrow FFI, LZ4_HADOOP Parquet, ORC predicate pushdown, and Parquet statistics handling.
  • Fixed Iceberg changelog projection, nested schema evolution, metadata columns, partition metrics, scan locality, and unsupported-data-type fallback behavior.
  • Fixed Spark UDF synchronization, shuffle cleanup, spill-file handling, and native environment initialization failures.

Notable Changes

  • Apache top-level project release: Removed the incubating designation and updated project and release infrastructure following Auron’s graduation.
  • Safer large joins: Hybrid hash-join fallback is enabled by default to reduce memory pressure and avoid expensive full sort-merge fallback for large build sides.
  • Stable Rust toolchain: Native-engine builds no longer require a pinned nightly Rust toolchain.
  • Datetime conversion default: Native datetime-function conversion is disabled by default. Set spark.auron.enableNativeDateTimeFunctions=true to restore the previous behavior.
  • String-cast behavior: spark.auron.cast.trimString now defaults to false. Set it to true if applications rely on automatic whitespace trimming before casts.
  • Spark 4.2 requirements: Spark 4.2 builds require JDK 17 or later and Scala 2.13. Iceberg integration is not currently supported with Spark 4.2.

What's Changed

New Contributors

Full Changelog: v8.0.0...v9.0.0