Skip to content

v8.0.0-incubating

Latest
Latest

Choose a tag to compare

View all tags
@slfan1989 slfan1989 released this 18 Jul 03:20
v8.0.0
This tag was signed with the committer’s verified signature.
slfan1989
GPG key ID: 978456E3964DB701
Verified
Learn about vigilant mode.
f2a603a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

New Features

  • Flink integration: Introduced the Flink Auron adaptor, configuration framework, expression converter framework, native Calc operators, and the structural Flink assembly deployment JAR.
  • Native Kafka source for Flink: Added a native Kafka table source and scan execution with checkpointing, watermark generation, idleness handling, partition discovery, JSON format support, and Flink metrics integration.
  • Lakehouse native scans: Added and expanded native scan support for Apache Iceberg, Apache Hudi, and Apache Paimon, including Copy-on-Write tables, Iceberg changelog scans, metadata columns, residual filters, scan metrics, and Paimon DataSource V2 scans.
  • Native expressions and functions: Added native support for aggregate FILTER (WHERE ...) clauses, Cast, comparison, logical, and mathematical operators, along with additional array, map, date, string, and unary expressions.
  • Window functions and joins: Added native implementations of lead, nth_value, percent_rank, and cume_dist, plus residual join-condition support for SortMergeJoin and ShuffledHashJoin.
  • Native serialization: Introduced native JSON and Protocol Buffers deserializers and expanded Arrow-to-Flink conversion support for basic, temporal, and composite types.

Improvements

  • Correctness testing: Expanded test scaffolding and suite wrappers across supported Spark versions for expressions, functions, aggregates, sorting, and Parquet behavior.
  • Spark compatibility: Added Spark 4.0 and 4.1 support to build and development workflows and fixed Scala 2.13 CI compatibility.
  • Build and packaging: Added macOS build support, Maven thread controls, improved command-line option parsing, standardized Cargo workspace settings, and refined release artifact naming.
  • Flink behavior: Improved configuration naming, metrics reporting, watermark handling, partition discovery, adaptor lifecycle management, and deployment packaging.
  • Native scan observability: Added conversion-reason reporting and exposed file and partition metrics for supported table-format scans.
  • Dependency maintenance: Updated Maven, Hadoop, Celeborn, Kafka, Tokio, Prost, Chrono, Log, and other build/runtime dependencies.

Bug Fixes

  • Fixed Kafka source cases that emitted no records or failed to generate watermarks.
  • Fixed native Paimon filtering and metadata-column correctness issues.
  • Fixed Iceberg partitioning, schema evolution, metadata-column, and residual-filter behavior.
  • Fixed Hudi scan option handling and prevented unsupported incremental queries from being converted to native scans.
  • Fixed native engine panics, core dumps, memory leaks, closed-channel crashes, and Java thread interruption errors.
  • Fixed driver-side spill NPEs, Uniffle reader and writer synchronization issues, and History Server compatibility when build information is missing.
  • Fixed timestamp timezone handling, ORC range bounds, stale build-version values, Scala 2.13 CI errors, and native metrics retrieval.

Notable Changes

  • Flink becomes a first-class integration: Auron 8.0.0 introduces a broader Flink runtime and planning foundation, including native Kafka ingestion and Calc conversion.
  • Broader lakehouse acceleration: Native scan coverage now spans key Apache Iceberg, Apache Hudi, and Apache Paimon scenarios with improved correctness and observability.
  • Expanded native execution: More Spark and Flink operators, scalar functions, aggregate filters, window functions, and join conditions can execute natively.
  • Stronger cross-version validation: Correctness suites and build workflows now cover more Spark, Scala, JDK, operating-system, and architecture combinations.
  • Improved runtime stability: This release resolves multiple crash, panic, NPE, thread-safety, watermark, and scan-correctness problems.

What's Changed

Flink integration and Kafka source

Native scans and table-format support

  • Support Iceberg column rename and drop-then-add in the native scan by @lyne7-sc in #2322
  • fix: Handle Paimon metadata columns in V2 native scan by @lyne7-sc in #2367
  • Support insert-only Iceberg changelog native scan by @weimingdiit in #2254
  • fix: Native Paimon V2 scan returns wrong results with native filter by @lyne7-sc in #2365
  • Support native scan for Paimon DSv2 (BatchScanExec) COW tables by @zhuxiangyi in #2313
  • Fix NativeIcebergTableScan to respect SinglePartition outputPartitioning by @lyne7-sc in #2342
  • Add support for auron.never.convert.reason in Hudi scan scenarios by @guixiaowen in #2293
  • Add parquet suite wrappers for Spark versions by @myandpr in #2281
  • Fallback Hudi incremental queries from native scan conversion by @weimingdiit in #2274
  • Implement native support for ORC InsertIntoHiveTable writes by @weimingdiit in #2191
  • Add support for auron.never.convert.reason in Iceberg scan scenarios by @guixiaowen in #2237
  • Add native Hudi scan coverage for partitioned COW tables by @weimingdiit in #2251
  • Support native Hudi scan for MOR read-optimized queries by @weimingdiit in #2249
  • Reuse Hudi table properties when detecting native scan support by @weimingdiit in #2247
  • Support Iceberg _spec_id metadata in native scan by @weimingdiit in #2218
  • Expose numFiles and numPartitions metrics for native Iceberg scan by @weimingdiit in #2220
  • Handle Hudi scan options case-insensitively by @weimingdiit in #2235
  • Remove hard-coded Iceberg scan class name detection using type-based check by @guixiaowen in #2226
  • Add native support for the _file metadata column by @weimingdiit in #2184
  • Support native Iceberg scans with residual filters via scan pruning and post-scan native filter by @weimingdiit in #2164
  • Add Native Scan Support for Apache Hudi Copy-On-Write Tables. by @slfan1989 in #2031
  • chore: add Iceberg version to AuronBuildInfo by @XorSum in #2095
  • Fix ORC range start index out of range for slice of length by @cxzl25 in #1986
  • Add Native Scan Support for Apache Iceberg Copy-On-Write Tables. by @slfan1989 in #2016

Native expressions, operators, and functions

Window functions and joins

Correctness testing and Spark-version coverage

  • Add sort suite wrappers for Spark versions by @myandpr in #2275
  • Add aggregate suite wrappers for Spark versions by @myandpr in #2256
  • [Correctness Testing] Add function-related tests by @lyne7-sc in #2245
  • correctness testing: setup correctness testing scaffolding for all spark versions by @ShreyeshArangath in #2209
  • Fix rust lint test fail by @Tartarus0zm in #2100
  • skipping the plan stability validation for tpcds q54 test in spark3.5 by @XorSum in #2071

Build, CI, packaging, and dependencies

Reliability, bug fixes, and cleanup

  • Honor config alt keys when reading from SQLConf by @robreeves in #2361
  • chore: enable copilot code review on push by @cxzl25 in #2355
  • Optimize native metrics retrieval by passing keys directly by @lsm1 in #1982
  • Upgrade hashbrown to 0.17.1 and adapt API by @cxzl25 in #2302
  • Support compile with Celeborn 0.6 and spark 4.0 by @ftong2020 in #2243
  • Date-part extraction functions missing timezone handling for Timestamp inputs by @ShreyeshArangath in #2156
  • Wire existing native scalar functions in NativeConverters by @ShreyeshArangath in #2162
  • Fix typos and unnecessary memory allocation in native engine by @Deegue in #2077
  • fix: change const to static in batch_size() to fix OnceCell caching by @yew1eb in #2159
  • Set thread name for ArrowFFIExporter by @cxzl25 in #2148
  • Native thread name appends the original thread name by @cxzl25 in #2144
  • fix pb list name missing by @Tartarus0zm in #2143
  • Fix some typos in native by @XorSum in #2139
  • Add isTaskRunning method to check task status in SparkAuronAdaptor by @cxzl25 in #2120
  • Introduce native json deserializer by @Tartarus0zm in #2112
  • Fix driver spill NPE error by @Deegue in #2108
  • Migrate the Native dependency configuration from SparkAuronConfiguration to AuronConfiguration by @Tartarus0zm in #2105
  • Initializing JavaClasses in JNI and decoupling Spark by @Tartarus0zm in #2103
  • Fix panic/coredumps/memory leaks and Java thread InterruptedException errors. by @BrytonLee in #1980
  • Introduce native protobuf deserializer by @Tartarus0zm in #2069
  • NPE fix in AuronUniffleShuffleReader by @slfan1989 in #2053
  • PR title check by @XorSum in #2066
  • Migrate from deprecated JavaConverters to CollectionConverters. by @slfan1989 in #2027
  • [BUG] Native engine panic on closed channel causes JVM crash. by @slfan1989 in #2023
  • Fix extra brace in SparkOnHeapSpillManager log message. by @slfan1989 in #2029
  • Fix thread-safety issues in UnifflePartitionWriter synchronization. by @slfan1989 in #2033

Contributors

Thanks to all contributors who made this release possible:

bkhan, Bryton Lee, Calvin Kirs, cxzl25, dependabot[bot], ftong2020, guixiaowen, linfeng, LSM, Ming Wei, Peter Chen J., Rob Reeves, Shreyesh, slfan1989, Stefan Wang, The Apache Software Foundation, Weiqing Yang, Xiangyi Zhu, xTong, xuzifu666, yaommen, yew1eb, Yizhong Zhang, Yuepeng Pan, zhangmang

Reviewers

Thanks to everyone who reviewed pull requests for this release:

@yew1eb, @cxzl25, @slfan1989, @ShreyeshArangath, @richox, @merrily01, @Tartarus0zm, @SteNicholas, @lihao712, @weiqingy, @zuston

Full Changelog

Contributors

BrytonLee, richox, and 28 other contributors
Assets 2
Loading