October 16, 2026: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days saw very high activity in apache/incubator-gluten: ~75 pull requests, ~18 issues, and 5 discussions/announcements. Two big themes dominate the week: Spark 4.2 readiness (both Delta-on-Spark 4.2 and the large body of re-enabled Spark 4.2 test suites) and Delta Lake native write-path correctness. Alongside those, the Velox backend received several performance and stability fixes (shuffle reader SIMD padding, broadcast hash-table thread pools, driver-side hash-table key aliasing), while the Iceberg effort continued expanding its surface (sorted writes, MERGE INTO, equality deletes) and its CI coverage. There was also a substantial amount of cleanup work — removing dead code, CentOS 7 remnants, and legacy metric plumbing — driven largely by @LuciferYang and @malinjawi. As in prior weeks, a large share of contributions are AI-assisted, with explicit Generated-by: IBM Bob / Claude / Copilot declarations in descriptions.

Key Ongoing Projects

Delta on Spark 4.2 (one Delta release for Spark 4.x) — @malinjawi's #13271 is the centerpiece, using a single Delta 4.4.1 for Spark 4.0/4.1/4.2, renaming src-delta40 to src-delta4, and gating the native writer behind GlutenDeltaNativeWriteGate. It is validated by the temporary CI-only follow-up #13322 and documented in the rewritten Velox Delta page #13338.

Spark 4.2 test-suite re-enablement — @akshaytayal drove a large, deliberately split series of independent PRs that re-enable the tests disabled by #13163. Representative examples include the TPC-DS plan-stability refresh #13312 and a long run of golden-file refreshes (#13291–#13299, #13300–#13306), plus Gluten-native versions of Spark 4.2 UPDATE NOT NULL and MERGE/DELETE summary tests (#13313–#13318, #13244, #13247). @akshaytayal also added the new "Enhanced" DSv2 partition-filter suites in #13164.

Delta native write & metrics offload — @malinjawi's #13324 extends native writes to MERGE, and #13165 counts Delta's IncrementMetric natively so MERGE's write projection can run in Velox. @felipepessoto contributed several correctness pieces here: TIMESTAMP_NTZ literal preservation #13041, generated DV metadata fallback #13042, transaction-observer preservation #13134, and the CDF assertion workaround #13129.

Iceberg coverage and capability expansion — @infvg continues to push Iceberg forward with global sorted writes #13199, MERGE INTO (copy-on-write) #12964, top-level equality-delete reads #12418, a large upstream Iceberg test harness overhaul #13224, and the UT CI whitelist #13323. The RFC discussion #13211 lays out a driver-heap reduction proposal with prototype numbers.

Velox backend performance & stability — @yikf introduced a dedicated CPU thread pool for parallel broadcast hash-table builds in #13115 and fixed concurrent broadcast resource-ID registration in #13320. @zhouyuan landed the SIMD padding fix for shuffle reader buffers in #13337 (fixing a reproducible SIGSEGV on TPC-DS q78/q97 at SF10000). @malinjawi converted ValueStreamDataSource::next() from recursion to iteration in #13346 (reviving closed #12563).

Native Kafka connector — @ReemaAlzaid continued the Kafka series with the native connector #13183 (built on #13182) and the source-build change #13270.

Build/CI/infra modernization — @malinjawi removed the remaining CentOS 7 support and references in #13336 and fixed the stale license-header check path in #13339. @baibaichen fixed macOS/OpenEuler weekly setup drift in #13168, and @zhouyuan added spark-4.2 to the nightly build and fixed the iceberg-1.11 build in #13327.

Priority Items

• Crash/data-loss risk requiring review: [VL] Keep Velox SIMD padding on shuffle reader buffers #13337 by @zhouyuan fixes an intermittent SIGSEGV in Velox SIMD gatherBits on shuffle null buffers. It is still a draft and "not yet run end to end"; reviewers with shuffle/TPC-DS access should prioritize confirming it before it drags.
• High-traffic correctness PR: [VL] Respect caseSensitiveAnalysis in IcebergScanTransformer metadata-column handling #12726 by @sabbasani (caseSensitiveAnalysis in Iceberg metadata columns) has ~62 comments — by far the busiest thread of the week — and should be steered to a resolution or split.
• Large stacked Delta work: [GLUTEN-12569][VL] Enable Delta on Spark 4.2: one Delta 4.4.1 release for Spark 4.x, following Delta's write path #13271 by @malinjawi (11 comments, 71 files) and [VL] Write Delta MERGE data files natively #13324 by @malinjawi (native MERGE write, draft) are the critical path for Delta-on-Spark 4.2; review order matters.
• Wrong-results clusters with ready fixes: [VL] regr_r2 returns wrong results when its phases are split between Velox and vanilla Spark #13340 → fixed by [GLUTEN-13340][VL] Fix regr_r2's intermediate field order #13343 (@LuciferYang, regr_r2 field order), and [VL] Driver-side broadcast hash table build falls back when a build key is a generated alias #13330 → fixed by [GLUTEN-13330][VL] Remap generated build key aliases in the driver-side hash table build #13332 (@LuciferYang, driver-side hash-table build aliasing). [VL] Shared library loading fails on TencentOS Server with loadLibFromJar enabled #13329 → fixed by [GLUTEN-13329][VL] Match the TencentOS Server os-release name in the shared library loaders #13331 (@LuciferYang, TencentOS loader matching). These small, self-contained fixes are low-risk and should land quickly.
• Resource leak / init failure: [CELEBORN] FailedShuffleCleaner instance is never returned to CelebornShuffleManager, leaking its ScheduledExecutorService on stop() #13279 by @maoyuwei1 describes a FailedShuffleCleaner leak in Celeborn; [GLUTEN-13276][VL]Fix CelebornShuffleManager init fails with NoSuchMe… #13283 by @maoyuwei1 fixes a NoSuchMethodException in CelebornShuffleManager init. Both deserve maintainer attention for long-lived drivers.
• Confirmed production bugs needing fixes: VeloxRssSortShuffleReaderDeserializer: infinite loop on EOS mid-page and corrupted deserialization on multi-window Presto pages #12985 by @kecookier documents two rss-sort shuffle reader bugs (infinite loop on EOS mid-page, corrupted multi-window Presto pages). The regression tests [DNM][VL] Add regression tests for rss-sort shuffle reader EOS handling #12983 were merged as probes; a fix PR is still expected.
• Cleanup batch (safe, mergeable): [GLUTEN-13341][VL] Remove the unused veloxToArrow metric #13344, [GLUTEN-13342][VL] Remove the unused ShuffleSchema.h #13345, [VL] Remove the unused veloxToArrow metric #13341, [VL] Remove the unused ShuffleSchema.h #13342 by @LuciferYang remove dead metrics and headers; [CORE] Remove arrow relocation #13096 by @marin-ma removes the Arrow relocation that broke PySpark Arrow conversion; [GLUTEN-13290][VL] Remove remaining CentOS 7 support and references #13336 by @malinjawi removes remaining CentOS 7 plumbing.

Notable Discussions

• Release Cadence Improvement Proposal #13281: @zhouyuan proposes a release-cadence overhaul — promoting the nightly server in docs/README, then a release-branch model moving from quarterly to monthly/6-week cadence, plus patch releases. This affects how every other item on this list eventually ships.
• [Show & Tell] Native Regex Unicode Escape Support in Gluten (Velox & Bolt) #13321: @KushShriv's Show & Tell walks through native RE2 translation of Java Unicode escapes for Velox and Bolt, with end-to-end validation and 15/15 query-shape parity against vanilla Spark. Pairs with PR [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs #13278.
• RFC: Reduce Driver Heap Usage with Early Iceberg Task Encoding #13211: @infvg's RFC on early Iceberg task encoding to cut driver heap, with prototype numbers showing ~34% lower retained heap and ~10% warm-query speedup on a 100k-file workload.
• [VL] Spark PR59313 comparison #13333: @FelixYBW compares Spark PR #59313 (native data sources) with Gluten's Arrow work, and raises an unconfirmed wrong-results concern with reused VectorSchemaRoot batches feeding Python UDF pass-through columns.
• [VL] Fully arrow support in Spark and Gluten #13140: @FelixYBW tracks full Arrow support in Spark and Gluten, laying out the current batch-type/transition map and the SPARK-57468 proposal to enable plugin interop without per-operator Gluten reimplementations.

Emerging Trends

• Spark 4.2 migration as the dominant coordination axis. Nearly every substantive area — Delta, Iceberg, test golden files, nightly builds — now has a Spark 4.2 leg. Expect this to continue through the next several weeks.
• Graceful fallback over hard failure remains a design principle. Examples this week include DV metadata fallbacks ([GLUTEN-12743][VL] Fall back Delta scans requiring generated DV metadata #13042), token-less TencentOS loader matching ([GLUTEN-13329][VL] Match the TencentOS Server os-release name in the shared library loaders #13331), driver-side hash-table build fallback ([GLUTEN-13330][VL] Remap generated build key aliases in the driver-side hash table build #13332), and the RE2 C++ validator guard ([VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs #13278).
• AI-assisted contributions are now the default. A large majority of the PRs sampled declare Generated-by: (IBM Bob, Claude, Copilot). This raises the value of clear test evidence in descriptions, which most contributors are supplying.
• Aggressive dead-code and legacy-platform cleanup. Removing CentOS 7, unused metrics, unused headers, arrow relocation, and stale test gates suggests the project is deliberately shrinking its maintenance surface.
• Isolated thread pools and resource isolation for CPU vs. I/O. [VL] Use a dedicated CPU thread pool for parallel broadcast hash-table build #13115's dedicated broadcast-hash-table pool and [VL] Keep Velox SIMD padding on shuffle reader buffers #13337's padding/resize helpers both point to hardening the boundary between native execution and the surrounding runtime.
• Native write-path correctness as the frontier for data lakes. Both Delta ([VL] Write Delta MERGE data files natively #13324, [GLUTEN-9003][VL] Count Delta's IncrementMetric natively #13165) and Iceberg ([VL][Iceberg] Add global iceberg sorted write #13199, [VL][Iceberg] Support MERGE INTO for parquet #12964) are now pushing writes into Velox, not just reads.

Good First Issues

• [CORE] Drop the now-constant ">= 3.4" test gates #12992: Drop the now-constant >= 3.4 test gates. Purely mechanical Scala test-scaffolding cleanup across ~14 files and 57 call sites, with zero behavior change — an ideal first PR for someone comfortable reading Scala test helpers.
• [VL] Support TIMESTAMP_NTZ Type #11622: Support TIMESTAMP_NTZ. The umbrella is well-documented and several sub-tasks remain; enabling extract.sql, timestamp-ltz.sql, and timestamp-ntz.sql in VeloxSQLQueryTestSettings.scala is a concrete, test-driven entry point that needs only basic Parquet/Spark type knowledge.
• [VL] Add ANSI mode support #10134: ANSI mode support. A long-running checklist with clearly-scoped, self-contained items (e.g., Abs, ElementAt, GetTimestamp, BRound, RoundCeil/Floor) that pair one Velox function with Spark validation. Good for contributors with C++/Velox or Scala skills; verify against Velox upstream before claiming.
• [CH] support split_part function #6807: Support split_part on the ClickHouse backend. A single built-in function implementation; well-suited to someone learning the CH backend's UDF plumbing.
• [CH] support MonotonicallyIncreasingID expression #6811: Support MonotonicallyIncreasingID on ClickHouse. Requires maintaining partition_id in the query context and a modest C++ built-in — a good next step up from a single-function task.

Contributors interested in any of these can comment on the issue to claim it; the discuss thread at https://github.com/apache/incubator-gluten/discussions is the right place for design questions before opening a PR.