September 25, 2026: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past week was one of the busiest on record, with roughly 60 pull requests and 20 issues touched across Velox, cuDF, Iceberg, Delta, Flink, and core. The dominant story is Delta-on-Gluten hardening: a large cluster of work from @felipepessoto is driving the known-failure baseline down (from 716 entries) while migrating the Delta CI builder to AlmaLinux 8. Alongside it, Spark 4.2 bring-up, ANSI-mode correctness, TIMESTAMP_NTZ, and a new native UDF/UDAF-by-name framework are all advancing. A steady stream of small, high-value core robustness fixes came in from @LuciferYang. Notably, almost every new PR now discloses generative AI tooling (Copilot, Claude, Gemini, Codex).

Key Ongoing Projects

Delta-on-Gluten stabilization (umbrella #12743) — @felipepessoto is the central driver, with a very large set of interlocking PRs:

• Native scan/metadata correctness: [GLUTEN-12743][VL] Support generated Delta DV metadata in native scans #13123, [VL] Keep synthetic scan metadata out of physical file schemas #13136, [GLUTEN-12743][VL] Fall back Delta scans requiring generated DV metadata #13042, and the observer-lifecycle fix in [GLUTEN-12743][VL] Preserve transaction observers in native Delta commands #13134.
• CI/test-context correctness: [MINOR][VL] Configure Gluten for all Delta Spark test contexts #13131 (uniform Delta test configuration) and the AlmaLinux 8 builder migration in [GLUTEN-12412][VL] Migrate Delta CI native builder to AlmaLinux 8 #13132.
• A shared JNI exception-transport foundation in [CORE] Preserve original Java exceptions across JNI #13133, which [VL] Translate native cast failures into Spark exceptions #13135 builds on to translate native cast failures into Spark exceptions.
• CI workarounds tracked to their root causes: [GLUTEN-12753][CI] Match Delta CDF filter assertions to Gluten plans #13129 and diagnostic drafts [GLUTEN-9003][VL] Disable IncrementMetric offload for Delta CI investigation #13112/[GLUTEN-9003][CORE] Disable IncrementMetric checks for Delta CI experiment #13113 for the IncrementMetric question in #9003.

Spark 4.2 support (#12569) — @akshaytayal landed the shim layer and build profile in #13126 (41/41 CI jobs green) and filed #13139 for the new KeyGroupedPartitioning scan layout. @manoj-ragupathy contributed version-agnostic compatibility fixes in #13020 and the enhanced DSv2 filter suites in #13023, with follow-on Spark 4.1.2 bump by @yaooqinn in #12177.

ANSI mode (#10134) — @luis4a0 is progressing the math side with #13047 (BROUND) and #13103 (ROUND), both gated on upstream Velox changes; @zhouyuan added the elt overlay in #12873; @anshulbaliga7 enforced format_number fallback restrictions in #13137.

TIMESTAMP_NTZ (#11622) — @felipepessoto enabled native min/max for Delta statistics in #12967 and preserved literals end-to-end in #13041.

Native UDF/UDAF by name — @kevinwilfong is building this out across #13105 (signature-less entries bound by Velox), #13124 (callable UDAF), and #13125 (MapType aggregate results).

cuDF / GPU — @ReemaAlzaid added row_constructor_with_null (#13104) and fixed host→GPU conversion in CudfVectorStream (#13067); the TPC-DS failure taxonomy is captured in #13122.

Data lake connectors — @infvg (Iceberg: #13073, #12901, #12364), @sezruby (Lance scans via Arrow C stream, #12958), @zhouyuan (experimental Kafka reads, #13130), @nielspardon (Substrait 0.98 ReadRel rebase, #12832).

Priority Items

• Native memory/teardown correctness — #12911 (reported by @boneanxs) is a long-running crash; @rui-mo's [GLUTEN-12911][VL] Fix memory pool holding in async thread #13038 proposes a ReleasableExecutor that avoids the deadlock of the earlier [GLUTEN-12911][VL] Fix memory pool holding in async thread #12919, but it is still a draft and needs review.
• Core robustness bugs from @LuciferYang — TaskResources release loop aborts on the first failing release, leaking the rest #13106 (resource release loop aborting), ShuffleManagerRegistry.register recursion guard is inverted, permits GlutenShuffleManager subclasses #13099 (inverted shuffle-manager guard), and SparkDirectoryUtil hangs when no Gluten local directory can be created #13097 (silent hang when no local dir can be created) each already have fixes: [GLUTEN-13106][CORE] Release all task resources even when one release fails #13107, [GLUTEN-13099][CORE] Fix inverted recursion guard in ShuffleManagerRegistry.register #13100, and [GLUTEN-13097][CORE] Fail fast when no Gluten local directory can be created #13098.
• Delta write fallback gaps — #13138 (DOUBLE partition columns) is currently worked around in [MINOR][VL] Configure Gluten for all Delta Spark test contexts #13131's baseline and would benefit from a runtime fix.
• Sort shuffle crash — #13120 (offset truncation above 128 MiB) has a candidate fix in [GLUTEN-13120][VL] Sort shuffle row ID truncates offsets above 128 MiB #13121 by @jianzhenwu.
• Dependency maintenance — a batch of Dependabot PRs (e.g. Bump actions/checkout from 4 to 7 #13082, Bump actions/download-artifact from 4 to 8 #13080, Bump actions/upload-artifact from 4 to 7 #13081, Bump docker/setup-buildx-action from 4.2.0 to 4.4.1 #13083) is waiting on review.

Notable Discussions

• Drop CentOS 7 support and set CentOS 8 as the minimum for Gluten #12312 by @zhouyuan — dropping CentOS 7 in favor of AlmaLinux 8 as the minimum platform, directly tied to the Velox Folly/FBThrift upgrade pressure. This is the most strategic discussion of the week.
• [Umbrella] Delta-on-Gluten: follow-ups after the Delta Spark UT pipeline #12743 by @felipepessoto — the Delta-on-Gluten umbrella. Worth reading for the 716→652 baseline trajectory, the IncrementMetric experiment results, and the proposed prebuilt-native-library cache.
• [VL] Fully arrow support in Spark and Gluten #13140 by @FelixYBW — a proposal for first-class Arrow support so Gluten stops re-implementing Spark's Python runners; references upstream SPARK-57468 and SPARK-56350.
• [VL] cuDF with CPU fallback disabled: TPC-DS SF1 failures and remaining CPU stages #13122 by @ReemaAlzaid — a detailed breakdown of 55 failing TPC-DS queries with cuDF fallback disabled, grouped into six root causes.
• [VL] Add ANSI mode support #10134 by @philo-he — the living ANSI-mode tracker, still open and accepting subtasks.

Emerging Trends

• Delta CI is maturing fast. Multiple PRs now remove verified-fixed entries from known-failures.txt rather than skipping tests, and temporary patches are being extracted into a dedicated apply-delta-test-patches.sh.
• JNI/exception correctness is becoming a shared concern. The JavaException carrier from [CORE] Preserve original Java exceptions across JNI #13133 is explicitly reused by [VL] Translate native cast failures into Spark exceptions #13135, suggesting a growing pattern of backend-neutral native/Java error transport rather than ad-hoc string envelopes.
• Thread-pool isolation. @yikf's [VL] Use a dedicated CPU thread pool for parallel broadcast hash-table build #13115 moves CPU-bound hash-table builds off the IO executor — the same class of fix that [GLUTEN-12911][VL] Fix memory pool holding in async thread #13038 is trying to make safe for async loads.
• AI-authored contributions are now the norm, not the exception. This is changing review dynamics: several PRs explicitly scope "what this does not claim" to keep AI-generated test evidence honest.
• Spark version matrix creep. With Spark 4.2 in progress, 4.1.2 bumping in, and Spark 3.3 deprecation closed ([VL] Deprecate Spark-33 support #12807), shim-layer and CI maintenance is an increasingly significant share of work.

Good First Issues

• #12992 by @jackylee-ch — remove now-constant >= 3.4 test gates. A purely mechanical cleanup across 14 files, ideal for a first contribution with no C++ or Spark-internals knowledge required.
• #6807 and #6811 by @taiyang-li — ClickHouse backend additions (split_part, MonotonicallyIncreasingID). Good entry points for anyone who wants to learn the ClickHouse expression pipeline.
• #10134 by @philo-he — ANSI mode is a large tracker, but many subtasks are small and well-scoped (e.g. rounding, date functions). Requires some Scala + Velox familiarity.
• #11622 by @rui-mo — TIMESTAMP_NTZ support has clearly marked checkboxes; the remaining item (enabling test files in VeloxSQLQueryTestSettings.scala) is a low-risk starting point.
• Core test/robustness fixes — the pattern set by @LuciferYang's [GLUTEN-13097][CORE] Fail fast when no Gluten local directory can be created #13098, [GLUTEN-13099][CORE] Fix inverted recursion guard in ShuffleManagerRegistry.register #13100, and [GLUTEN-13106][CORE] Release all task resources even when one release fails #13107 (each adding a focused suite plus a guard) is a great template for newcomers to the core module: small diff, clear regression test, no native build required.

Contributors this week included @felipepessoto, @kevinwilfong, @akshaytayal, @zhouyuan, @LuciferYang, @luis4a0, @rui-mo, @infvg, @sezruby, @ReemaAlzaid, @yikf, @marin-ma, @jianzhenwu, @minni31, @malinjawi, @FelixYBW, @wecharyu, @KushShriv, @anshulbaliga7, @JkSelf, @sabbasani, @vectorijk, @ggjh-159, @zhangxffff, @yaooqinn, @manoj-ragupathy, @Smallfu666, @acvictor, @WangGuangxin, @nielspardon, @boneanxs, @AgenticSpark, @zhanglistar, plus the automated GlutenPerfBot and Dependabot. Thank you all — and if you're reading this and haven't contributed yet, the Good First Issues above are waiting for you!