35 Pull requests opened by 28 people

• [SPARK-51534][Core] Improve parquet read performance by reducing unnecessary operations when converting parquet to row

  #50297 opened Mar 17, 2025

• [SPARK-51503][SQL] Support Variant type in XML scan

  #50300 opened Mar 17, 2025

• [SPARK-XXXXX][SQL] Add maxRecordsPerOutputBatch to limit the number of record of Arrow output batch

  #50301 opened Mar 18, 2025

• [SPARK-51540][PS][DOCS] Best practice for distributed-sequence misalignment case

  #50302 opened Mar 18, 2025

• [SPARK-50838][SQL] Add Cross Join as legal in recursion of Recursive CTE

  #50308 opened Mar 18, 2025

• [SPARK-51566][PYTHON] Python UDF traceback improvement

  #50313 opened Mar 18, 2025

• [SPARK-51187][SQL][SS] Introduce the migration logic of config removal from SPARK-49699

  #50314 opened Mar 19, 2025

• [SPARK-49082][SQL] Support widening Date to TimestampNTZ in Avro reader

  #50315 opened Mar 19, 2025

• [SPARK-51548][SQL] Provides configuration to decide whether to copy objects before shuffle.

  #50318 opened Mar 19, 2025

• [SPARK-40353][PS][CONNECT] Fix index nullable mismatch in `ps.read_excel`

  #50323 opened Mar 19, 2025

• [SPARK-51551] [ML] [PYTHON] [CONNECT] For tuning algorithm, allow using save / load to replace cache

  #50324 opened Mar 19, 2025

• [SPARK-51552] [SQL] Disallow temporary variables in persisted views when under identifier

  #50325 opened Mar 19, 2025

• [SPARK-51584][SQL] Add rule that pushes Project through Offset and Suite that tests it

  #50326 opened Mar 19, 2025

• [WIP][PYTHON] Fix ThreadPoolExecutor failure in python 3.13 daily test

  #50332 opened Mar 20, 2025

• [SPARK-51568][SQL] Introduce isSupportedExtract to prevent happening unexpected behavior

  #50333 opened Mar 20, 2025

• [SPARK-51537][CONNECT][CORE] construct classpath using both global jars and session specific jars on executor

  #50334 opened Mar 20, 2025

• [WIP][SPARK-51423][SQL] Add the current_time() function for TIME datatype

  #50336 opened Mar 20, 2025

• [SPARK-51578][PYTHON][TESTS] Add a helper function to fail time outed tests

  #50337 opened Mar 20, 2025

• [SPARK-51575][PYTHON] Combine Python Data Source pushdown & plan read workers

  #50340 opened Mar 20, 2025

• [SPARK-51118][PYTHON] Fix ExtractPythonUDFs to check the chained UDF input types for fallback

  #50341 opened Mar 20, 2025

• [SPARK-51576] Prohibit ANSI cast from `array<variant, containsNull = false>` to `array<_, containsNull = false>`

  #50343 opened Mar 20, 2025

• [SS][SPARK-51573] Fix Streaming State Checkpoint v2 checkpointInfo race condition

  #50344 opened Mar 21, 2025

• [SPARK-51580][SQL] Throw proper user facing error message when lambda function is out of place in `HigherOrderFunction`

  #50345 opened Mar 21, 2025

• [SPARK-51581][CORE][SQL] Use `nonEmpty`/`isEmpty` for empty check for explicit `Iterable`

  #50346 opened Mar 21, 2025

• [SPARK-51583] [SQL] Improve error message when to_timestamp function has arguments of wrong type

  #50347 opened Mar 21, 2025

• [SPARK-51586][SS] initialize input partitions independent of columnar support

  #50348 opened Mar 21, 2025

• [SPARK-51587][PYTHON][SS] Fix an issue where timestamp cannot be used in ListState when multiple state data is involved

  #50349 opened Mar 21, 2025

• [SPARK-48466][SQL][UI] Handle nested empty relation in SparkPlanInfo

  #50350 opened Mar 21, 2025

• [SPARK-51588][SQL] Validate default values handling in micro-batch writes

  #50351 opened Mar 21, 2025

• [SPARK-51589][SQL] Fix small bug failing to check for aggregate functions in |> SELECT

  #50352 opened Mar 22, 2025

• [SPARK-51585][SQL] Oracle dialect supports pushdown datetime functions

  #50353 opened Mar 22, 2025

• [SPARK-51419][SQL] Get hour of TIME datatype

  #50355 opened Mar 23, 2025

• [MINOR][SQL][TESTS] Check file based V2 datasources on unsupported types

  #50356 opened Mar 23, 2025

• [FOLLOW-UP][SPARK-42746][SQL] Fixing potential flakiness in ListAgg golden files

  #50357 opened Mar 23, 2025

• [SPARK-51590][SQL] Disable TIME in builtin file-based datasources

  #50358 opened Mar 23, 2025

29 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-51358] [SS] Introduce snapshot upload lag detection through StateStoreCoordinator

  #50123 commented on Mar 22, 2025 • 64 new comments

• [SPARK-51372][SQL] Introduce TableInfo for table creations

  #50137 commented on Mar 21, 2025 • 33 new comments

• [SPARK-51350][SQL] Implement Show Procedures

  #50109 commented on Mar 22, 2025 • 15 new comments

• [SPARK-51414][SQL] Add the make_time() function

  #50269 commented on Mar 22, 2025 • 12 new comments

• [SPARK-51270] Support nanosecond precision timestamp in Variant

  #50270 commented on Mar 18, 2025 • 7 new comments

• [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce

  #50273 commented on Mar 22, 2025 • 5 new comments

• [SPARK-51441][SQL] Add DSv2 APIs for constraints

  #50253 commented on Mar 21, 2025 • 5 new comments

• [WIP][SPARK-51395][SQL] Refine handling of default values in procedures

  #50197 commented on Mar 21, 2025 • 5 new comments

• [SPARK-51191][SQL] Validate default values handling in DELETE, UPDATE, MERGE

  #50271 commented on Mar 21, 2025 • 4 new comments

• [SPARK-51400] Replace ArrayContains nodes to InSet

  #50170 commented on Mar 21, 2025 • 3 new comments

• [SPARK-44856][PYTHON] Improve Python UDTF arrow serializer performance

  #50099 commented on Mar 18, 2025 • 3 new comments

• [SPARK-51513][SQL] Fix RewriteMergeIntoTable rule produces unresolved plan

  #50281 commented on Mar 23, 2025 • 2 new comments

• [SPARK-51479][SQL] Nullable in Row Level Operation Column is not correct

  #50246 commented on Mar 20, 2025 • 2 new comments

• [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys

  #50033 commented on Mar 18, 2025 • 1 new comment

• [SPARK-51338][INFRA] Add automated CI build for `connect-examples`

  #50187 commented on Mar 17, 2025 • 1 new comment

• [SPARK-50992][SQL] OOMs and performance issues with AQE in large plans

  #49724 commented on Mar 23, 2025 • 1 new comment

• [SPARK-51332][SQL] DS V2 supports push down BIT_AND, BIT_OR, BIT_XOR, BIT_COUNT and BIT_GET

  #50097 commented on Mar 18, 2025 • 0 new comments

• [SPARK-51252] [SS] Add instance metrics for last uploaded snapshot version in HDFS State Stores

  #50030 commented on Mar 21, 2025 • 0 new comments

• [WIP][SPARK-51411][SS][DOCS] Add documentation for the transformWithState operator

  #50177 commented on Mar 21, 2025 • 0 new comments

• [SPARK-51016][SQL] Stage.isIndeterminate gives wrong result in case the shuffle partitioner uses an inDeterministic attribute or expression.

  #50029 commented on Mar 20, 2025 • 0 new comments

• [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout

  #50198 commented on Mar 19, 2025 • 0 new comments

• [SPARK-51574][PYTHON] Filter serialization for Python Data Source filter pushdown

  #50252 commented on Mar 20, 2025 • 0 new comments

• [SPARK-50806][SQL] Support InputRDDCodegen interruption on task cancellation

  #49501 commented on Mar 19, 2025 • 0 new comments

• [WIP]: Spark 51272 51016 combined: For testing of HA Test

  #50263 commented on Mar 17, 2025 • 0 new comments

• [SPARK-50572][SQL] Fix race condition in CachedRDDBuilder.cachedColumnBuffers

  #49179 commented on Mar 24, 2025 • 0 new comments

• [SPARK-50417] Make number of FallbackStorage sub-directories configurable

  #48960 commented on Mar 20, 2025 • 0 new comments

• [SPARK-50354][SQL] No need to set initialInputBufferOffset when Aggregate mode is Complete

  #48895 commented on Mar 24, 2025 • 0 new comments

• [SPARK-37019][SQL] Add codegen support to array higher-order functions

  #34558 commented on Mar 19, 2025 • 0 new comments

• [SPARK-35564][SQL] Support subexpression elimination for conditionally evaluated expressions

  #32987 commented on Mar 17, 2025 • 0 new comments