31 Pull requests opened by 24 people

• Supports file pruning when underlying Delta Parquet files are explici…

  #51049 opened May 29, 2025

• [SPARK-52346][SQL] Declarative Pipeline `DataflowGraph` execution and event logging

  #51050 opened May 29, 2025

• [SPARK-48356][FOLLOW UP][SQL] Improve FOR statement's column schema inference

  #51053 opened May 30, 2025

• [SPARK-52353][SQL] Fix bug with wrong constraints in LogicalRDDs referencing previous iterations

  #51056 opened May 30, 2025

• [SPARK-52348][CONNECT] Add support for Spark Connect handlers for pipeline commands

  #51057 opened May 30, 2025

• [SPARK-52356][PS] Enable divide-by-zero for boolean mod/rmod with ANSI enabled

  #51058 opened May 30, 2025

• [SPARK-52381][Core] JsonProtocol: Only accept subclasses of SparkListenerEvent

  #51061 opened Jun 2, 2025

• [SPARK-52354] Add type coercion to UnionLoop

  #51063 opened Jun 2, 2025

• [SQL] Give a separate error class to failures in `BindReferences`

  #51066 opened Jun 2, 2025

• [SPARK-47404][SQL] Add configurable size limits for ANTLR DFA cache

  #51069 opened Jun 3, 2025

• [SPARK-52388][SQL] Handle named and positional parameters under `PlanWithUnresolvedIdentifier`

  #51073 opened Jun 3, 2025

• Dev 132930 spark image research

  #51075 opened Jun 3, 2025

• Creating minimal CVE spark base image

  #51076 opened Jun 3, 2025

• [SPARK-52391][SS] Refactor TransformWithStateExec to extract shared functions and variables into an abstract base class for Scala and Python

  #51077 opened Jun 3, 2025

• [SPARK-52356][PS] Enable divide-by-zero for boolean floordiv with ANSI enabled

  #51079 opened Jun 3, 2025

• [SPARK-52393][SDP] Pipeline SQL Graph Element Registration

  #51080 opened Jun 3, 2025

• [SPARK-50815][PYTHON] Allow Variant in Python Datasources

  #51081 opened Jun 4, 2025

• [DO NOT MERGE YET] Run tests after removing Unnecessary Variant Code

  #51082 opened Jun 4, 2025

• [SPARK-52396][SQL] Artifact Root Directory should use tmpdir

  #51083 opened Jun 4, 2025

• [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches

  #51084 opened Jun 4, 2025

• [MINOR] Improve documentation of concat functions in pyspark

  #51086 opened Jun 4, 2025

• [MINOR] Ensure structured_logging_style runs out of correct directory

  #51087 opened Jun 4, 2025

• [SPARK-52400] [SQL] [TESTS] Cover SQL test gaps in golden files

  #51088 opened Jun 4, 2025

• Simplify build files for pipelines module

  #51089 opened Jun 4, 2025

• [SPARK-52402][PS] Fix divide-by-zero error in groupby().corr('kendall') with ANSI mode enabled

  #51090 opened Jun 4, 2025

• [SPARK-52403][SQL] Add numTargetRowsCopied metric to MergeRowExec

  #51091 opened Jun 4, 2025

• [SPARK-52235][SQL][FOLLOW-UP] Add ANSI mode check for new tests

  #51092 opened Jun 5, 2025

• Spark 52386 followup

  #51093 opened Jun 5, 2025

• [SPARK-52238][TESTS][FOLLOW-UP] Fix test failure in classic-only envs

  #51094 opened Jun 5, 2025

• [SPARK-52384][CONNECT][branch-4.0] Fix bug Connect should case insensitive for JDBC options

  #51095 opened Jun 5, 2025

• [SPARK-52399][Kubernetes] Support local ephemeral-storage resource

  #51096 opened Jun 5, 2025

18 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-52187][SQL] Introduce Join pushdown for DSv2

  #50921 commented on Jun 4, 2025 • 13 new comments

• [SPARK-52265][SQL][TEST] Fix regex leading to empty PROCESS_TABLES.testingVersions in HiveExternalCatalogVersionsSuite

  #50989 commented on Jun 5, 2025 • 6 new comments

• [SPARK-51831][SQL] Column pruning with existsJoin for Datasource V2

  #51046 commented on Jun 5, 2025 • 5 new comments

• [SPARK-52334][CORE][K8S] update all files, jars, and pyFiles to reference the working directory after they are downloaded

  #51037 commented on Jun 5, 2025 • 4 new comments

• [SPARK-52082][PYTHON][DOCS] Improve ExtractPythonUDF docs

  #50867 commented on Jun 4, 2025 • 4 new comments

• [SPARK-47547] BloomFilter fpp degradation

  #50933 commented on May 30, 2025 • 4 new comments

• [SPARK-52288][PS] Avoid INVALID_ARRAY_INDEX in `split`/`rsplit` when ANSI mode is on

  #51006 commented on Jun 5, 2025 • 3 new comments

• [SPARK-52065][SQL] Produce another plan tree with output columns (name, data type, nullability) in plan change logging

  #50852 commented on May 29, 2025 • 1 new comment

• [SPARK-52088][CORE] Refactor the `ClosureCleaner#clean` and avoid using the `-Djdk.reflect.useDirectMethodHandle=false`

  #51028 commented on May 30, 2025 • 1 new comment

• Enable -Xsource:3 compiler flag

  #50474 commented on May 29, 2025 • 1 new comment

• [SPARK-52186][SQL] Rewrite EXISTS to add Scalar Project

  #50917 commented on Jun 5, 2025 • 0 new comments

• [Spark-50873][SQL] Prune column for ExistsSubquery in DSV2

  #50916 commented on Jun 5, 2025 • 0 new comments

• [SPARK-44223][CORE] Remove LevelDB support

  #51027 commented on Jun 2, 2025 • 0 new comments

• [SPARK-52024][SQL] Support cancel ShuffleQueryStage when propagate empty relations

  #50814 commented on Jun 2, 2025 • 0 new comments

• Added s3 offloading technique for RDD

  #51042 commented on Jun 4, 2025 • 0 new comments

• [SPARK-52339][SQL] Fix comparison of `InMemoryFileIndex` instances

  #51043 commented on May 29, 2025 • 0 new comments

• Move `DefineSqlGraphElements` to `PipelineCommand`

  #51044 commented on May 30, 2025 • 0 new comments

• [SPARK-52327][Core] Glob based provider for history server

  #51045 commented on Jun 2, 2025 • 0 new comments