36 Pull requests opened by 27 people

• [SPARK-52535][SQL] Improve code readability of rule ApplyColumnarRulesAndInsertTransitions

  #51227 opened Jun 20, 2025

• [WIP][SPARK-51224][BUILD] Test Maven 4

  #51230 opened Jun 20, 2025

• [SPARK-52541] [SDP] Add programming guide for Declarative Pipelines

  #51234 opened Jun 20, 2025

• [SPARK-52544][SQL] Allow configuring Json datasource string length limit through SQLConf

  #51235 opened Jun 20, 2025

• Initial commit

  #51238 opened Jun 21, 2025

• [SPARK-51035][BUILD] Upgrade Janino to 3.1.12

  #51239 opened Jun 21, 2025

• [SPARK-52401][SQL] Fix DataFrame.collect() cache invalidation after saveAsTable append; add regression test

  #51240 opened Jun 21, 2025

• [SPARK-52401][SQL] Fix DataFrame.collect() cache invalidation after saveAsTable append; add regression test

  #51241 opened Jun 21, 2025

• [SPARK-52545][SQL] Double-quote should prioritize over implicit string concatenation

  #51242 opened Jun 22, 2025

• [SPARK-52551][SQL] Add a new v2 Predicate BOOLEAN_EXPRESSION

  #51247 opened Jun 23, 2025

• [SPARK-52552][SQL] Skip CHECK constraint enforcement for deletion vector deletes

  #51251 opened Jun 23, 2025

• [SPARK-52563][PS] Fix var naming bug in _assert_pandas_almost_equal

  #51253 opened Jun 23, 2025

• [SPARK-52560][BUILD] Bump ap-loader 4.0(v10) to support for async-profiler 4.0

  #51257 opened Jun 24, 2025

• [SPARK-52561][PYTHON][INFRA] Upgrade the minimum version of Python to 3.10

  #51259 opened Jun 24, 2025

• Test rocksdb 10.2.1

  #51262 opened Jun 24, 2025

• SPARK-52564 configuration changes not require deleting the checkpoint

  #51264 opened Jun 24, 2025

• [SPARK-50603][SQL] Respect user-provided basePath for streaming file source reads without glob

  #51267 opened Jun 24, 2025

• [SPARK-52565] [SQL] Enforce ordinal resolution before other sort order expressions

  #51268 opened Jun 24, 2025

• [SPARK-37467][SQL] Consolidate subexpression elimination code for whole stage and non-whole stage

  #51269 opened Jun 24, 2025

• [SPARK-37466][SQL] Support subexpression elimination in higher order functions

  #51272 opened Jun 24, 2025

• [SPARK-51885][SQL] Change AnalysisContext.outerPlan from Option[LogicalPlan] to Seq[LogicalPlan]

  #51274 opened Jun 24, 2025

• [SPARK-52570][PS] Enable divide-by-zero for numeric rmod with ANSI enabled

  #51275 opened Jun 24, 2025

• [SDP] [SPARK-52576] Drop/recreate on full refresh and MV update

  #51280 opened Jun 25, 2025

• [SPARK-52497][DOCS] Add docs for SQL user-defined functions

  #51281 opened Jun 25, 2025

• [SPARK-52575][SQL] Introduce contextIndependentFoldable attribute for Expressions

  #51282 opened Jun 25, 2025

• [SDP] [SPARK-52577] Add tests for Declarative Pipelines DatasetManager with Hive catalog

  #51283 opened Jun 25, 2025

• [CORE] Let LocalSparkContext clear active context in beforeAll

  #51284 opened Jun 25, 2025

• [SPARK-52578][SQL] Add metrics for rows to track case and action in MergeRowsExec

  #51285 opened Jun 25, 2025

• [SPARK-52582][SQL] Improve the memory usage of XML parser

  #51287 opened Jun 26, 2025

• [SPARK-50686][SQL] Hash to sort aggregation fallback - memory usage optimization

  #51290 opened Jun 26, 2025

• [WIP][PYTHON] Arrow UDF for aggregation

  #51292 opened Jun 26, 2025

• Fix-AQE-OOM

  #51295 opened Jun 26, 2025

• [SPARK-52590][SQL][TESTS] Add SQL query tests for SQL functions without explicit return types

  #51296 opened Jun 26, 2025

• [SPARK-52580][PS] Avoid CAST_INVALID_INPUT of `replace` in ANSI mode

  #51297 opened Jun 26, 2025

• [SPARK-52407][SQL] Add support for Theta Sketch

  #51298 opened Jun 27, 2025

• [SPARK-52547][INFRA][FOLLOW-UP] Set default GIT_BRANCH even for scheduled jobs

  #51299 opened Jun 27, 2025

28 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-52591][SDP] Validate streaming-ness of DFs returned by SDP table and standalone flow definitions

  #51208 commented on Jun 26, 2025 • 11 new comments

• [SPARK-52439][SQL] Support creating check constraint with NULL

  #51146 commented on Jun 23, 2025 • 11 new comments

• [SPARK-52503][SQL][CONNECT] Fix `drop` when the input column is not existent

  #51196 commented on Jun 26, 2025 • 6 new comments

• [SPARK-52494][SQL] Support colon-sign operator syntax to access Variant fields

  #51190 commented on Jun 26, 2025 • 5 new comments

• [DRAFT][PYTHON] Improve Python UDF Arrow Serializer Performance

  #51225 commented on Jun 26, 2025 • 4 new comments

• [SPARK-49386][SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill

  #47856 commented on Jun 26, 2025 • 3 new comments

• [SPARK-52381][CORE] JsonProtocol: Only accept subclasses of SparkListenerEvent

  #51061 commented on Jun 26, 2025 • 3 new comments

• [SPARK-52148][SQL] Fix CREATE OR REPLACE function for SQL user-defined TVFs

  #51191 commented on Jun 24, 2025 • 2 new comments

• [SPARK-51268][SQL] Make `TreeNode` lock-free

  #50019 commented on Jun 25, 2025 • 1 new comment

• [SPARK-52444][SQL][CONNECT] Add support for Variant/Char/Varchar Literal

  #51215 commented on Jun 26, 2025 • 0 new comments

• Fabric Spark - EH Connector

  #51211 commented on Jun 24, 2025 • 0 new comments

• [SPARK-52495] Allow including partition columns in the single variant column

  #51206 commented on Jun 23, 2025 • 0 new comments

• [SPARK-52486][SQL] Fix Spark Driver Planning OOM issue due to unworthwhile dpp expression before Execution when enabling AQE

  #51184 commented on Jun 25, 2025 • 0 new comments

• [WIP][INFRA] Retry SBT compilation when OOM

  #51149 commented on Jun 25, 2025 • 0 new comments

• [SPARK-51168][BUILD] Test Hadoop 3.4.2

  #51127 commented on Jun 24, 2025 • 0 new comments

• [SPARK-52327][Core] Glob based provider for history server

  #51045 commented on Jun 23, 2025 • 0 new comments

• [SPARK-52334][CORE][K8S] update all files, jars, and pyFiles to reference the working directory after they are downloaded

  #51037 commented on Jun 21, 2025 • 0 new comments

• [SPARK-47547] BloomFilter fpp degradation

  #50933 commented on Jun 26, 2025 • 0 new comments

• [SPARK-52024][SQL] Support cancel ShuffleQueryStage when propagate empty relations

  #50814 commented on Jun 24, 2025 • 0 new comments

• [SPARK-51359][CORE][SQL] Set INT64 as the default timestamp type for Parquet files

  #50215 commented on Jun 25, 2025 • 0 new comments

• [SPARK-51332][SQL] DS V2 supports push down BIT_AND, BIT_OR, BIT_XOR, BIT_COUNT and BIT_GET

  #50097 commented on Jun 27, 2025 • 0 new comments

• [BUILD] Upgrade `RoaringBitmap` to 1.4.1

  #49710 commented on Jun 24, 2025 • 0 new comments

• [SPARK-49547][SQL][PYTHON] Add iterator of `RecordBatch` API to `applyInArrow`

  #49005 commented on Jun 24, 2025 • 0 new comments

• [SPARK-49984][CORE] Fix duplicate JVM options

  #48488 commented on Jun 24, 2025 • 0 new comments

• [SPARK-22876][YARN] Respect YARN AM failure validity interval

  #42570 commented on Jun 24, 2025 • 0 new comments

• [SPARK-44639][SS][YARN] Use Java tmp dir for local RocksDB state storage on Yarn

  #42301 commented on Jun 25, 2025 • 0 new comments

• [SPARK-37019][SQL] Add codegen support to array higher-order functions

  #34558 commented on Jun 24, 2025 • 0 new comments

• [SPARK-35564][SQL] Support subexpression elimination for conditionally evaluated expressions

  #32987 commented on Jun 24, 2025 • 0 new comments