38 Pull requests opened by 28 people

• [SPARK-51372][SQL] Introduce a builder pattern in TableCatalog

  #50137 opened Mar 3, 2025

• [SPARK-51380][SQL] Make sure JDBC dialect could check the supported functions by Introduce functionToSQL and aggregateFunctionToSQL

  #50143 opened Mar 4, 2025

• [MINOR][DOCS] Add `spark.taskMetrics.trackUpdatedBlockStatuses` description for configuration.md

  #50147 opened Mar 4, 2025

• Test sbt 1.10.10

  #50151 opened Mar 4, 2025

• [WIP] [SPARK-51097] [SS] Split apart SparkPlan metrics and instance metrics

  #50157 opened Mar 4, 2025

• [SPARK-51397][SS] Fix maintenance pool shutdown handling issue causing long test times

  #50168 opened Mar 5, 2025

• [SPARK-51400] Replace ArrayContains nodes to InSet

  #50170 opened Mar 5, 2025

• [SPARK-51387][BUILD][4.0] Upgrade Netty to 4.1.119.Final

  #50174 opened Mar 5, 2025

• [SPARK-51409][SS] Add error classification in the changelog writer creation path

  #50176 opened Mar 6, 2025

• [WIP][SPARK-51411][SS][DOCS] Add documentation for the transformWithState operator

  #50177 opened Mar 6, 2025

• [SPARK-51365][TESTS] Test maven + macos

  #50178 opened Mar 6, 2025

• [SPARK-51281][SQL][3.5] DataFrameWriterV2 should respect the path option

  #50179 opened Mar 6, 2025

• [SPARK-51418][SQL] Fix DataSource PARTITON TABLE w/ Hive type incompatible partition columns

  #50182 opened Mar 6, 2025

• [SPARK-48311][SQL] Fix nested pythonUDF in groupBy and aggregate in Binding Exception

  #50183 opened Mar 6, 2025

• [MINOR][DOCS] Remove u prefix for py str

  #50185 opened Mar 6, 2025

• [MINOR][DOCS] IP -> HOST

  #50186 opened Mar 6, 2025

• [SPARK-51338][INFRA] Add automated CI build for `connect-examples`

  #50187 opened Mar 6, 2025

• [MINOR][DOCS] Update foreachbatch docs

  #50188 opened Mar 6, 2025

• [SPARK-51442][SQL] Add time formatters

  #50190 opened Mar 6, 2025

• [WIP][SPARK-51428][SQL] Reassign Aliases for collated expression trees deterministically

  #50192 opened Mar 6, 2025

• [SPARK-51429][Connect] Add "Acknowledgement" message to ExecutePlanResponse

  #50193 opened Mar 6, 2025

• [SPARK-51402][SQL][TESTS] Test TimeType in UDF

  #50194 opened Mar 6, 2025

• [SPARK-51097] [SS] Re-introduce RocksDB state store's last uploaded snapshot version instance metrics

  #50195 opened Mar 6, 2025

• [WIP][SPARK-51395][SQL] Refine handling of default values in procedures

  #50197 opened Mar 6, 2025

• [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout

  #50198 opened Mar 7, 2025

• [WIP][ML][CONNECT] ML transformed dataframe keep a reference to the model

  #50199 opened Mar 7, 2025

• log executing SQL text

  #50202 opened Mar 7, 2025

• session extension comments enhancement

  #50204 opened Mar 7, 2025

• fix: handle compare_vals turn into str when parse IgnoreColumnType

  #50205 opened Mar 7, 2025

• [SPARK-51365][SQL][TESTS] Reduce `SHUFFLE_EXCHANGE_MAX_THREAD_THRESHOLD/RESULT_QUERY_STAGE_MAX_THREAD_THRESHOLD` for tests related to `SharedSparkSession/TestHive` when using `macOS + Apple Silicon` on GitHub Action

  #50206 opened Mar 7, 2025

• [SPARK-51436][CORE][SQL][K8s][SS] Fix bug that cancel Future specified mayInterruptIfRunning with true

  #50209 opened Mar 7, 2025

• [SPARK-51438][SQL] Make CatalystDataToProtobuf and ProtobufDataToCatalyst properly comparable and hashable

  #50212 opened Mar 7, 2025

• [WIP][SPARK-51348][BUILD][SQL] Upgrade Hive to 4.0

  #50213 opened Mar 7, 2025

• [SPARK-51440] classify the NPE when null topic field value is in kafka message data and there is no topic option

  #50214 opened Mar 7, 2025

• [SPARK-51359][CORE][SQL] Set INT64 as the default timestamp type for Parquet files

  #50215 opened Mar 8, 2025

• Change host to ip

  #50216 opened Mar 8, 2025

• [SPARK-51443] Fix singleVariantColumn in DSv2 and readStream.

  #50217 opened Mar 8, 2025

• [SPARK-51444][CORE] Remove the unreachable `if` branch from `TaskSchedulerImpl#statusUpdate`

  #50218 opened Mar 9, 2025

41 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-50892][SQL]Add UnionLoopExec, physical operator for recursion, to perform execution of recursive queries

  #49955 commented on Mar 7, 2025 • 33 new comments

• [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys

  #50033 commented on Mar 8, 2025 • 27 new comments

• [SPARK-51358] [SS] Introduce snapshot upload lag detection through StateStoreCoordinator

  #50123 commented on Mar 7, 2025 • 23 new comments

• [SPARK-51271][PYTHON] Add filter pushdown API to Python Data Sources

  #49961 commented on Mar 8, 2025 • 22 new comments

• [SPARK-50763][SQL] Add Analyzer rule for resolving SQL table functions

  #49471 commented on Mar 7, 2025 • 14 new comments

• [SPARK-44856][PYTHON] Improve Python UDTF arrow serializer performance

  #50099 commented on Mar 7, 2025 • 11 new comments

• [SPARK-51340][ML][CONNECT] Model size estimation for linear classification & regression models

  #50106 commented on Mar 7, 2025 • 9 new comments

• [SPARK-51350][SQL] Implement Show Procedures

  #50109 commented on Mar 7, 2025 • 8 new comments

• [SPARK-51349][SQL][TESTS] Change precedence of null and "null" in sorting in QueryTest

  #50108 commented on Mar 5, 2025 • 6 new comments

• [SPARK-51334][CONNECT] Add java/scala version in analyze spark_version response

  #50102 commented on Mar 7, 2025 • 3 new comments

• [SPARK-51252] [SS] Add instance metrics for last uploaded snapshot version in HDFS State Stores

  #50030 commented on Mar 7, 2025 • 3 new comments

• [SPARK-51298][SQL] Support variant in CSV scan

  #50052 commented on Mar 8, 2025 • 2 new comments

• [SPARK-43221][CORE] Host local block fetching should use a block status of a block stored on disk

  #50122 commented on Mar 8, 2025 • 2 new comments

• [SPARK-47573][K8S] Support custom driver log url

  #45728 commented on Mar 5, 2025 • 2 new comments

• [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC

  #49528 commented on Mar 6, 2025 • 2 new comments

• [SPARK-51182][SQL] DataFrameWriter should throw dataPathNotSpecifiedError when path is not specified

  #49928 commented on Mar 8, 2025 • 2 new comments

• [SPARK-51301][BUILD] Bump zstd-jni 1.5.7-1

  #50057 commented on Mar 3, 2025 • 1 new comment

• [SPARK-51069][SQL] Add big-endian support to UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl

  #49773 commented on Mar 6, 2025 • 1 new comment

• [SPARK-51332][SQL] DS V2 supports push down BIT_AND, BIT_OR, BIT_XOR, BIT_COUNT and BIT_GET

  #50097 commented on Mar 8, 2025 • 0 new comments

• [SPARK-51016][SQL] Stage.isIndeterminate gives wrong result in case the shuffle partitioner uses an inDeterministic attribute or expression.

  #50029 commented on Mar 7, 2025 • 0 new comments

• [SPARK-51256][SQL] Increase parallelism if joining with small bucket table

  #50004 commented on Mar 3, 2025 • 0 new comments

• [MINOR][SQL] Format the SqlBaseParser.g4

  #49987 commented on Mar 8, 2025 • 0 new comments

• [SPARK-51187][SQL][SS] Implement the graceful deprecation of incorrect config introduced in SPARK-49699

  #49983 commented on Mar 4, 2025 • 0 new comments

• [SPARK-37019][SQL] Add codegen support to array higher-order functions

  #34558 commented on Mar 7, 2025 • 0 new comments

• [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC using MERGE INTO with temp table

  #41611 commented on Mar 5, 2025 • 0 new comments

• [SPARK-44639][SS][YARN] Use Java tmp dir for local RocksDB state storage on Yarn

  #42301 commented on Mar 6, 2025 • 0 new comments

• [SPARK-22876][YARN] Respect YARN AM failure validity interval

  #42570 commented on Mar 6, 2025 • 0 new comments

• [SPARK-48821][SQL] Support `Update` in `DataFrameWriterV2`

  #47233 commented on Mar 7, 2025 • 0 new comments

• [MINOR][INFRA] Do not upload docker build record

  #48012 commented on Mar 5, 2025 • 0 new comments

• [SPARK-50188][CONNECT][PYTHON] When the connect client starts, print the server's webUrl

  #48720 commented on Mar 7, 2025 • 0 new comments

• [SPARK-50417] Make number of FallbackStorage sub-directories configurable

  #48960 commented on Mar 5, 2025 • 0 new comments

• Remove session string calls

  #48974 commented on Mar 9, 2025 • 0 new comments

• [SPARK-33152][SQL] Implement new optimized logic for constraint propagation rule

  #49117 commented on Mar 3, 2025 • 0 new comments

• [SPARK-50639][SQL] Improve warning logging in CacheManager

  #49276 commented on Mar 6, 2025 • 0 new comments

• [SPARK-50903][CONNECT] Cache logical plans after analysis

  #49584 commented on Mar 5, 2025 • 0 new comments

• [SPARK-50992][SQL] OOMs and performance issues with AQE in large plans

  #49724 commented on Mar 7, 2025 • 0 new comments

• [CORE][CONNECT] Add cogroup and cogroupSorted variants for additional KeyValueGroupedDatasets.

  #49754 commented on Mar 6, 2025 • 0 new comments

• [SPARK-51094][SQL][TESTS] Adjust some test parameters to work on s390x

  #49813 commented on Mar 6, 2025 • 0 new comments

• [SPARK-51149][CORE] Log classpath in SparkSubmit on ClassNotFoundException

  #49870 commented on Mar 6, 2025 • 0 new comments

• Log exception on Python runner termination

  #49890 commented on Mar 5, 2025 • 0 new comments

• [WIP][SPARK-51180][BUILD] Upgrade Arrow to 19.0.0

  #49909 commented on Mar 5, 2025 • 0 new comments