Skip to content

fix: support Spark Connect sessions - #548

Closed
patrickswedish wants to merge 1 commit into
capitalone:mainfrom
patrickswedish:agent/fix-spark-connect
Closed

fix: support Spark Connect sessions#548
patrickswedish wants to merge 1 commit into
capitalone:mainfrom
patrickswedish:agent/fix-spark-connect

Conversation

@patrickswedish

Copy link
Copy Markdown

Summary

  • dispatch Spark SQL functions and window expressions to the implementation matching the actual classic or Connect DataFrame/Column
  • apply the same dispatch throughout SparkSQLCompare and its built-in array, boolean, numeric, and string comparators
  • add regression coverage proving Connect expressions can be built without an active local SparkContext

Validation

  • ruff check and ruff format --check on every changed Python file
  • Spark Connect regression tests: passed
  • Spark comparator and helper suite: 34 passed
  • focused end-to-end SparkSQLCompare paths (merge, duplicate handling, mismatch reporting, windows, masking, and cache behavior): 8 passed

Closes #535

@patrickswedish
patrickswedish requested review from a team as code owners August 2, 2026 23:30
@fdosani

fdosani commented Aug 5, 2026

Copy link
Copy Markdown
Member

Appreciate the PR. I've made some changes here: main...spark-connect-sessions. There are a few PRs in the pipeline which need reviewing, so appreciate the patients here. I've incorporated some of your design into this new standalone branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SparkSQLCompare fails with Spark Connect sessions (pyspark.sql.connect.session.SparkSession)

2 participants