[SPARK-57065][PYTHON][TESTS] Add with_sql_conf decorator in SQLTestUtils#56106
Open
zhengruifeng wants to merge 6 commits into
Open
[SPARK-57065][PYTHON][TESTS] Add with_sql_conf decorator in SQLTestUtils#56106zhengruifeng wants to merge 6 commits into
zhengruifeng wants to merge 6 commits into
Conversation
…lass duplication in parity tests
Many parity tests override setUpClass/tearDownClass solely to set a Spark
conf and unset it on teardown. This adds a `with_class_conf({K: V, ...})`
class decorator in `pyspark.testing.sqlutils` that wraps the inherited
setUpClass/tearDownClass to apply and revert the confs.
Migrates three callsites as proof:
- test_parity_column.py (2 classes)
- test_parity_udf_combinations.py (1 class; also makes the previously
asymmetric set-without-unset pair symmetric)
- test_parity_arrow_python_udf.py (5 classes)
Adds `WithClassConfTests` in test_utils.py to verify the decorator
applies the configured confs to the active SparkSession.
…ame to with_sql_conf Promotes the decorator to a static method of SQLTestUtils so it lives alongside the existing sql_conf context manager. Updates all callers and the test class accordingly. No behavior change.
… callsites Replaces `@SQLTestUtils.with_sql_conf(...)` with a local rebind `with_sql_conf = SQLTestUtils.with_sql_conf` per file, then `@with_sql_conf(...)` for the decorator. Keeps the decorator definition inside SQLTestUtils while making callsites read more like a regular imported helper.
…sqlutils Adds `with_sql_conf = SQLTestUtils.with_sql_conf` at module scope in pyspark.testing.sqlutils, so callsites can simply do `from pyspark.testing.sqlutils import with_sql_conf` instead of rebinding the static method locally. The decorator definition still lives on SQLTestUtils.
…Tests Mirrors WithSqlConfTests as WithSqlConfParityTests in test_parity_utils.py, extending ReusedConnectTestCase, so the decorator's behavior is regression- tested against both classic and Connect sessions.
…le scope Defines with_sql_conf as a plain module-level function in sqlutils.py instead of a static method on SQLTestUtils with a module-level alias. Callsites already import it via `from pyspark.testing.sqlutils import with_sql_conf` so no other changes are needed.
zhengruifeng
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds
with_sql_conf({K: V, ...})as a class decorator inpyspark.testing.sqlutilsthat wrapssetUpClass/tearDownClassto apply and revert Spark confs around the inherited fixtures.
Migrates three parity files as proof of usage:
test_parity_column.py(2 classes)test_parity_udf_combinations.py(1 class — also makes thepreviously asymmetric set-without-unset pair symmetric)
test_parity_arrow_python_udf.py(5 classes)Usage:
Why are the changes needed?
Many parity tests override
setUpClass/tearDownClasssolely toset one or two Spark confs and unset them on teardown. Each such
override is ~10 lines of identical-shape boilerplate, and the
hand-written
set/unsetpairs are easy to drift apart (one suchasymmetric pair is fixed by this PR). Consolidating to a single
declarative decorator removes the duplication and provides a common
seam for migrating the remaining callsites in a follow-up.
Does this PR introduce any user-facing change?
No — test infrastructure only.
How was this patch tested?
WithSqlConfTestsinpython/pyspark/sql/tests/test_utils.pyexercises the decorator end-to-end against
ReusedSQLTestCase,asserting both decorator-set keys land via
self.spark.conf.get(...).WithSqlConfParityTestsinpython/pyspark/sql/tests/connect/test_parity_utils.pyis theConnect counterpart, exercising the decorator against
ReusedConnectTestCase.session; in particular
ColumnParityTests.test_df_col_resolution_modeand its
WithNonStrictDFColResolutionsubclass read thedecorator-set conf back through
self.spark.conf.get(...)and pass,confirming the decorator composes through layered subclassing.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (model: claude-opus-4-7)