[SPARK-56861][PS][TESTS] Fix ArrowInterfaceTests being silently skipped by unittest#55870
Closed
zhengruifeng wants to merge 1 commit into
Closed
[SPARK-56861][PS][TESTS] Fix ArrowInterfaceTests being silently skipped by unittest#55870zhengruifeng wants to merge 1 commit into
ArrowInterfaceTests being silently skipped by unittest#55870zhengruifeng wants to merge 1 commit into
Conversation
`ArrowInterfaceTests` was declared as
class ArrowInterfaceTests(ArrowInterfaceTestsMixin, SQLTestUtils):
pass
`SQLTestUtils` is a plain mixin that does not extend `unittest.TestCase`, so the class itself is not a `TestCase` subclass and `unittest`'s discovery (`TestLoader.loadTestsFromModule`) silently skips it. Verified with `loader.loadTestsFromModule(...).countTestCases()`:
- Before: 0
- After: 3 (including `test_spark_arrow_c_streamer_arrow_consumer`)
The sibling Spark Connect parity test (`ArrowInterfaceParityTests`) extends `ReusedConnectTestCase` and so was unaffected, which masked the issue.
Fix by switching the base to `PandasOnSparkTestCase`, matching the other `pyspark.pandas.tests.test_*` classes (e.g. `NamespaceTests`, `UtilsTests`, `InternalFrameTests`, `SQLTests`). `PandasOnSparkTestCase` chains through `ReusedSQLTestCase -> ReusedPySparkTestCase -> PySparkBaseTestCase -> unittest.TestCase`, so the test is now collected and run.
Generated-by: Claude opus-4-7
zhengruifeng
changed the title
ArrowInterfaceTests being silently skipped by unittestArrowInterfaceTests being silently skipped by unittest
HyukjinKwon
approved these changes
May 14, 2026
zhengruifeng
added a commit
that referenced
this pull request
May 14, 2026
…pped by unittest
### What changes were proposed in this pull request?
Fix the base class of `ArrowInterfaceTests` in `python/pyspark/pandas/tests/test_arrow_interface.py` so the test is actually discovered by `unittest`.
The class was declared as:
```python
class ArrowInterfaceTests(ArrowInterfaceTestsMixin, SQLTestUtils):
pass
```
`SQLTestUtils` is a plain mixin that does not extend `unittest.TestCase`, so this class itself is not a `TestCase` subclass and `unittest`'s discovery (`TestLoader.loadTestsFromModule`) silently skips it.
Verified with `loader.loadTestsFromModule(...).countTestCases()`:
| | tests collected | `issubclass(TestCase)` |
| --- | --- | --- |
| Before fix | 0 | `False` |
| After fix | 3 | `True` |
The sibling Spark Connect parity test `ArrowInterfaceParityTests` in `tests/connect/test_parity_arrow_interface.py` extends `ReusedConnectTestCase` (which does chain to `TestCase`) and so was unaffected, masking the issue.
Fix: switch the base to `PandasOnSparkTestCase`, matching the other `pyspark.pandas.tests.test_*` classes (e.g. `NamespaceTests`, `UtilsTests`, `InternalFrameTests`, `SQLTests`). `PandasOnSparkTestCase` chains through `ReusedSQLTestCase -> ReusedPySparkTestCase -> PySparkBaseTestCase -> unittest.TestCase`.
### Why are the changes needed?
`test_spark_arrow_c_streamer_arrow_consumer` (added in SPARK-55124) has not actually been running in CI since it was introduced; it is a latent gap in coverage of the pandas-on-Spark Arrow C-stream interface.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
- Discovery check (above): `unittest.TestLoader.loadTestsFromModule` now collects the test.
- Ran the test locally:
```
$ python -m unittest -v pyspark.pandas.tests.test_arrow_interface
test_assert_classic_mode (...ArrowInterfaceTests) ... ok
test_spark_arrow_c_streamer_arrow_consumer (...ArrowInterfaceTests) ... ok
test_assert_classic_mode (...PandasOnSparkTestCase) ... ok
----------------------------------------------------------------------
Ran 3 tests in 10.683s
OK
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude opus-4-7
Closes #55870 from zhengruifeng/SPARK-arrow-interface-tests.
Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
(cherry picked from commit 9ab8e43)
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
Contributor
Author
|
merged to master/4.x |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix the base class of
ArrowInterfaceTestsinpython/pyspark/pandas/tests/test_arrow_interface.pyso the test is actually discovered byunittest.The class was declared as:
SQLTestUtilsis a plain mixin that does not extendunittest.TestCase, so this class itself is not aTestCasesubclass andunittest's discovery (TestLoader.loadTestsFromModule) silently skips it.Verified with
loader.loadTestsFromModule(...).countTestCases():issubclass(TestCase)FalseTrueThe sibling Spark Connect parity test
ArrowInterfaceParityTestsintests/connect/test_parity_arrow_interface.pyextendsReusedConnectTestCase(which does chain toTestCase) and so was unaffected, masking the issue.Fix: switch the base to
PandasOnSparkTestCase, matching the otherpyspark.pandas.tests.test_*classes (e.g.NamespaceTests,UtilsTests,InternalFrameTests,SQLTests).PandasOnSparkTestCasechains throughReusedSQLTestCase -> ReusedPySparkTestCase -> PySparkBaseTestCase -> unittest.TestCase.Why are the changes needed?
test_spark_arrow_c_streamer_arrow_consumer(added in SPARK-55124) has not actually been running in CI since it was introduced; it is a latent gap in coverage of the pandas-on-Spark Arrow C-stream interface.Does this PR introduce any user-facing change?
No. Test-only change.
How was this patch tested?
unittest.TestLoader.loadTestsFromModulenow collects the test.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude opus-4-7