[SPARK-58602][PYTHON][TESTS][FOLLOWUP] Fix the test entry point of test_pyarrow_array_from_pandas_default - #57864
Closed
Spenserrrr wants to merge 1 commit into
Closed
[SPARK-58602][PYTHON][TESTS][FOLLOWUP] Fix the test entry point of test_pyarrow_array_from_pandas_default#57864Spenserrrr wants to merge 1 commit into
Spenserrrr wants to merge 1 commit into
Conversation
…st_pyarrow_array_from_pandas_default ### What changes were proposed in this pull request? Change the `__main__` entry point of `test_pyarrow_array_from_pandas_default.py` from `main(globals()["__file__"])` to a bare `main()`. ### Why are the changes needed? `pyspark.testing.main(module=None)` expects either no argument (it then resolves the real module name via `__spec__`) or a module name -- it forwards its argument straight to `unittest.main(module=...)`. Passing `globals()["__file__"]` hands it a filesystem path, which `unittest` tries to import as a module name and fails: ``` ModuleNotFoundError: No module named '/__w/spark/spark/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default' ``` This breaks the `pyspark-sql` test module on master (the run aborts with exit code 19 before the rest of the shard completes). Every other pyspark test file uses a bare `main()`; this was the only one passing a path. ### Does this PR introduce _any_ user-facing change? No. Test-only fix. ### How was this patch tested? Ran the module both ways locally. Before, `python -m pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_default` raised `ModuleNotFoundError`; after, it runs to completion (`Ran 1 test ... OK`). `dev/lint-python --ruff` clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8)
Spenserrrr
marked this pull request as ready for review
August 7, 2026 22:46
Contributor
Author
|
Hi @zhengruifeng @Yicong-Huang! This is a one-line follow-up fixing a CI break on master: test_pyarrow_array_from_pandas_default.py calls main(globals()["file"]), which passes a file path to unittest.main(module=...) and fails the whole pyspark-sql shard with ModuleNotFoundError (exit code 19). Every other pyspark test uses a bare main(). This restores that. Could one of you take a look and merge when you have a moment, since it's currently red on master? |
uros-b
approved these changes
Aug 8, 2026
Member
|
Thank you @Spenserrrr and @Yicong-Huang! |
uros-b
pushed a commit
that referenced
this pull request
Aug 8, 2026
…t_pyarrow_array_from_pandas_default ### What changes were proposed in this pull request? Change the `__main__` entry point of `test_pyarrow_array_from_pandas_default.py` from `main(globals()["__file__"])` to a bare `main()`. ### Why are the changes needed? `pyspark.testing.main(module=None)` expects either no argument (it then resolves the real module name via `__spec__`) or a module name -- it forwards its argument straight to `unittest.main(module=...)`. Passing `globals()["__file__"]` hands it a filesystem path, which `unittest` tries to import as a module name and fails: ``` ModuleNotFoundError: No module named '/__w/spark/spark/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default' ``` This aborts the `pyspark-sql` test module on master (exit code 19) before the rest of the shard completes, so it is currently failing CI on master and on open PRs. Every other pyspark test file uses a bare `main()`; this was the only one passing a path. ### Does this PR introduce _any_ user-facing change? No. Test-only fix. ### How was this patch tested? Ran the module both ways locally. Before, `python -m pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_default` raised `ModuleNotFoundError`; after, it runs to completion (`Ran 1 test ... OK`). `dev/lint-python --ruff` is clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes #57864 from Spenserrrr/from-pandas-main-fix. Authored-by: Spenser Sun <hsun112358@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit e2f176e) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Change the
__main__entry point oftest_pyarrow_array_from_pandas_default.pyfrommain(globals()["__file__"])to a baremain().Why are the changes needed?
pyspark.testing.main(module=None)expects either no argument (it then resolves the real module name via__spec__) or a module name -- it forwards its argument straight tounittest.main(module=...). Passingglobals()["__file__"]hands it a filesystem path, whichunittesttries to import as a module name and fails:This aborts the
pyspark-sqltest module on master (exit code 19) before the rest of the shard completes, so it is currently failing CI on master and on open PRs. Every other pyspark test file uses a baremain(); this was the only one passing a path.Does this PR introduce any user-facing change?
No. Test-only fix.
How was this patch tested?
Ran the module both ways locally. Before,
python -m pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_defaultraisedModuleNotFoundError; after, it runs to completion (Ran 1 test ... OK).dev/lint-python --ruffis clean.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)