[SPARK-58553][PS][TESTS][FOLLOWUP] Skip NumPy comparison tests where NumPy's own result is platform-dependent - #58257
Closed
Spenserrrr wants to merge 1 commit into
Closed
Conversation
…NumPy's own result is platform-dependent np.reciprocal(int 0) and the fmax/fmin signed-zero tie are unspecified by C/IEEE, so NumPy's own answer varies by CPU architecture (x86-64 casts the intermediate +inf to INT64_MIN, aarch64 saturates to INT64_MAX) and by NumPy version (the scalar tie returns the first operand from 2.3.0 on, the second before). pandas-on-Spark deliberately returns one fixed value on every platform, so the locally installed NumPy is only a valid reference on the environment that value was verified against. Skip test_np_reciprocal_integer and test_np_fmax_fmin unless running on Linux x86-64 with NumPy >= 2.3.0, and exclude reciprocal from the random-integer sweeps there, which fixes the scheduled ARM and macOS jobs. The implementation is unchanged.
Spenserrrr
marked this pull request as ready for review
August 24, 2026 20:27
Contributor
Author
|
Hi @zhengruifeng! This is the PR skipping test for fmax/fmin and reciprocal on macOS and ARM CI. Could you take a look when you have time? Thanks! cc @Yicong-Huang @HyukjinKwon |
This was referenced Aug 24, 2026
zhengruifeng
approved these changes
Aug 25, 2026
zhengruifeng
pushed a commit
that referenced
this pull request
Aug 25, 2026
…umPy's own result is platform-dependent ### What changes were proposed in this pull request? Skip the NumPy-comparison tests for `reciprocal` and `fmax`/`fmin` unless running on Linux x86-64 with NumPy >= 2.3.0, and exclude `reciprocal` from the random-integer sweeps (`test_np_spark_compat_series` / `_frame`) elsewhere. Test-only change; `numpy_compat.py` is untouched. This is an alternative to #58019, which instead changed the implementation to fall back to `pandas_udf` off that platform. ### Why are the changes needed? `np.reciprocal(int 0)` and the `fmax`/`fmin` signed-zero tie are unspecified by C/IEEE, so NumPy's own answer varies by CPU architecture (x86-64 casts the intermediate `+inf` to `INT64_MIN`, aarch64 saturates to `INT64_MAX`) and by NumPy version (the scalar tie returns the first operand from 2.3.0 on, the second before). pandas-on-Spark returns one fixed value everywhere, so NumPy is only a valid reference on the platform it was verified against. The scheduled ARM and macOS jobs fail on `test_np_fmax_fmin`, `test_np_reciprocal_integer`, and intermittently `test_np_spark_compat_series`. ### Does this PR introduce _any_ user-facing change? No. Tests only. ### How was this patch tested? Existing tests. On Linux x86-64 with NumPy 2.4.1 both tests still run and pass; with `platform.machine`, `platform.system`, and `np.__version__` patched to macOS arm64, Linux aarch64, and NumPy 1.23.2, they are skipped as intended. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 Closes #58257 from Spenserrrr/numpy-crossplatform-skip-tests. Authored-by: Spenser Sun <hsun112358@gmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com> (cherry picked from commit 204a73d) Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Skip the NumPy-comparison tests for
reciprocalandfmax/fminunless running on Linux x86-64 with NumPy >= 2.3.0, and excludereciprocalfrom the random-integer sweeps (test_np_spark_compat_series/_frame) elsewhere. Test-only change;numpy_compat.pyis untouched.This is an alternative to #58019, which instead changed the implementation to fall back to
pandas_udfoff that platform.Why are the changes needed?
np.reciprocal(int 0)and thefmax/fminsigned-zero tie are unspecified by C/IEEE, so NumPy's own answer varies by CPU architecture (x86-64 casts the intermediate+inftoINT64_MIN, aarch64 saturates toINT64_MAX) and by NumPy version (the scalar tie returns the first operand from 2.3.0 on, the second before).pandas-on-Spark returns one fixed value everywhere, so NumPy is only a valid reference on the platform it was verified against. The scheduled ARM and macOS jobs fail on
test_np_fmax_fmin,test_np_reciprocal_integer, and intermittentlytest_np_spark_compat_series.Does this PR introduce any user-facing change?
No. Tests only.
How was this patch tested?
Existing tests. On Linux x86-64 with NumPy 2.4.1 both tests still run and pass; with
platform.machine,platform.system, andnp.__version__patched to macOS arm64, Linux aarch64, and NumPy 1.23.2, they are skipped as intended.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5