Skip to content

[SPARK-56014][PS][TESTS] Fix to_numeric ignore test for pandas 3.0 #54836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ueshin wants to merge 1 commit into from
Closed

[SPARK-56014][PS][TESTS] Fix to_numeric ignore test for pandas 3.0 #54836

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion python/pyspark/pandas/tests/test_namespace.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import pandas as pd
import numpy as np

from pyspark.loose_version import LooseVersion
from pyspark import pandas as ps
from pyspark.pandas.exceptions import PandasNotImplementedError
from pyspark.pandas.namespace import _get_index_map, read_delta
Expand Down Expand Up @@ -579,7 +580,13 @@ def test_to_numeric(self):
# "coerce", "ignore" and "raise" with non-Series.
data = ["1", "2", None, "4", "hello"]
self.assert_eq(pd.to_numeric(data, errors="coerce"), ps.to_numeric(data, errors="coerce"))
self.assert_eq(pd.to_numeric(data, errors="ignore"), ps.to_numeric(data, errors="ignore"))
if LooseVersion(pd.__version__) < "3.0.0":
self.assert_eq(
pd.to_numeric(data, errors="ignore"), ps.to_numeric(data, errors="ignore")
)
else:
with self.assertRaisesRegex(ValueError, "invalid error value specified"):
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the default value was raise, this looks okay to me.

BTW, do you think we need to re-fresh the doc, @ueshin ? The docs says ignore doesn't work for pandas-on-Spark Series args.

spark/python/pyspark/pandas/namespace.py

Lines 3595 to 3608 in 7eef6f7

def to_numeric(arg, errors="raise"):
"""
Convert argument to a numeric type.
Parameters
----------
arg : scalar, list, tuple, 1-d array, or Series
Argument to be converted.
errors : {'raise', 'coerce'}, default 'raise'
* If 'coerce', then invalid parsing will be set as NaN.
* If 'raise', then invalid parsing will raise an exception.
* If 'ignore', then invalid parsing will return the input.
.. note:: 'ignore' doesn't work yet when `arg` is pandas-on-Spark Series.

Copy link
Copy Markdown
Member Author

@ueshin ueshin Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, eventually we should update the docs, but so far pandas 3 is not fully supported and I'm still not sure whether we can make it by 4.2.0 release, so I think we should keep it as-is for now.

Also I see another PR to show warning with pandas 3.

Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it~

ps.to_numeric(data, errors="ignore")

self.assertRaisesRegex(
ValueError,
Expand Down