Skip to content

BUG: to_numeric fails to convert a Pyarrow Decimal series containing NA values #61659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

chilin0525
Copy link
Contributor

Comment on lines +214 to +218
if is_numeric_dtype(values_dtype):
if is_series:
return arg._constructor(values, index=arg.index, name=arg.name)
else:
return values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you try to specify if not is_numeric_dtype here, and let the logic below try to convert this case anyways? This short circuit would skip some of the options set in to_numeric

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also as @mroeschke suggested in #61641 (comment) documentation update would also be welcome

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @simonjayhawkins , I already create anothter PR for document issue: #61904

@simonjayhawkins simonjayhawkins added Bug Dtype Conversions Unexpected or buggy dtype conversions Arrow pyarrow functionality labels Jun 25, 2025
@chilin0525 chilin0525 marked this pull request as draft July 2, 2025 15:03
Copy link
Member

@simonjayhawkins simonjayhawkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @chilin0525 for the PR

result = to_numeric(series, errors="coerce")

expected = Series([1.00, pd.NA], dtype=decimal_type)
tm.assert_series_equal(result, expected)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC we want a no-op here?

so does

Suggested change
tm.assert_series_equal(result, expected)
tm.assert_series_equal(result, series)

work?

def test_to_numeric_arrow_decimal_with_na():
# GH 61641
pa = pytest.importorskip("pyarrow")
decimal_type = ArrowDtype(pa.decimal128(3, scale=2))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could add the dtype as parameterization and also add any other EA dtypes that are expected to no-op?

so we could call the test something like.. test_to_numeric_EA_is_numeric?

Comment on lines +214 to +218
if is_numeric_dtype(values_dtype):
if is_series:
return arg._constructor(values, index=arg.index, name=arg.name)
else:
return values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also as @mroeschke suggested in #61641 (comment) documentation update would also be welcome

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Arrow pyarrow functionality Bug Dtype Conversions Unexpected or buggy dtype conversions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: to_numeric fails to convert a Pyarrow Decimal series containing NA values.
3 participants