Skip to content

BUG: .map & .apply would convert element type for extension array. #60766

Open
@colinfang

Description

@colinfang

Pandas version checks

  • I have checked that this issue has not already been reported.

    I have confirmed this bug exists on the latest version of pandas.

    I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

s = pd.Series([1, 2, None], dtype="Int32")
s.apply(lambda x: print(x))
s.map(lambda x: print(x))
# print 1.0 2.0 nan instead of 1 2 <NA>

Issue Description

s has dtype Int32Dtype which can encode nullable integers

However, when we use .apply(f) or .map(f), the element passed to f becomes float, and pd.NA becomes np.nan

Expected Behavior

f should receive unconverted value, as if we are doing f(s.iloc[0]), f(s.iloc[1]), ...

Installed Versions

tested with Pandas 2.2.3

Activity

added
Needs TriageIssue that has not been reviewed by a pandas team member
on Jan 22, 2025
rhshadrach

rhshadrach commented on Jan 22, 2025

@rhshadrach
Member

Thanks for the report! This is due to #55058, but I don't see much discussion there nor issues that change closes. It does seem surprising to me that one gets np.nan here. If we remove the method on BaseMaskedArray entirely, I see expected behavior (at least in this case). That includes the use of apply (which uses map internally).

cc @WillAyd @mroeschke @phofl

added
ExtensionArrayExtending pandas with custom dtypes or arrays.
NA - MaskedArraysRelated to pd.NA and nullable extension arrays
Needs DiscussionRequires discussion from core team before further action
ApplyApply, Aggregate, Transform, Map
and removed
ExtensionArrayExtending pandas with custom dtypes or arrays.
Needs TriageIssue that has not been reviewed by a pandas team member
on Jan 22, 2025
WillAyd

WillAyd commented on Jan 22, 2025

@WillAyd
Member

Agreed this looks strange - the function should be receiving the arguments as integers / pd.NA

removed
Needs DiscussionRequires discussion from core team before further action
on Jan 22, 2025
KevsterAmp

KevsterAmp commented on Mar 13, 2025

@KevsterAmp
Contributor

take

KevsterAmp

KevsterAmp commented on Mar 26, 2025

@KevsterAmp
Contributor

@rhshadrach - What method are you referring on BaseMaskedArray to be removed? Thanks

rhshadrach

rhshadrach commented on Mar 26, 2025

@rhshadrach
Member

map

added 2 commits that reference this issue on Mar 28, 2025

Fix pandas-dev#60766: .map & .apply would convert element type for ex…

be46492

Fix pandas-dev#60766:.map,.apply would convert element type for exten…

1c55a7f
added a commit that references this issue on Apr 22, 2025

Fix pandas-dev#60766:.map,.apply would convert element type for exten…

f8153f3
pedromfdiogo

pedromfdiogo commented on Apr 23, 2025

@pedromfdiogo
Contributor

@KevsterAmp what do you think about my commit? Can i do the PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ApplyApply, Aggregate, Transform, MapBugNA - MaskedArraysRelated to pd.NA and nullable extension arrays

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Participants

    @WillAyd@colinfang@rhshadrach@KevsterAmp@pedromfdiogo

    Issue actions

      BUG: `.map` & `.apply` would convert element type for extension array. · Issue #60766 · pandas-dev/pandas