ENH:

[GoHuskers]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

I wish that I could use the duplicated() function on data which has na values

Feature Description

Add a new parameter to the duplicated function:

keepna = {'False, True}, default False

when keepna = False, current functionality works as is.
if keepna = True, the duplicated function would find the duplicate rows but including all rows with na values on any column within the specificed subset

Alternative Solutions

current alternative solution which I have found is to :

1. overwrite na values with some particular constant value;
2. execute the duplicated function; and then,
3. change the all of the constant values from step 1. above back to nan.

Additional Context

No response

[phofl]

Hi,

can you please provide a reproducible example?

[rhshadrach]

if keepna = True, the duplicated function would find the duplicate rows but including all rows with na values on any column within the specificed subset

I think you're asking for something like:

result = df.duplicated(subset=my_subset) | df[my_subset].isna().any(axis=1)

but as @phofl requested, please provide an example.

[mroeschke]

Closing as there not enough info here to be actionable