DEPR: na_sentinel in factorize #47157

rhshadrach · 2022-05-28T05:08:07Z

closes Deprecate na_sentinel, add use_na_sentinel #46910 (Replace xxxx with the Github issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

The NotImplementedError will be removed as part of #46601

pep8speaks · 2022-05-28T05:08:15Z

Hello @rhshadrach! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-06-24 18:33:26 UTC

jbrockmendel · 2022-05-31T16:43:11Z

The warning handling in algorithms.factorize isn't my favorite, but i'll take your word for it that this is the best viable option.

jbrockmendel · 2022-05-31T16:43:38Z

pandas/core/common.py

+    use_na_sentinel: bool | lib.NoDefault,
+    warn: bool = True,
+) -> int | None:
+    """Determine value of na_sentinel for factorize methods.


nitpick newline before Determine

jbrockmendel · 2022-05-31T16:43:54Z

needs rebase, one nitpick, otherwise LGTM

… into depr_na_sentinel � Conflicts: � doc/source/whatsnew/v1.5.0.rst � pandas/core/algorithms.py � pandas/core/arrays/base.py � pandas/core/arrays/sparse/array.py � pandas/core/common.py � pandas/tests/extension/base/methods.py

…_na_sentinel � Conflicts: � doc/source/whatsnew/v1.5.0.rst

rhshadrach · 2022-05-31T21:00:08Z

The warning handling in algorithms.factorize isn't my favorite, but i'll take your word for it that this is the best viable option.

The other option is to use catch_warnings when the call to EA.factorize is made from within pd.factorize. I typically try to avoid using it, but perhaps this is a place where it's worth it. It would certainly simplify that logic (which I myself am not a fan of).

jorisvandenbossche · 2022-05-31T23:39:54Z

pandas/core/algorithms.py

+        if passed_na_sentinel is lib.no_default:
+            # User didn't specify na_sentinel; avoid warning. Note EA path always
+            # uses a na_sentinel value.
+            codes, uniques = values.factorize(use_na_sentinel=True)
+        elif passed_na_sentinel is None:
+            # Emit the appropriate warning message for None
+            _ = com.resolve_na_sentinel(passed_na_sentinel, use_na_sentinel)
+            codes, uniques = values.factorize(use_na_sentinel=True)
+        else:
+            # EA.factorize will warn
+            codes, uniques = values.factorize(na_sentinel=na_sentinel)


I think we need to distinguish deprecating the keyword that the user passes (I think that is handled above?) vs deprecating what the EA can accept.

Currently, if you have an external EA that has a custom factorize(self, na_sentinel=-1) method, then passing use_na_sentinel will raise an error. So to provide a smooth upgrade path for EA authors (and not users calling Series.factorize()), we might need to check here if the new keyword can be passed or not? (and if not, raising a deprecation warning urging the EA author to update their factorize() implementation)

i guess also worth clarifying is whether we expect the user to call EA.factorize directly vs pd.factorize

Generally we don't "expect" users to directly call EA.factorize, but it is a public method on a public object, so it falls under a deprecation policy (as is being done in this PR already). So I think for EA authors that have a custom EA.factorize, we should recommend doing a similar deprecation in their custom method?

Thanks @jorisvandenbossche. I think anyone using a mypy will be getting errors that their signature does not match the base class. I guess this not sufficient? It does seem noisy to add warnings for users, so I added the warning via __init_subclass__ so that it's emitted only once for each EA subclass. The potential downside is that it's emitted regardless of whether the method is used. Note users can silence warnings that are emitted when using pd.factorize even before the EA author adopts the new signature.

This also caught a defect in my previous implementation - pd.factorize was passing use_na_sentinel in some cases, which would cause code to fail if the EA author has not yet made the update. I went the catch_warnings route as I mentioned above.

Ah, I didn't notice you mentioned deprecation warning instead of future warning. That makes sense and will be less noisy.

rhshadrach · 2022-06-05T17:20:46Z

@jorisvandenbossche - friendly ping

jorisvandenbossche

For testing, I think it would be good if we have one of our test extension arrays (eg decimal) to still have the old signature, to ensure that we don't start to error for that

jorisvandenbossche · 2022-06-08T20:07:14Z

pandas/core/algorithms.py

+            The na_sentinel argument is deprecated and
+            will be removed in a future version of pandas. Specify use_na_sentinel as


Suggested change

The na_sentinel argument is deprecated and

will be removed in a future version of pandas. Specify use_na_sentinel as

The `na_sentinel` argument is deprecated and

will be removed in a future version of pandas. Specify `use_na_sentinel` as

I took this to mean use backticks for any argument / code in the warnings. I went and implemented that for all warnings.

jorisvandenbossche · 2022-06-09T05:57:53Z

pandas/core/algorithms.py

+        with warnings.catch_warnings():
+            # We've already warned above
+            warnings.filterwarnings("ignore", ".*use_na_sentinel.*", FutureWarning)
+            codes, uniques = values.factorize(na_sentinel=na_sentinel)


Should we do a similar "if use_na_sentinel in signature" check (as in init_subclass) and in that case already pass the new keyword to the underlying EA? (in that case we don't have to catch any warning)

It's unclear to me as we'd be trading out catch_warnings for inspect and more complex logic (though not by much). I thought inspect would take somewhat longer (on the order of 1ms) but on my machine it's pretty quick:

values = pd.array([1, 2, 3]) print("use_na_sentinel" in inspect.signature(values.factorize).parameters) %timeit "use_na_sentinel" in inspect.signature(values.factorize).parameters # True # 12.8 µs ± 2.97 µs per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

I've gone an implemented it; easy to revert if we don't like it.

…_na_sentinel � Conflicts: � doc/source/whatsnew/v1.5.0.rst

rhshadrach · 2022-06-10T21:15:38Z

Thanks @jorisvandenbossche - changes made and ready for another review.

For testing, I think it would be good if we have one of our test extension arrays (eg decimal) to still have the old signature, to ensure that we don't start to error for that

I don't think there is decimal, so I did this for datetimelike. It has the easiest implementation to change and doesn't generate warnings on its own (it uses super to do that), so seemed like the best option. Plus it also starts with a 'd' :D

rhshadrach · 2022-06-10T21:35:10Z

Encountered a minor difficulty with the warnings and leaving TimelikeOps with the old signature. In order to not get the warning on import of pandas, need to special case it in __init_subclass__. However, as this "method" is run at "import time", we can't even import DatetimeArray et al (gives circular imports) to test whether cls == DatetimeArray. The best I think we can do is check it's name, which is not exactly ideal.

With the current implementation, worst case is some code does not get warned to update the signature for classes named "TimelikeOps", "DatetimeArray", "TimedeltaArray".

…_na_sentinel � Conflicts: � doc/source/whatsnew/v1.5.0.rst

jreback · 2022-06-11T15:22:55Z

pandas/core/algorithms.py

+    elif not isinstance(values.dtype, np.dtype):
+        if (
+            na_sentinel == -1
+            and "use_na_sentinel" in inspect.signature(values.factorize).parameters


why are you inspecting like this? its always passed (with no_default maybe)

We're inspecting whether the EA values can accept the new argument "use_na_sentinel". #47157 (comment)

Might be good to add a comment to that effect

jreback · 2022-06-11T15:24:16Z

pandas/core/common.py

@@ -700,3 +700,53 @@ def deprecate_numeric_only_default(cls: type, name: str, deprecate_none: bool =
    )

    warnings.warn(msg, FutureWarning, stacklevel=find_stack_level())
+
+
+def resolve_na_sentinel(


can you put this somewhere else, closer to where its used

Moved to algorithms.

…_na_sentinel

rhshadrach · 2022-06-15T00:16:32Z

@jreback @jorisvandenbossche friendly ping.

…_na_sentinel

pandas/core/indexes/range.py

pandas/tests/test_algos.py

mroeschke

IMO looks pretty good

…_na_sentinel � Conflicts: � pandas/core/arrays/arrow/array.py

jorisvandenbossche · 2022-06-24T20:11:08Z

@rhshadrach thanks for the updates! Looks good to me from a quick look. I won't be able to further look at it the coming two weeks, so don't wait on me to move forward here

mroeschke · 2022-06-24T21:31:44Z

Thanks @rhshadrach. Follow ups can be made as needed

rhshadrach · 2022-06-24T21:48:53Z

@rhshadrach thanks for the updates! Looks good to me from a quick look. I won't be able to further look at it the coming two weeks, so don't wait on me to move forward here

Thanks @jorisvandenbossche; if you do end up circling back, let me know if you have any more comments and happy to do a followup.

* DEPR: na_sentinel in factorize * WIP * DEPR: na_sentinel in factorize * Fixups * Fixups * black * fixup * docs * newline * Warn on class construction, rework pd.factorize warnings * FutureWarning -> DeprecationWarning * Remove old comment * backticks in warnings, revert datetimelike, avoid catch_warnings * fixup for warnings * mypy fixups * Move resolve_na_sentinel * Remove underscores Co-authored-by: Jeff Reback <jeff@reback.net>

rhshadrach added 5 commits May 27, 2022 14:44

DEPR: na_sentinel in factorize

552775f

WIP

79231e7

DEPR: na_sentinel in factorize

c822282

Fixups

05fa0ca

Fixups

f626dd8

rhshadrach added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Deprecate Functionality to remove in pandas labels May 28, 2022

rhshadrach added this to the 1.5 milestone May 28, 2022

rhshadrach added 3 commits May 28, 2022 01:10

black

a15e43a

fixup

9a33637

docs

46e7a8d

rhshadrach requested review from jorisvandenbossche and jbrockmendel May 30, 2022 12:06

jbrockmendel reviewed May 31, 2022

View reviewed changes

rhshadrach added 2 commits May 31, 2022 16:49

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

c8d6fa2

…_na_sentinel � Conflicts: � doc/source/whatsnew/v1.5.0.rst

newline

0fd1ea7

jorisvandenbossche reviewed May 31, 2022

View reviewed changes

rhshadrach added 3 commits May 31, 2022 21:14

Warn on class construction, rework pd.factorize warnings

465ab2b

FutureWarning -> DeprecationWarning

6b4917c

Remove old comment

d8e3d6b

rhshadrach requested a review from jorisvandenbossche June 2, 2022 22:17

jorisvandenbossche reviewed Jun 9, 2022

View reviewed changes

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

0111eef

…_na_sentinel � Conflicts: � doc/source/whatsnew/v1.5.0.rst

backticks in warnings, revert datetimelike, avoid catch_warnings

39b3747

fixup for warnings

5842053

rhshadrach added 2 commits June 11, 2022 09:01

mypy fixups

945bb04

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

8f8cb18

…_na_sentinel � Conflicts: � doc/source/whatsnew/v1.5.0.rst

jreback mentioned this pull request Jun 11, 2022

DEPR: log of deprecations in 1.x (to be removed in 2.0) #30228

Closed

Merge branch 'main' into depr_na_sentinel

9630edc

jreback requested changes Jun 11, 2022

View reviewed changes

rhshadrach added 2 commits June 12, 2022 08:26

Move resolve_na_sentinel

5524d53

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

fcd8a75

…_na_sentinel

rhshadrach added 2 commits June 19, 2022 12:18

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

c615de4

…_na_sentinel

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

80aaf5e

…_na_sentinel

mroeschke reviewed Jun 23, 2022

View reviewed changes

pandas/core/indexes/range.py Outdated Show resolved Hide resolved

mroeschke reviewed Jun 23, 2022

View reviewed changes

pandas/tests/test_algos.py Outdated Show resolved Hide resolved

mroeschke reviewed Jun 23, 2022

View reviewed changes

rhshadrach added 2 commits June 24, 2022 14:31

Remove underscores

dc97a7b

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

eb6d5a1

…_na_sentinel � Conflicts: � pandas/core/arrays/arrow/array.py

mroeschke approved these changes Jun 24, 2022

View reviewed changes

jreback approved these changes Jun 24, 2022

View reviewed changes

mroeschke merged commit d580826 into pandas-dev:main Jun 24, 2022

rhshadrach deleted the depr_na_sentinel branch June 24, 2022 21:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPR: na_sentinel in factorize #47157

DEPR: na_sentinel in factorize #47157

rhshadrach commented May 28, 2022

pep8speaks commented May 28, 2022 •

edited

jbrockmendel commented May 31, 2022

jbrockmendel May 31, 2022

jbrockmendel commented May 31, 2022

rhshadrach commented May 31, 2022

jorisvandenbossche May 31, 2022

jbrockmendel May 31, 2022

jorisvandenbossche May 31, 2022

rhshadrach Jun 1, 2022

rhshadrach Jun 1, 2022 •

edited

rhshadrach commented Jun 5, 2022

jorisvandenbossche left a comment

jorisvandenbossche Jun 8, 2022

rhshadrach Jun 10, 2022

jorisvandenbossche Jun 9, 2022

rhshadrach Jun 10, 2022

rhshadrach commented Jun 10, 2022

rhshadrach commented Jun 10, 2022 •

edited

jreback Jun 11, 2022

rhshadrach Jun 12, 2022

mroeschke Jun 23, 2022

jreback Jun 11, 2022

rhshadrach Jun 12, 2022

rhshadrach commented Jun 15, 2022

mroeschke left a comment

jorisvandenbossche commented Jun 24, 2022

mroeschke commented Jun 24, 2022

rhshadrach commented Jun 24, 2022

		The na_sentinel argument is deprecated and
		will be removed in a future version of pandas. Specify use_na_sentinel as

DEPR: na_sentinel in factorize #47157

DEPR: na_sentinel in factorize #47157

Conversation

rhshadrach commented May 28, 2022

pep8speaks commented May 28, 2022 • edited

Comment last updated at 2022-06-24 18:33:26 UTC

jbrockmendel commented May 31, 2022

Choose a reason for hiding this comment

jbrockmendel commented May 31, 2022

rhshadrach commented May 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhshadrach Jun 1, 2022 • edited

Choose a reason for hiding this comment

rhshadrach commented Jun 5, 2022

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhshadrach commented Jun 10, 2022

rhshadrach commented Jun 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhshadrach commented Jun 15, 2022

mroeschke left a comment

Choose a reason for hiding this comment

jorisvandenbossche commented Jun 24, 2022

mroeschke commented Jun 24, 2022

rhshadrach commented Jun 24, 2022

pep8speaks commented May 28, 2022 •

edited

rhshadrach Jun 1, 2022 •

edited

rhshadrach commented Jun 10, 2022 •

edited