ENH: Compatibility between `isin()` methods and python-holidays

[alexmalins]

Is your feature request related to a problem?

Pandas DataFrame & Series isin() methods are currently incompatible with python-holidays objects. Holidays objects are dict-like objects for determining whether a specific date is a holiday or not. There are two issues I've found:

• pd.DataFrame.isin() overwrites dict-like objects with collections.defaultdict(), which disregards the designed behaviour of some objects that inherit from dict such as holidays objects.

• pd.Series.isin() ends up casting holidays-like objects to a list. Holidays objects can dynamically build holidays on the fly, but the type casting makes this fail.

See code examples at bottom showing the failures.

Describe the solution you'd like

Ideally the isin() methods would support holidays objects. This means pd.DataFrame.isin() should treat holidays objects like as if a list (iterable) had been passed because the keys of holidays objects store the actual holiday dates. This is different from the current behaviour in treating it like a normal dict and trying to match key:value combinations to column:field value in the DataFrame.

The pd.Series.isin() method should support the ability of holidays objects to dynamically check if a date is a holiday on the fly.

API breaking implications

If the solution adds isinstance(values, holidayobject) checks to the isin() methods, I guess this won't change the existing API for non-holidays objects. However this might require holidays becomes a new dependency of pandas... which is maybe a problem?!

Describe alternatives you've considered

A hacky fix is to always pre-populate the list of holidays into holiday objects via the years=... parameter in their constructor, then supply list(holiday_object) as the parameter to the DataFrame.isin() method. This makes things work.

However it is not at all obvious for users that this is necessary, and on-the-fly checking of holiday dates won't work this way, and more importantly, if users don't realize this pandas will silently give incorrect results if a holidays object is supplied to isin() directly.

Additional context

Incorrect behaviour:

>>> import holidays
>>> import pandas as pd

# Series.isin() example failure - on-the-fly checking if a date is a holiday won't work
>>> s = pd.Series([pd.Timestamp("2022-12-25")])
>>> uk = holidays.UK()
>>> s.isin(uk)
0    False
dtype: bool

# DataFrame.isin() always fails because it treats the holidays object like a dict
# when it should be treated like a list (iterable)
>>> uk = holidays.UK(years=2022)
>>> df = pd.DataFrame([pd.Timestamp("2022-12-25")])
>>> df.isin(uk)
       0
0  False

Hacky fixes:

>>> uk = holidays.UK(years=2022)
>>> s.isin(uk)
0    True
dtype: bool

>>> df.isin(list(uk))
       0
0  True

[alexmalins]

cf. Issue #45310 discussing whether to defer holiday implementations away from pandas

[jreback]

pandas cannot depend on pandas-holidays

in fact the reverse is true

so you would need to fix something in that package

-1 on any change here - what you are proposing is to make an already complex method even more complex

[alexmalins]

@jreback thanks for looking at this

pandas cannot depend on pandas-holidays

in fact the reverse is true

Did you mean python-holidays here? python-holidays has no dependency on pandas.

Fixing something in python-holidays would mean a major change for that project as it would involve changing the super class of its fundamental objects away from dict.

But I understand where you are coming from. Maybe this is an issue is that is better to raise awareness of and describe a solution (passing only lists of pre-calculated holidays to isin()) than to try and fix with more code...

Edit: the only other way I could think of for fixing this via pandas without adding a new dependency would be if pandas just attempted value in values statements. But this would also have its own drawbacks I guess...

[dr-prodigy]

@jreback although already closed, I jump into this conversation because @alexmalins came to report this issue on python-holidays before coming to here, so I know what's behind it.
I find the explanation above a bit misleading / "messy", so I'll try to clarify.

1. Pandas holidays is not in scope here
2. Not even the named python-holidays has any dependency neither on pandas nor on pandas holidays, although the issue raises while using it with pandas
3. No package dependency review is required

To get more in deep, the implementation made in pandas here:
https://github.com/pandas-dev/pandas/blob/main/pandas/core/algorithms.py#L482
is problematic, in that isin() method internally overwrites one of the input parameters sent to it (values), thus basically destroying the relevant object reference for its caller too.
Consequently, this pandas' code could provide issues to any software using extended array/dict classes: named python-holidays library (based in fact on dict-inherited classes) is just an example of this.

At first sight, this isin() misbehaving logic seems not required, in that the method is meant to return just a boolean value, and therefore shouldn't require any changes to the provided input parameters.

Conclusion: reviewing isin() code to use local copies of its input variables only should be an easy (and correct) solution to sort this issue out.

[jreback]

@dr-prodigy sure if we are actually modifying the inputs then consider that a bug

can u demonstrate a test that shows this? i don't believe by code inspection that this is true but it's possible we hit a path where this happens

[dr-prodigy]

hey @jreback thx!
I am sorry, I made some further tests and I have to amend what previously stated (I'm no pandas user so I've been misleaded as well :-) ): no input modification seems to occur in fact, so as of now the reported issue is not to be considered on pandas' side.
I will analyze this further: if sth comes out on your side I'll surely come back here.
Sorry again for the inconvenience, and thank you for your support 👍

[alexmalins]

Yes, isin() is not modifying or changing the input object it receives - sorry for the confusion. It is just that under the hood isin() doesn't behave like element in original_argument, because original_argument gets copied locally and the local copy undergoes a type change.

In the python-holidays case that change causes the loss of dynamic ability to calculate holiday dates on the fly. So I am not sure anything can be done on the python-holidays side to get it to work. The change happens before the in / __contains__ operation.

Sorry also for the confusion about pandas-holidays. I just linked issue #45310 because python-holidays was suggested as an alternative and to document this incompatibility.

Backtracking, these are other workarounds I thought of for the examples. They do not require pre-initialization of the holiday object:

>>> uk = holidays.UK()
>>> s = pd.Series([pd.Timestamp("2022-12-25")])
>>> s.apply(lambda x: x in uk)
0    True
dtype: bool
>>> uk = holidays.UK()
>>> df = pd.DataFrame([pd.Timestamp("2022-12-25")])
>>> df.applymap(lambda x: x in uk)
      0
0  True