BUG: `Series.map` passes `DatetimeIndex` as first argument on some series #44392

pierremonico · 2021-11-11T13:31:35Z

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

from datetime import datetime
from random import randint

import pandas as pd


def mapping_func(ts):
    assert type(ts) == pd.Timestamp


# --- works as expected ---

ts_list = [
    datetime(2021, randint(1, 12), randint(1, 28))
    for _ in range(5)
]

df01 = pd.DataFrame(ts_list, columns=["timestamps"])
df01["timestamps"].map(mapping_func)


# --- doesn't work as expected ---


ts_list_02 = [
    '2020-09-01T15:47:33.953000+00:00',
    '2020-09-02T16:57:16.547000+00:00',
    '2020-09-02T16:57:16.887000+00:00',
    '2020-09-02T16:57:12.377000+00:00',
    '2020-09-02T16:57:12.667000+00:00',
]

df02 = pd.DataFrame(ts_list_02, columns=["timestamps"])
df02["timestamps"] = pd.to_datetime(df02["timestamps"])
df02["timestamps"].map(mapping_func)


# --- Type checks ---

assert isinstance(df01["timestamps"], pd.Series)
assert isinstance(df02["timestamps"], pd.Series)

assert df01["timestamps"].dtype == df02["timestamps"].dtype
# fails because (respectively) dtype('<M8[ns]') != datetime64[ns, UTC]

Issue Description

I am having a weird issued when using Series.map(arg=fn) on a series containing Timestamps:

In some cases the first time fn is called by map, the argument is a DatetimeIndex containing all timestamps in the series. Further calls pass every individual member of the series to fn as expected.
In other cases it passes every individual member of the series to fn as expected.

I was able to reproduce it by:

creating one series based on a list of python datetimes instances (df01 in the example - works as expected).
creating one series based on a list of ISO timestamps and casting the series to Timestamp using pd.to_datetime (df02 in the example - doesn't work as expected aka. passes a full DatetimeIndex the first time it calls fn).

Expected Behavior

def mapping_func(ts):
    assert type(ts) == pd.Timestamp

ts_list = [
    datetime(2021, randint(1, 12), randint(1, 28))
    for _ in range(5)
]

df01 = pd.DataFrame(ts_list, columns=["timestamps"])
df01["timestamps"].map(mapping_func)

Works as expected, the ts parameter of mapping_func always being a Timestamp instance.

def mapping_func(ts):
    assert type(ts) == pd.Timestamp

ts_list_02 = [
    '2020-09-01T15:47:33.953000+00:00',
    '2020-09-02T16:57:16.547000+00:00',
    '2020-09-02T16:57:16.887000+00:00',
    '2020-09-02T16:57:12.377000+00:00',
    '2020-09-02T16:57:12.667000+00:00',
]

df02 = pd.DataFrame(ts_list_02, columns=["timestamps"])
df02["timestamps"] = pd.to_datetime(df02["timestamps"])
df02["timestamps"].map(mapping_func)

Should not raise an error but actually does, since the first time mapping_func is called, the ts parameter is passed a DatetimeIndex.

Installed Versions

INSTALLED VERSIONS

commit : 945c9ed
python : 3.7.9.final.0
python-bits : 64
OS : Darwin
OS-release : 20.5.0
Version : Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : None.UTF-8

pandas : 1.3.4
numpy : 1.21.4
pytz : 2021.3
dateutil : 2.8.2
pip : 21.1.3
setuptools : 57.4.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.2
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.1 (dt dec pq3 ext lo64)
jinja2 : 3.0.3
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.11.0
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.4.26
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
numba : None

The text was updated successfully, but these errors were encountered:

mroeschke · 2021-11-14T05:04:24Z

Hmm I don't see an error on master and don't really follow how map applies?

In [2]: from datetime import datetime
   ...: from random import randint
   ...:
   ...: import pandas as pd

In [3]: def mapping_func(ts):
   ...:     assert type(ts) == pd.Timestamp
   ...:
   ...: ts_list = [
   ...:     datetime(2021, randint(1, 12), randint(1, 28))
   ...:     for _ in range(5)
   ...: ]
   ...:
   ...: df01 = pd.DataFrame(ts_list, columns=["timestamps"])
   ...: df01["timestamps"].map(mapping_func)
Out[3]:
0    None
1    None
2    None
3    None
4    None
Name: timestamps, dtype: object

In [4]: def mapping_func(ts):
   ...:     assert type(ts) == pd.Timestamp
   ...:
   ...: ts_list_02 = [
   ...:     '2020-09-01T15:47:33.953000+00:00',
   ...:     '2020-09-02T16:57:16.547000+00:00',
   ...:     '2020-09-02T16:57:16.887000+00:00',
   ...:     '2020-09-02T16:57:12.377000+00:00',
   ...:     '2020-09-02T16:57:12.667000+00:00',
   ...: ]
   ...:
   ...: df02 = pd.DataFrame(ts_list_02, columns=["timestamps"])
   ...: df02["timestamps"] = pd.to_datetime(df02["timestamps"])
   ...: df02["timestamps"].map(mapping_func)
Out[4]:
0    None
1    None
2    None
3    None
4    None
Name: timestamps, dtype: object

pierremonico · 2021-11-15T13:32:14Z

def mapping_func(ts):
    print(ts)

ts_list = [
    datetime(2021, randint(1, 12), randint(1, 28))
    for _ in range(5)
]

df01 = pd.DataFrame(ts_list, columns=["timestamps"])
df01["timestamps"].map(mapping_func)

Output:

2021-12-03 00:00:00
2021-06-23 00:00:00
2021-10-13 00:00:00
2021-07-06 00:00:00
2021-04-03 00:00:00

ts_list_02 = [
    '2020-09-01T15:47:33.953000+00:00',
    '2020-09-02T16:57:16.547000+00:00',
    '2020-09-02T16:57:16.887000+00:00',
    '2020-09-02T16:57:12.377000+00:00',
    '2020-09-02T16:57:12.667000+00:00',
]

df02 = pd.DataFrame(ts_list_02, columns=["timestamps"])
df02["timestamps"] = pd.to_datetime(df02["timestamps"])
df02["timestamps"].map(mapping_func)

Output:

DatetimeIndex(['2020-09-01 15:47:33.953000+00:00',
               '2020-09-02 16:57:16.547000+00:00',
               '2020-09-02 16:57:16.887000+00:00',
               '2020-09-02 16:57:12.377000+00:00',
               '2020-09-02 16:57:12.667000+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)
2020-09-01 15:47:33.953000+00:00
2020-09-02 16:57:16.547000+00:00
2020-09-02 16:57:16.887000+00:00
2020-09-02 16:57:12.377000+00:00
2020-09-02 16:57:12.667000+00:00

As you see in the second example, before running the actual five iterations, mapping_func is called once with the whole DatetimeIndex as an argument.

pierremonico · 2021-11-15T13:40:18Z

I also reproduced it in this notebook in case it helps.

mroeschke · 2021-11-15T20:28:57Z

I see. Thanks, the print call reveals the issue a lot clearer.

Mikhaylov-yv · 2021-11-18T16:31:03Z

Confirm that this issue is present on the master branch.

Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'1.4.0.dev0+1143.gbe1ef62bec'
>>> def mapping_func(ts):
...     print(ts)
... 
>>> ts_list_02 = [
...     '2020-09-01T15:47:33.953000+00:00',
...     '2020-09-02T16:57:16.547000+00:00',
...     '2020-09-02T16:57:16.887000+00:00',
...     '2020-09-02T16:57:12.377000+00:00',
...     '2020-09-02T16:57:12.667000+00:00',
... ]
>>> 
>>> df02 = pd.DataFrame(ts_list_02, columns=["timestamps"])
>>> df02["timestamps"] = pd.to_datetime(df02["timestamps"])
>>> df02["timestamps"].map(mapping_func)
DatetimeIndex(['2020-09-01 15:47:33.953000+00:00',
               '2020-09-02 16:57:16.547000+00:00',
               '2020-09-02 16:57:16.887000+00:00',
               '2020-09-02 16:57:12.377000+00:00',
               '2020-09-02 16:57:12.667000+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)
2020-09-01 15:47:33.953000+00:00
2020-09-02 16:57:16.547000+00:00
2020-09-02 16:57:16.887000+00:00
2020-09-02 16:57:12.377000+00:00
2020-09-02 16:57:12.667000+00:00
0    None
1    None
2    None
3    None
4    None
Name: timestamps, dtype: object

I'll try to take this problem. I'll start by creating a test that will show this.

Mikhaylov-yv · 2021-11-18T22:42:41Z

I'm trying to catch the exception with this test:

def test_series_datetime_index():
    tm_arr = [f"2021-01-0{i} 12:24:34.024322+00:00" for i in range(1, 5)]
    ser = Series(to_datetime(tm_arr))

    def f(x):
        assert type(x).__name__ == "Timestamp"

    ser.map(f)
    ser.apply(f)

The test showed nothing. In this case, the function returns nothing. The error is only visible when using print ()

pierremonico · 2021-11-19T09:40:53Z

Very weird, because I can't reproduce it either when trying in the notebook I sent above; but I can reproduce it locally on my machine with 1.3.4 (an exception is raised for assert type(ts) == pd.Timestamp) since the first argument passed is a DatetimeIndex instance.

mzeitlin11 · 2021-11-21T02:39:18Z

Maybe related to #43940?

pierremonico · 2021-11-22T12:26:17Z

Maybe related to #43940?

Absolutely, thanks for pointing out!
I tried it out by just running the script in the console instead of debugging in VS Code, and the assert statement doesn't raise an error. The print statement however will always print a DatetimeIndex.

To sum it up:

print in the mapping function will always print a DatetimeIndex before running the iterations over the Series, no matter if the script is run with debugging or simply through the console.
when running in debug, it will actually pass a DatetimeIndex as a first argument before running the iterations.

def mapping_func(ts):
    print(ts)
    assert isinstance(ts, pd.Timestamp)

ts_list_02 = [
    '2020-09-01T15:47:33.953000+00:00',
    '2020-09-02T16:57:16.547000+00:00',
    '2020-09-02T16:57:16.887000+00:00',
    '2020-09-02T16:57:12.377000+00:00',
    '2020-09-02T16:57:12.667000+00:00',
]

df02 = pd.DataFrame(ts_list_02, columns=["timestamps"])
df02["timestamps"] = pd.to_datetime(df02["timestamps"])
df02["timestamps"].map(mapping_func)

Output of debug:

DatetimeIndex(['2020-09-01 15:47:33.953000+00:00',
               '2020-09-02 16:57:16.547000+00:00',
               '2020-09-02 16:57:16.887000+00:00',
               '2020-09-02 16:57:12.377000+00:00',
               '2020-09-02 16:57:12.667000+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)

and raises an error from the assert.

Output of running the script from console:

DatetimeIndex(['2020-09-01 15:47:33.953000+00:00',               '2020-09-02 16:57:16.547000+00:00',               '2020-09-02 16:57:16.887000+00:00',
               '2020-09-02 16:57:12.377000+00:00',
               '2020-09-02 16:57:12.667000+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)
2020-09-01 15:47:33.953000+00:00
2020-09-02 16:57:16.547000+00:00
2020-09-02 16:57:16.887000+00:00
2020-09-02 16:57:12.377000+00:00
2020-09-02 16:57:12.667000+00:00

with no error raised.

pierremonico added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 11, 2021

mroeschke added Needs Info Clarification about behavior needed to assess issue and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 14, 2021

mroeschke added Series Series data structure and removed Needs Info Clarification about behavior needed to assess issue labels Nov 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: `Series.map` passes `DatetimeIndex` as first argument on some series #44392

BUG: `Series.map` passes `DatetimeIndex` as first argument on some series #44392

pierremonico commented Nov 11, 2021 •

edited

INSTALLED VERSIONS

mroeschke commented Nov 14, 2021 •

edited

pierremonico commented Nov 15, 2021

pierremonico commented Nov 15, 2021

mroeschke commented Nov 15, 2021

Mikhaylov-yv commented Nov 18, 2021

Mikhaylov-yv commented Nov 18, 2021

pierremonico commented Nov 19, 2021

mzeitlin11 commented Nov 21, 2021

pierremonico commented Nov 22, 2021

BUG: Series.map passes DatetimeIndex as first argument on some series #44392

BUG: Series.map passes DatetimeIndex as first argument on some series #44392

Comments

pierremonico commented Nov 11, 2021 • edited

Reproducible Example

Issue Description

Expected Behavior

Installed Versions

INSTALLED VERSIONS

mroeschke commented Nov 14, 2021 • edited

pierremonico commented Nov 15, 2021

pierremonico commented Nov 15, 2021

mroeschke commented Nov 15, 2021

Mikhaylov-yv commented Nov 18, 2021

Mikhaylov-yv commented Nov 18, 2021

pierremonico commented Nov 19, 2021

mzeitlin11 commented Nov 21, 2021

pierremonico commented Nov 22, 2021

BUG: `Series.map` passes `DatetimeIndex` as first argument on some series #44392

BUG: `Series.map` passes `DatetimeIndex` as first argument on some series #44392

pierremonico commented Nov 11, 2021 •

edited

mroeschke commented Nov 14, 2021 •

edited