BUG: able to create nonexistent Timestamp

[AlexKirko]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd
ts = pd.Timestamp(year=2001, month=10, day=14, hour=0, minute=30, tz='America/Santiago')
print(ts)

Problem description

There was a DST transition on the 14th of October 2001, and the time from midnight to 1 a.m. does not exist. Ran into this while working on #40517 where we ran into an error while normalizing a Timestamp when midnight does not exist in a particular timezone. Yet, we are able to create this timestamp directly, even though it does not make any sense.

Expected Output

I would expect NonExistentTimeError: 2001-10-14 00:30:00

Notes

I wasn't able to find a prior issue for this. I'm a bit swamped right now, so I would appreciate if someone else looked at this bug. If no one takes it up, I'll investigate next week.

Output of pd.show_versions()

This is on my dev environment, but I'm getting the same behavior on the latest stable version.

INSTALLED VERSIONS

commit : 692bba5
python : 3.8.8.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1251

pandas : 0.26.0.dev0+6766.g692bba578e
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 49.6.0.post20210108
Cython : 0.29.22
pytest : 6.2.2
hypothesis : 6.3.4
sphinx : 3.5.1
blosc : None
feather : None
xlsxwriter : 1.3.7
lxml.etree : 4.6.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.21.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.8.7
fastparquet : 0.5.0
gcsfs : 0.7.2
matplotlib : 3.3.4
numexpr : 2.7.2
odfpy : None
openpyxl : 3.0.6
pandas_gbq : None
pyarrow : 3.0.0
pyxlsb : None
s3fs : 0.5.2
scipy : 1.6.0
sqlalchemy : 1.3.23
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.17.0
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.52.0

[mroeschke]

Might be the same core issue as discovered in #40915 (comment)

Generally. I think the tz argument in Timestamp (and DatetimeIndex, date_range) should be deprecated in favor of explicitly calling tz_localize and tz_convert

[rauniyars]

Take - Working with my team to contribute to this issue.

[AlexKirko]

@rauniyars Thanks for taking this up! If you get stuck, don't hesitate to ask.

[hewittk]

@AlexKirko do you know what method(s) for creating the time stamp our team should modify to add this error? We have found the timestamp class and its cpython derivatives but we're still not sure where exactly to intervene.

[AlexKirko]

@hewittk The way I would handle this would be to first check @mroeschke 's idea and look at #40915 . If he's right, then this is caused specifically by a tz_localize shift issue issue. Otherwise, just go through the Timestamp.__new__ for ts = pd.Timestamp(year=2001, month=10, day=14, hour=0, minute=30, tz='America/Santiago'), add a bunch of print everywhere and see how we end up with with a nonexistant time.

The full Tiimestamp creation code is a bit lengthy, I know, but the logic is traceable for a particular example as long as you are willing to give it time (and Cython recompiling attempts).

If this doesn't help, and you can't pinpoint the specific issue, please ping me, and I'll try to trace the problem and narrow it down for you.

[hewittk]

@AlexKirko we have been looking through the timestamps and timezones classes for how the timezone validation of a new Timestamp is done, and haven't been able to pinpoint the specific issue so far. It seems like this issue isn't within tz_localize because the documentation of tz_localize has said that it is only used either when a Timestamp is being converted into another timezone or a timezone is being added to a naive Timestamp, and because the method hasn't been entered in our attempted print-statement-based tracing. We tried putting print statements at the top of the TimeStamp, Timestamp._new_, tz_localize, maybe_get_tz, get_dst_info, and _get_utc_trans_times_from_dateutil_tz methods, as well as on the tz and its type in Timestamp.__new__, and in each of the if statements for how maybe_get_tz handles strings, and received the following output:

>>> import pandas as pd
In class _Timestamp
In __new__
None
<class 'NoneType'>
In maybe_get_tz
In __new__
None
<class 'NoneType'>
In maybe_get_tz
>>> ts = pd.Timestamp(year=2001, month=10, day=14, hour=0, minute=30, tz='America/Santiago')
In __new__
America/Santiago
<class 'str'>
In maybe_get_tz
5
In maybe_get_tz

This output seemed to confirm that localized wasn't likely involved in the creation of a new timestamp. We did some further experimentation with a nonexistent time in the United States and had it seemingly accept that timestamp's creation but that same timestamp when a time zone was assigned to a naive version of it using tz_localize.

>>> ts = pd.Timestamp(year=2021, month = 3, day = 14, hour = 2, minute = 15, tz = 'America/New_York')
In __new__
America/New_York
<class 'str'>
In maybe_get_tz
5
In maybe_get_tz
>>> ts
Timestamp('2021-03-14 02:15:00-0500', tz='America/New_York')
>>> naive_ts = pd.Timestamp(year=2021, month = 3, day = 14, hour = 2, minute = 15)
In __new__
None
<class 'NoneType'>
In maybe_get_tz
>>> naive_ts.tz_localize('America/New_York')
In localize
In maybe_get_tz
5
in get_dst_info
In __new__
None
<class 'NoneType'>
In maybe_get_tz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/_libs/tslibs/timestamps.pyx", line 1295, in pandas._libs.tslibs.timestamps.Timestamp.tz_localize
    value = tz_localize_to_utc_single(self.value, tz,
  File "pandas/_libs/tslibs/tzconversion.pyx", line 74, in pandas._libs.tslibs.tzconversion.tz_localize_to_utc_single
    return tz_localize_to_utc(
  File "pandas/_libs/tslibs/tzconversion.pyx", line 330, in pandas._libs.tslibs.tzconversion.tz_localize_to_utc
    raise pytz.NonExistentTimeError(stamp)
pytz.exceptions.NonExistentTimeError: 2021-03-14 02:15:00

So we could be doing something wrong or not realizing something but it seems like the Timestamp.__new__ method just doesn't check for nonexistent times. From what we can tell, our timezone in the first example is being handled by the maybe_get_tz method's fifth if statement that seems to simply release it to the pytz package in the singlehanded line tz = pytz.timezone(tz) which may mean that pandas has no innate way of checking new timestamp's time zones at this point.

Let me know if you're able to trace the problem more or if you have any insights. Sorry for the wall of text.

[hewittk]

Update: We think that we managed to resolve this issue by making a naive timestamp and then checking it against the user-requested timezone using tz_localize_to_utc_single. We'll be adding test cases and making a PR soon.

[hewittk]

Take