to_datetime %Y%m%d does not coerce correctly

[CarstVaartjes]

Hi,

see this example:

# imagine a dataframe with a from/to 
x_df = DataFrame([[20120101, 20121231], [20130101, 20131231], [20140101, 20141231], [20150101, 99991231]])
x_df.columns = ['date_from', 'date_to']
date_def = '%Y%m%d'
# so everything is ok & peachy for the from dates
x_df['date_from_2'] = pd.to_datetime(x_df['date_from'], format=date_def, coerce=True)
x_df['date_from_2'].dtype
list(x_df['date_from_2'])
# but with out of bound dates it goes horribly wrong
x_df['date_to_2'] = pd.to_datetime(x_df['date_to'], format=date_def, coerce=True)
x_df['date_to_2'].dtype 
list(x_df['date_to_2']) # note the lack of NATs and conversion to datetime.datetime instead of np.datetime64
# now we can do 
x_df['date_to_3'] = [np.datetime64(date_val, unit='s') for date_val in x_df['date_to_2']] # works great but unfortunately pandas chose to aim for nanoseconds as a standard for date detail...
x_df['date_to_3'] = [pd.Timestamp(date_val) for date_val in x_df['date_to_2']] # which breaks on the 99991231 example

This is a bug, but also related to the discussion in #7307. Probably has to do with:
"Note Specifying a format argument will potentially speed up the conversion considerably and on versions later then 0.13.0 explicitly specifying a format string of ‘%Y%m%d’ takes a faster path still."

[jreback]

easy enough

[CarstVaartjes]

Thanks! (insightful commit, I guess it's "easy" when you know the libtools and all places by heart hehe ;)

[jreback]

haha...(and I wrote the original :)

[jrovegno]

where is the like button? when you search for an error and you realize that is a bug fixed in a newer release. 👍

[jreback]

@jrovegno thanks!