to_datetime parsing bug when using format

[michaelaye]

import datetime as dt
import pandas as pd
val = '01-Apr-2011 00:00:01.978'
print 'pandas version:',pd.__version__
print 'Value to parse:',val
format = '%d-%b-%Y %H:%M:%S.%f'
print 'datetime.strptime        :',dt.datetime.strptime(val, format)
print 'to_datetime, w/out format:',pd.to_datetime(val)
print 'to_datetime, w/ format   :', pd.to_datetime(val, format=format)

pandas version: 0.12.0.dev-1101391
Value to parse: 01-Apr-2011 00:00:01.978
datetime.strptime        : 2011-04-01 00:00:01.978000
to_datetime, w/out format: 2011-04-01 00:00:01.978000
to_datetime, w/ format   : 2011-03-31 23:24:13.516352

[michaelaye]

FYI, the reason why I want to use to_datetime() with a given format string is speed.

[hayd]

perhaps #3669 was prematurely closed (cc #2213 and #3890...)

[jorisvandenbossche]

@hayd I don't think so, #3669 was about that the "format" argument was ignored completely. This is about that there seems to be a bug in the parser used when format is given. As an example, it doesn't matter if you give a single string or an array, both give a wrong result:

In [12]:  pd.to_datetime('01-Apr-2011 00:00:01.978', format= '%d-%b-%Y %H:%M:%S.%f')
Out[12]: Timestamp('2011-03-31 23:24:13.516352', tz=None)

In [14]:  pd.to_datetime(np.array(['01-Apr-2011 00:00:01.978']), format= '%d-%b-%Y %H:%M:%S.%f')
Out[14]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2011-03-31 23:24:13.516352]
Length: 1, Freq: None, Timezone: None

And it seems that it has something to do with the parsing of the microseconds:

In [17]:  pd.to_datetime('01-Apr-2011 00:00:01.000', format= '%d-%b-%Y %H:%M:%S.
Out[17]: Timestamp('2011-04-01 00:00:01', tz=None)

In [18]:  pd.to_datetime('01-Apr-2011 00:00:01.001', format= '%d-%b-%Y %H:%M:%S.
Out[18]: Timestamp('2011-04-01 00:16:41', tz=None)