New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:00:03'), try using the 'ambiguous' argument #14402

Closed
adamschultz opened this Issue Oct 12, 2016 · 5 comments

Comments

Projects
None yet
3 participants
@adamschultz

adamschultz commented Oct 12, 2016

In Pandas 0.19.0, there seems to be an error when setting the timezone for times close to the dst cutoff.

This issue appears very similar to #11619 and #11626, both of which have been closed. The traceback is below.

df['Timestamp'] = df['Timestamp Date'] + " " + df['Timestamp Time']
df['Timestamp'] = pd.to_datetime(df['Timestamp']).dt.tz_localize('US/Central')

---------------------------------------------------------------------------
AmbiguousTimeError                        Traceback (most recent call last)
 in ()
----> 1 df['Timestamp'] = pd.to_datetime(df['Timestamp']).dt.tz_localize('US/Central')

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in f(self, *args, **kwargs)
    208 
    209             def f(self, *args, **kwargs):
--> 210                 return self._delegate_method(name, *args, **kwargs)
    211 
    212             f.__name__ = name

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/tseries/common.pyc in _delegate_method(self, name, *args, **kwargs)
    130 
    131         method = getattr(self.values, name)
--> 132         result = method(*args, **kwargs)
    133 
    134         if not is_list_like(result):

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/util/decorators.pyc in wrapper(*args, **kwargs)
     89                 else:
     90                     kwargs[new_arg_name] = new_arg_value
---> 91             return func(*args, **kwargs)
     92         return wrapper
     93     return _deprecate_kwarg

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/tseries/index.pyc in tz_localize(self, tz, ambiguous, errors)
   1823             new_dates = tslib.tz_localize_to_utc(self.asi8, tz,
   1824                                                  ambiguous=ambiguous,
-> 1825                                                  errors=errors)
   1826         new_dates = new_dates.view(_NS_DTYPE)
   1827         return self._shallow_copy(new_dates, tz=tz)

pandas/tslib.pyx in pandas.tslib.tz_localize_to_utc (pandas/tslib.c:70327)()

AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:00:03'), try using the 'ambiguous' argument

Here is some relevant information on the data frame.

df[['Timestamp Date', 'Timestamp Time', 'Timestamp']].head()
  Timestamp Date Timestamp Time               Timestamp
0     01-04-2016    07:10:03 AM  01-04-2016 07:10:03 AM
1     01-04-2016    07:20:04 AM  01-04-2016 07:20:04 AM
2     01-04-2016    07:30:03 AM  01-04-2016 07:30:03 AM
3     01-04-2016    07:40:03 AM  01-04-2016 07:40:03 AM
4     01-04-2016    07:50:03 AM  01-04-2016 07:50:03 AM

df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 286834 entries, 0 to 286833
Timestamp Date       286834 non-null object
Timestamp Time       286834 non-null object
Timestamp            286834 non-null object

I also had this same issue in Pandas 0.18.0 before upgrading to 0.19.0.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 12, 2016

Contributor

can you show a copy-pastable example. e.g. construct a frame exactly like what you need. Then apply operations to show an error.

Contributor

jreback commented Oct 12, 2016

can you show a copy-pastable example. e.g. construct a frame exactly like what you need. Then apply operations to show an error.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 12, 2016

Contributor
In [25]: Timestamp('2015-11-01 01:00:03').tz_localize('US/Central',ambiguous=False)
Out[25]: Timestamp('2015-11-01 01:00:03-0600', tz='US/Central')

In [26]: Timestamp('2015-11-01 01:00:03').tz_localize('US/Central',ambiguous=True)
Out[26]: Timestamp('2015-11-01 01:00:03-0500', tz='US/Central')

Contributor

jreback commented Oct 12, 2016

In [25]: Timestamp('2015-11-01 01:00:03').tz_localize('US/Central',ambiguous=False)
Out[25]: Timestamp('2015-11-01 01:00:03-0600', tz='US/Central')

In [26]: Timestamp('2015-11-01 01:00:03').tz_localize('US/Central',ambiguous=True)
Out[26]: Timestamp('2015-11-01 01:00:03-0500', tz='US/Central')

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 12, 2016

Contributor
In [45]: pd.to_datetime(df.iloc[:,1] + ' ' + df.iloc[:, 2])
Out[45]: 
0   2016-01-04 07:10:03
1   2016-01-04 07:20:04
2   2016-01-04 07:30:03
3   2016-01-04 07:40:03
4   2016-01-04 07:50:03
dtype: datetime64[ns]

In [46]: pd.to_datetime(df.iloc[:,1] + ' ' + df.iloc[:, 2]).dt.tz_localize('US/Central')
Out[46]: 
0   2016-01-04 07:10:03-06:00
1   2016-01-04 07:20:04-06:00
2   2016-01-04 07:30:03-06:00
3   2016-01-04 07:40:03-06:00
4   2016-01-04 07:50:03-06:00
dtype: datetime64[ns, US/Central]

Contributor

jreback commented Oct 12, 2016

In [45]: pd.to_datetime(df.iloc[:,1] + ' ' + df.iloc[:, 2])
Out[45]: 
0   2016-01-04 07:10:03
1   2016-01-04 07:20:04
2   2016-01-04 07:30:03
3   2016-01-04 07:40:03
4   2016-01-04 07:50:03
dtype: datetime64[ns]

In [46]: pd.to_datetime(df.iloc[:,1] + ' ' + df.iloc[:, 2]).dt.tz_localize('US/Central')
Out[46]: 
0   2016-01-04 07:10:03-06:00
1   2016-01-04 07:20:04-06:00
2   2016-01-04 07:30:03-06:00
3   2016-01-04 07:40:03-06:00
4   2016-01-04 07:50:03-06:00
dtype: datetime64[ns, US/Central]

@adamschultz

This comment has been minimized.

Show comment
Hide comment
@adamschultz

adamschultz Oct 12, 2016

Thanks for the quick reply. Here's the traceback from a minimal, reproducible example with a one-row data frame.

import pandas as pd

df2 = pd.DataFrame({'Timestamp Date': '11-01-2015', 
                   'Timestamp Time': '01:00:03'}, 
                    index=range(1))
df2['Timestamp'] = df2['Timestamp Date'] + " " + df2['Timestamp Time']
df2['Timestamp'] = pd.to_datetime(df2['Timestamp']).dt.tz_localize('US/Central')

---------------------------------------------------------------------------
AmbiguousTimeError                        Traceback (most recent call last)
 in ()
      5 df2.head()
      6 df2['Timestamp'] = df2['Timestamp Date'] + " " + df2['Timestamp Time']
----> 7 df2['Timestamp'] = pd.to_datetime(df2['Timestamp']).dt.tz_localize('US/Central')

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in f(self, *args, **kwargs)
    208 
    209             def f(self, *args, **kwargs):
--> 210                 return self._delegate_method(name, *args, **kwargs)
    211 
    212             f.__name__ = name

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/tseries/common.pyc in _delegate_method(self, name, *args, **kwargs)
    130 
    131         method = getattr(self.values, name)
--> 132         result = method(*args, **kwargs)
    133 
    134         if not is_list_like(result):

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/util/decorators.pyc in wrapper(*args, **kwargs)
     89                 else:
     90                     kwargs[new_arg_name] = new_arg_value
---> 91             return func(*args, **kwargs)
     92         return wrapper
     93     return _deprecate_kwarg

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/tseries/index.pyc in tz_localize(self, tz, ambiguous, errors)
   1823             new_dates = tslib.tz_localize_to_utc(self.asi8, tz,
   1824                                                  ambiguous=ambiguous,
-> 1825                                                  errors=errors)
   1826         new_dates = new_dates.view(_NS_DTYPE)
   1827         return self._shallow_copy(new_dates, tz=tz)

pandas/tslib.pyx in pandas.tslib.tz_localize_to_utc (pandas/tslib.c:70327)()

AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:00:03'), try using the 'ambiguous' argument

adamschultz commented Oct 12, 2016

Thanks for the quick reply. Here's the traceback from a minimal, reproducible example with a one-row data frame.

import pandas as pd

df2 = pd.DataFrame({'Timestamp Date': '11-01-2015', 
                   'Timestamp Time': '01:00:03'}, 
                    index=range(1))
df2['Timestamp'] = df2['Timestamp Date'] + " " + df2['Timestamp Time']
df2['Timestamp'] = pd.to_datetime(df2['Timestamp']).dt.tz_localize('US/Central')

---------------------------------------------------------------------------
AmbiguousTimeError                        Traceback (most recent call last)
 in ()
      5 df2.head()
      6 df2['Timestamp'] = df2['Timestamp Date'] + " " + df2['Timestamp Time']
----> 7 df2['Timestamp'] = pd.to_datetime(df2['Timestamp']).dt.tz_localize('US/Central')

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/core/base.pyc in f(self, *args, **kwargs)
    208 
    209             def f(self, *args, **kwargs):
--> 210                 return self._delegate_method(name, *args, **kwargs)
    211 
    212             f.__name__ = name

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/tseries/common.pyc in _delegate_method(self, name, *args, **kwargs)
    130 
    131         method = getattr(self.values, name)
--> 132         result = method(*args, **kwargs)
    133 
    134         if not is_list_like(result):

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/util/decorators.pyc in wrapper(*args, **kwargs)
     89                 else:
     90                     kwargs[new_arg_name] = new_arg_value
---> 91             return func(*args, **kwargs)
     92         return wrapper
     93     return _deprecate_kwarg

/Users/adam/anaconda/lib/python2.7/site-packages/pandas/tseries/index.pyc in tz_localize(self, tz, ambiguous, errors)
   1823             new_dates = tslib.tz_localize_to_utc(self.asi8, tz,
   1824                                                  ambiguous=ambiguous,
-> 1825                                                  errors=errors)
   1826         new_dates = new_dates.view(_NS_DTYPE)
   1827         return self._shallow_copy(new_dates, tz=tz)

pandas/tslib.pyx in pandas.tslib.tz_localize_to_utc (pandas/tslib.c:70327)()

AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:00:03'), try using the 'ambiguous' argument

@jreback jreback added the Bug label Oct 12, 2016

@jreback jreback added this to the 0.19.1 milestone Oct 12, 2016

jreback added a commit to jreback/pandas that referenced this issue Oct 12, 2016

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 12, 2016

Contributor

@adamschultz ok fixed in #14405 ; this was a bug in interpreting a passed boolean. Note that you still MUST pass the ambiguous argument, as this fundamentally IS an ambiguous time.

Contributor

jreback commented Oct 12, 2016

@adamschultz ok fixed in #14405 ; this was a bug in interpreting a passed boolean. Note that you still MUST pass the ambiguous argument, as this fundamentally IS an ambiguous time.

jreback added a commit to jreback/pandas that referenced this issue Oct 13, 2016

@jreback jreback closed this in a40e185 Oct 13, 2016

tworec pushed a commit to RTBHOUSE/pandas that referenced this issue Oct 21, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment