New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: indexing with monotonic decreasing DTI #19362

Closed
ruiyuanlu opened this Issue Jan 23, 2018 · 5 comments

Comments

Projects
None yet
5 participants
@ruiyuanlu

ruiyuanlu commented Jan 23, 2018

Hi there, I'm trying to add a column of time using pandas. But some dates doesn't work fine in string format. I'm working on jupyter notebook with Anaconda 4.3.0

Here's what I'm trying to do:

data1 = data.head()
date_list = ["2018-01-02", "2017-02-10","2016-03-10", "2015-03-15", "2014-03-16"]
date_index = pd.to_datetime(date_list)
data1['date'] = date_index
data1

And the output is shown below.
image

Then I try to set 'date' as the index of this data frame.

data1 = data1.set_index("date")
print(data1.index)
data1

This looks reasonable, too.
image

Something weird happens when I try to index data using:

data1.loc['2018-01-02']

It raises KeyError!!!
image

But other dates work fine. For example:

data1.loc['2015-03-15']

image

Why is that?

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.3
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.2.2
numexpr: 2.6.1
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: 0.9999999
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
boto: 2.45.0
pandas_datareader: None

@ruiyuanlu ruiyuanlu changed the title from Pandas raise KeyError or Existing Time Indexing. to Pandas raise KeyError for Existing Time Indexing. Jan 23, 2018

@TomAugspurger

This comment has been minimized.

Contributor

TomAugspurger commented Jan 23, 2018

Can you try on a more recent version of pandas to see if your issue is already fixed?

FYI, text tracebacks are generally more helpful than images.

@jreback

This comment has been minimized.

Contributor

jreback commented Jan 24, 2018

Here's a repro. looks like buggy on 1st element of a montononic descending partial string indexing (though here its an exact match).

In [1]: df = pd.DataFrame({'A': [1,2,3]},index=pd.date_range('20170101', periods=3)[::-1])

In [2]: df
Out[2]: 
            A
2017-01-03  1
2017-01-02  2
2017-01-01  3

In [3]: df.loc['2017-01-03']
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/pandas/pandas/core/indexing.py in _has_valid_type(self, key, axis)
   1501                 if not ax.contains(key):
-> 1502                     error()
   1503             except TypeError as e:

~/pandas/pandas/core/indexing.py in error()
   1496                                .format(key=key,
-> 1497                                        axis=self.obj._get_axis_name(axis)))
   1498 

KeyError: 'the label [2017-01-03] is not in the [index]'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-3-5dd0bbdc6e4e> in <module>()
----> 1 df.loc['2017-01-03']

~/pandas/pandas/core/indexing.py in __getitem__(self, key)
   1367 
   1368             maybe_callable = com._apply_if_callable(key, self.obj)
-> 1369             return self._getitem_axis(maybe_callable, axis=axis)
   1370 
   1371     def _is_scalar_access(self, key):

~/pandas/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1620 
   1621         # fall thru to straight lookup
-> 1622         self._has_valid_type(key, axis)
   1623         return self._get_label(key, axis=axis)
   1624 

~/pandas/pandas/core/indexing.py in _has_valid_type(self, key, axis)
   1508                 raise
   1509             except:
-> 1510                 error()
   1511 
   1512         return True

~/pandas/pandas/core/indexing.py in error()
   1495                 raise KeyError(u"the label [{key}] is not in the [{axis}]"
   1496                                .format(key=key,
-> 1497                                        axis=self.obj._get_axis_name(axis)))
   1498 
   1499             try:

KeyError: 'the label [2017-01-03] is not in the [index]'

In [4]: df.loc['2017-01-02']
Out[4]: 
            A
2017-01-02  2

@jreback jreback added this to the Next Major Release milestone Jan 24, 2018

@jreback jreback changed the title from Pandas raise KeyError for Existing Time Indexing. to BUG: indexing with monotonic decreasing DTI Jan 24, 2018

@swyoon

This comment has been minimized.

Contributor

swyoon commented Feb 7, 2018

may i work on this issue?

@jreback

This comment has been minimized.

Contributor

jreback commented Feb 7, 2018

@swyoon absolutely!

mapehe added a commit to mapehe/pandas that referenced this issue Apr 13, 2018

mapehe added a commit to mapehe/pandas that referenced this issue Apr 13, 2018

@mapehe

This comment has been minimized.

Contributor

mapehe commented Apr 13, 2018

Seems that this is caused by the fact that np.any([0]) is False. Added a suggested solution in #20677.

@jreback jreback modified the milestones: Next Major Release, 0.23.0 Apr 14, 2018

mapehe added a commit to mapehe/pandas that referenced this issue Apr 14, 2018

mapehe added a commit to mapehe/pandas that referenced this issue Apr 14, 2018

@jreback jreback modified the milestones: 0.23.0, Next Major Release Apr 14, 2018

mapehe added a commit to mapehe/pandas that referenced this issue Apr 16, 2018

mapehe added a commit to mapehe/pandas that referenced this issue Apr 16, 2018

jreback added a commit that referenced this issue Apr 20, 2018

fjetter added a commit to fjetter/pandas that referenced this issue Apr 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment