DataFrame slicing TimedeltaIndex with str

[Tux1]

Code Sample, a copy-pastable example if possible

cols = ['15:00:00', '15:01:00', '15:02:00', '15:02:30']
cols_td = pd.to_timedelta(cols)

df = pd.DataFrame(0, index=range(4), columns=cols_td)

vw1 = df.loc[:, :cols[-2]]
vw2 = df.loc[:, :cols_td[-2]]
print(vw1)
print(vw2)

print('Statement below must be equal')
print(vw1.equals(vw2))

Problem description

vw1 is wrong. vw2 is correct
Slicing TimdeltaIndex DataFrame with string doesn't seem to work. It doesn't take into account seconds

Expected Output

Both views must be equal

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 79 Stepping 1, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.20.3
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

[mroeschke]

Thanks for the report. I can replicate this on the latest release (v0.23.0). Investigation and PRs welcome!

[Tux1]

I don't manage to find what part of the code is used to process `loc` with Timedelta. If you can indicate me where it is, I can try to fix it

…

On 26 May 2018, at 09:32, Matthew Roeschke ***@***.***> wrote: Thanks for the report. I can replicate this on the latest release (v0.23.0). Investigation and PRs welcome! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[mroeschke]

I am not too familiar with that part of the codebase myself. I recommend using the trace module to see which path indexing takes with Timedelta.

[claria]

This bug still exists in v0.25.3
It must be somehow connected to the precision when the string is converted:

df = pd.DataFrame(index=pd.timedelta_range(start='00:00:00', end='02:00:00', freq='1min'))
df.loc['00:00:00':'01:00:00']
df.loc['00:00:00':'01:00:00.001']

The first .loc gives the wrong result, while the second call returns the correct selection

[jreback]

@claria this is marked as an open issue, PRs are welcome

[claria]

The issue seems to be that at [1], the string is parsed to a Timedelta and the resolution is 'wrongly' assumed to be hourly.

> pd.Timedelta('01:00:00').resolution_string
'H'

And that resolution is then used in the further indexing:

return lbound + to_offset(parsed.resolution_string) - Timedelta(1, "ns")

Should a Timedelta parsed from a string always have the in the string specified precision ('01:00:00' --> second instead of hour) or should we use the precision of the TimeDeltaIndex we are slicing?

[1] https://github.com/pandas-dev/pandas/blob/master/pandas/core/indexes/timedeltas.py#L543

[mroeschke]

It looks like this is a duplicate/similar issue as #33603. Going to consolidate and continue the discussion there