groupby(pd.Grouper) ignores loffset

[ghost]

Code Sample, a copy-pastable example if possible

import pandas as pd
df = pd.date_range(start="1/1/2018", end="1/2/2018", periods=1000).to_frame()
print(df.resample("1h", loffset="15min").last().index)
print(df.groupby(pd.Grouper(freq="1h", loffset="15min")).last().index)

Problem description

I thought the two calls should be equivalent. However, the output is:

DatetimeIndex(['2018-01-01 00:15:00', '2018-01-01 01:15:00',
               '2018-01-01 02:15:00', '2018-01-01 03:15:00',
               '2018-01-01 04:15:00', '2018-01-01 05:15:00',
               '2018-01-01 06:15:00', '2018-01-01 07:15:00',
               '2018-01-01 08:15:00', '2018-01-01 09:15:00',
               '2018-01-01 10:15:00', '2018-01-01 11:15:00',
               '2018-01-01 12:15:00', '2018-01-01 13:15:00',
               '2018-01-01 14:15:00', '2018-01-01 15:15:00',
               '2018-01-01 16:15:00', '2018-01-01 17:15:00',
               '2018-01-01 18:15:00', '2018-01-01 19:15:00',
               '2018-01-01 20:15:00', '2018-01-01 21:15:00',
               '2018-01-01 22:15:00', '2018-01-01 23:15:00',
               '2018-01-02 00:15:00'],
              dtype='datetime64[ns]', freq='H')
DatetimeIndex(['2018-01-01 00:00:00', '2018-01-01 01:00:00',
               '2018-01-01 02:00:00', '2018-01-01 03:00:00',
               '2018-01-01 04:00:00', '2018-01-01 05:00:00',
               '2018-01-01 06:00:00', '2018-01-01 07:00:00',
               '2018-01-01 08:00:00', '2018-01-01 09:00:00',
               '2018-01-01 10:00:00', '2018-01-01 11:00:00',
               '2018-01-01 12:00:00', '2018-01-01 13:00:00',
               '2018-01-01 14:00:00', '2018-01-01 15:00:00',
               '2018-01-01 16:00:00', '2018-01-01 17:00:00',
               '2018-01-01 18:00:00', '2018-01-01 19:00:00',
               '2018-01-01 20:00:00', '2018-01-01 21:00:00',
               '2018-01-01 22:00:00', '2018-01-01 23:00:00',
               '2018-01-02 00:00:00'],
              dtype='datetime64[ns]', freq='H')

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Linux
OS-release : 4.4.0-17134-Microsoft
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.1
numpy : 1.17.1
pytz : 2019.2
dateutil : 2.8.0
pip : 19.2.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.8.0
pandas_datareader: 0.7.4
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.1
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.14.1
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[WillAyd]

These calls aren't really equivalent as the first returns a reindexed data frame whereas the second maintains the original index values but buckets them according to a particular time grouping

[WillAyd]

Hmm actually not sure. I don't think we have a ton of test coverage on loffset with groupby though. If you want to take a look would certainly accept PRs to clean up

[ghost]

@WillAyd I forgot to mention that not only the index is "wrong", but also the aggregation does not take the offset into account. I will update the issue shortly.

[ghost]

I was mixing things up. loffset should not have an effect on the aggregation, but only on the "resampled time labels" (quote from resample doc). For Grouper, the documentation does not really state what it is actually for. For me it seems natural that it should be the same as in resample and therefore I still think it is an issue.

[WillAyd]

Yea I think so too. That part of GroupBy I don't think we have a lot of coverage on so yea would certainly welcome any investigation and PRs you have into it

[xuancong84]

WORKAROUND: use base= instead of loffset= in pd.Grouper, developer's implementation and documentation are still pending.

Ideally, there needs to be 3 parameters, loffset (label_offset, default=0, the starting boundary offset of every group label), gstart (group_start, default=loffset, the starting offset of every group's data) and gspan (group_span, default=1, the span of every group's data).

For example, if we want to split a time-series data from 9:00am every day to 9:00am the next day, while keeping the group label at 0am (so that the datetime object reduces to date, i.e., 2020-05-26 00:00:00 => 2020-05-26), however, within each group we want the past 2 days of data (i.e., from 9am two days before that day til 9am on that day) in addition to that day's data (i.e., from 9am on that day til 9am the next day), [in other words, there will be duplicate entries among adjacent groups and the total number of data rows will be tripled], then we should call pd.Grouper(freq='D', loffset=0, gstart=9.0/24-2, gspan=3). This will give rise to:
[('2020-01-01T00:00:00', <data from 2019-12-30T09:00:00 to 2020-01-02T09:00:00>), ('2020-01-02T00:00:00', <data from 2019-12-31T09:00:00 to 2020-01-03T09:00:00>), ('2020-01-03T00:00:00', <data from 2020-01-01T09:00:00 to 2020-01-04T09:00:00>), ('2020-01-04T00:00:00', <data from 2020-01-02T09:00:00 to 2020-01-05T09:00:00>), ...]