BUG: Groupby rolling count slower than rolling mean & sum (v1.1.0)

[nrcjea001]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Your code here
import pandas as pd

# Generate sample df
df = pd.DataFrame({'column1': range(600), 'group': 5*['l'+str(i) for i in range(120)]})

# sort by group for easy/efficient joining of new columns to df
df=df.sort_values('group',kind='mergesort').reset_index(drop=True)

# timing of groupby rolling count, sum and mean
%timeit df['mean']=df.groupby('group').rolling(3,min_periods=1)['column1'].mean().values
%timeit df['sum']=df.groupby('group').rolling(3,min_periods=1)['column1'].sum().values
%timeit df['count']=df.groupby('group').rolling(3,min_periods=1)['column1'].count().values

### Output
6.14 ms ± 812 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
5.61 ms ± 179 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
76.1 ms ± 4.78 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Problem description

I am running a groupby rolling count, sum & mean using Pandas v1.1.0 and I notice that the rolling count is considerably slower than the rolling mean & sum. This seems counter intuitive as we can derive the count from the mean and sum and save time.

Expected Output

Expecting more efficient computation of groupby rolling count

### df Output for illustration
print(df.head(10))

   column1 group   mean     sum  count
0        0    l0    0.0     0.0    1.0
1      120    l0   60.0   120.0    2.0
2      240    l0  120.0   360.0    3.0
3      360    l0  240.0   720.0    3.0
4      480    l0  360.0  1080.0    3.0
5        1    l1    1.0     1.0    1.0
6      121    l1   61.0   122.0    2.0
7      241    l1  121.0   363.0    3.0
8      361    l1  241.0   723.0    3.0
9      481    l1  361.0  1083.0    3.0

Output of pd.show_versions()

commit : d9fff27
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 11, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.1.0
numpy : 1.18.5
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.1
setuptools : 49.2.1.post20200802
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.17.0
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fsspec : 0.8.0
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.0
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.48.0

[simonjayhawkins]

Thanks @nrcjea001 for the report.

it appears that mean and sum have significant performance enhancements recently

%timeit df['mean']=df.groupby('group').rolling(3,min_periods=1)['column1'].mean().values
# 9.26 ms ± 170 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # master
# 149 ms ± 3.97 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # 1.0.1

%timeit df['sum']=df.groupby('group').rolling(3,min_periods=1)['column1'].sum().values
# 9.82 ms ± 319 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # master
# 78.3 ms ± 18.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) # 1.0.1

%timeit df['count']=df.groupby('group').rolling(3,min_periods=1)['column1'].count().values
# 128 ms ± 7.01 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) # master
# 140 ms ± 8.53 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # 1.0.1

[pdemarti]

I suspect @nrcjea001 was in fact looking for a size operator instead. I noticed it exists for groupby but not for RollingGroupBy. That might be a simple operation to add and could come in handy. The equivalent rollgrpby.apply(len) or rollgrpby.apply(lambda g: g.shape[0]) works but is 2x slower than rollgrpby.sum() for example. I presume it could be established very efficiently without creating groups, just looking at the indices differences (end - start) of the rolling windows.