BUG: GroupBy().fillna() performance regression

[smithto1]

• closes BUG: GroupBy().fillna() performance regression #36757
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

A performance regression was introduced with #30679 in handling grouping on a duplicate axis. The regression can be avoided by skipping the call to get_indexer_non_unique if the axis is unchanged (i.e. like it is in fillna).

I don't know how to write a test for this fix since it is just an issue of speed. If there is a test pattern or other check that should be included please highlight and I'm happy to add it.

In lieu of a test, running the minimal example from the issue report on the fixed branch shows the performance fix:

In [5]: import pandas as pd
   ...: import numpy as np
   ...: 
   ...: N = 2000
   ...: df = pd.DataFrame({"A": [1] * N, "B": [np.nan, 1.0] * (N // 2)})
   ...: df = df.sort_values("A").set_index("A")
   ...: 
   ...: %time df.groupby("A")["B"].fillna(method="ffill")
   ...:
Wall time: 1.03 ms
Out[5]:
A
1    NaN
1    1.0
1    1.0
1    1.0
1    1.0
    ...
1    1.0
1    1.0
1    1.0
1    1.0
1    1.0
Name: B, Length: 2000, dtype: float64            

[jreback]

an asv is appropriate here (or if we have one that covers just show the results)

[jreback]

if not u can add one along the lines of what i posted

[jreback]

pls add an asv or show an existing one that is caught by this

[pep8speaks]

Hello @smithto1! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-10-17 22:24:29 UTC

[smithto1]

pls add an asv or show an existing one that is caught by this

No existing one seemed to catch it so added a new one.

asv continuous --quick -f 1.1 -b groupby.FillNA upstream/master HEAD

(pandas-dev) C:\git\pandas-smithto1\asv_bench>python C:\anaconda3\envs\pandas-dev\Scripts\asv.exe continuous --quick -f 1.1 -b groupby.FillNA upstream/master HEAD
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
·· Installing 5412aaa into conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
· Running 4 total benchmarks (2 commits * 1 environments * 2 benchmarks)
[ 0.00%] · For pandas commit 58dcafa (round 1/2):
[ 0.00%] ·· Building for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 0.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 25.00%] · For pandas commit 5412aaa (round 1/2):
[ 25.00%] ·· Building for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 25.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 50.00%] · For pandas commit 5412aaa (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 62.50%] ··· groupby.FillNA.time_df_ffill 5.28±0ms
[ 75.00%] ··· groupby.FillNA.time_srs_ffill 3.12±0ms
[ 75.00%] · For pandas commit 58dcafa (round 2/2):
[ 75.00%] ·· Building for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 87.50%] ··· groupby.FillNA.time_df_ffill 18.8±0s
[100.00%] ··· groupby.FillNA.time_srs_ffill 17.7±0s
before after ratio
[58dcafa] [5412aaa]

•     18.8±0s         5.28±0ms     0.00  groupby.FillNA.time_df_ffill
  

•     17.7±0s         3.12±0ms     0.00  groupby.FillNA.time_srs_ffill
  

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

[jreback]

pls show the results of the asv

[smithto1]

pls show the results of the asv

(pandas-dev) C:\git\pandas-smithto1\asv_bench>python C:\anaconda3\envs\pandas-dev\Scripts\asv.exe continuous -f 1.1 -b groupby.FillNA upstream/master HEAD
       before           after         ratio
     [85793fb8]       [8a93e0b8]
     <issue36757~1^2>       <issue36757>
-     2.68±0.04ms      2.00±0.02ms     0.74  groupby.FillNA.time_df_bfill
-     2.68±0.01ms      1.99±0.05ms     0.74  groupby.FillNA.time_df_ffill
-     1.85±0.03ms      1.28±0.02ms     0.69  groupby.FillNA.time_srs_bfill
-     1.87±0.03ms      1.26±0.04ms     0.67  groupby.FillNA.time_srs_ffill

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

[alippai]

LGTM, thanks @smithto1 fixing the issue reported by me recently so swiftly

[smithto1]

@jreback can you have another look.

[jreback]

thanks @smithto1

[simonjayhawkins]

@meeseeksdev backport 1.1.x

[smithto1]

LGTM, thanks @smithto1 fixing the issue reported by me recently so swiftly

@alippai Thanks for reporting the issue.