progress_apply reports wrong total count

[joders]

I created a minimal replicating example:

import pandas as pd
from tqdm import tqdm
tqdm.pandas()
df=pd.DataFrame({'a':np.random.randint(2,5,size=100),'a':np.random.randint(2,9,size=100)})
gb = df.groupby('a')
print("group count: " + str(len(gb)))
gb.progress_apply(lambda x:x)

group count: 7
100%|█████████████████████████████████████████████| 8/8 [00:00<00:00, 1834.28it/s]

I did not find an open issue for this.

The relevant line in _tqdm.py is

614: total += 1 # pandas calls update once too many

tqdm/tqdm/_tqdm.py

Line 614 in 69ca16d

total += 1 # pandas calls update once too many

Python 3.6.1
pandas==0.20.3
tqdm==4.19.5

[joders]

I think the problem might be that pandas is actually performing len(gb)+1 operations when you invoke apply on a group with len(gb) items. From the docs:

In the current implementation apply calls func twice on the
first group to decide whether it can take a fast or slow code
path. This can lead to unexpected behavior if func has
side-effects, as they will take effect twice for the first
group.

[casperdcl]

yes, it's an upstream pandas issue, but if the display really annoys you, you could try:

tqdm.pandas(unit_scale=len(gb)/(len(gb)+1.0))

[joders]

dealt with issue now by printing for the user:

15 items to be processed (needs 16 operations)

[chengs]

once #524 is merge, this will be fixed

[chengs]

Since #524 is merge, this can be closed. @casperdcl

[casperdcl]

@joders please try with tqdm>=4.20.0 happy to reopen if it doesn't work for you