Fix Progress Bar in Notebook

[jeff-hernandez]

This closes #791 by preventing an interrupted progress bar from double printing when running in a jupyter notebook.

[codecov]

Codecov Report

Merging #932 into master will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #932      +/-   ##
==========================================
+ Coverage   98.17%   98.22%   +0.04%     
==========================================
  Files         119      119              
  Lines       10867    10859       -8     
==========================================
- Hits        10669    10666       -3     
+ Misses        198      193       -5     

Impacted Files	Coverage Δ
...computational_backends/calculate_feature_matrix.py	98.58% <100.00%> (+<0.01%)	⬆️
featuretools/utils/gen_utils.py	100.00% <100.00%> (+8.92%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcb0d87...d04ef35. Read the comment docs.

[rwedge]

Looks good

[casperdcl]

couple of lines should be removed and replaced with tqdm>=4.45.0 in requirements.txt

[casperdcl]

this line should be removed

[casperdcl]

this line should be removed

[rwedge]

@casperdcl, I implemented those changes on a different branch, tqdm-suggestions, but it did not resolve the issue

Running this code in a notebook:

import featuretools as ft
import tqdm

print(tqdm.__version__)

es = ft.demo.load_mock_customer(return_entityset=True)

cutoff_time = es['transactions'].df[["transaction_id", "transaction_time"]]

ft.dfs(entityset=es, target_entity='transactions', cutoff_time=cutoff_time, verbose=True)

Interrupting the kernel during the dfs call and rerunning dfs caused the progress bar to print on a new line each update

[casperdcl]

Thanks @rwedge - I suspect this may be a pandas wrapper issue. Can you try interrupting and re-running:

from tqdm import trange
for i in trange(int(1e9)):
    pass

[rwedge]

@casperdcl - running the code you provided, interrupting the loop, then re-running the loop does not result in unexpected progress bar behavior. It looks normal.

[casperdcl]

Great so this is a wrapper issue; will open a new issue upstream

[rwedge]

Thanks for helping look into this!

[casperdcl]

@kmax12 I assume the reason you became my first ever sponsor? ❤️ 🎊

[kmax12]

@casperdcl the reason is that I've been a longtime user of tqdm and wanted to support you. the help here is above and beyond. we really appreciate it. thanks!

[casperdcl]

:D I'd like to claim that I have free time I'd like to donate to the world as an act of kindness but the truth is more like this:

I mean; I wrote the tqdm.pandas wrapper years ago, continuously supported since then, and to date still have never used it myself.

[casperdcl]

With conda create -n ft 'featuretools<0.14.0' tqdm ipykernel:

#from tqdm.auto import tqdm
from tqdm import tqdm
import pandas as pd
from random import random
import warnings

with warnings.catch_warnings():
    warnings.filterwarnings('ignore', category=FutureWarning)
    tqdm.pandas()

df = pd.DataFrame((random() for _ in range(10)) for _ in range(int(1e6)))

try:
    df.progress_apply(lambda x: x**x ** (1/x) ** x ** (1/x) ** x ** (1/x))
except KeyboardInterrupt:
    print("finished with %d instances left" % len(tqdm._instances))

60%|██████    | 6/10 [00:02<00:01,  2.77it/s]
finished with 0 instances left

#from tqdm.auto import tqdm
from tqdm import tqdm
import featuretools as ft

es = ft.demo.load_mock_customer(return_entityset=True)
cutoff_time = es['transactions'].df[["transaction_id", "transaction_time"]]

try:
    ft.dfs(entityset=es, target_entity='transactions', cutoff_time=cutoff_time, verbose=True)
except KeyboardInterrupt:
    print("finished with %d instances left" % len(tqdm._instances))

Built 33 features
Elapsed: 00:02 | Progress:   6%|▌         finished with 1 instances left
Elapsed: 00:02 | Progress:   6%|▌         

So tqdm.pandas works as expected but featuretools doesn't.

Looks like featuretools may have some error handling. My guess is your custom function which you're handing to pandas will itself catch the error and not let it propagate to tqdm. If you're doing your own error handling then it's your responsibility to clean up everything, including but not limited to tqdm. You'd really need to close() the tqdm instance rather than hackily clearing the whole internal _instances set:

https://github.com/tqdm/tqdm/blob/5e8978909c639bc20f58bf877e66cc6a0b9fb5bf/tqdm/std.py#L766-L767

[rwedge]

@casperdcl thanks for looking into this, I've opened a new issue