Use tqdm for Jupyter Notebooks

[anderl80]

When using tsfresh in Jupyter Notebook, output from tqdm results bar is not overwriting itself but creating a line for every change in percentage. Can we somehow build a switch for using tqdm_notebook?

[rpanai]

This seems to see a bug. Check this issue out. It happens to me whenever I have to interrupt the kernel at some point. Do you mind to share a mcve?

[nils-braun]

@anderl80 Reading what @rpanai has posted, I think we can close this issue. Correct?

[rjurney]

Can someone pretty please fix this? It makes tqdm worse than worthless.

[dbarbier]

No problem here with tqdm in Jupyter notebooks; do you reproduce this issue with very simple workflows, as in https://tsfresh.readthedocs.io/en/latest/text/quick_start.html ?

[dbarbier]

@anderl80 @rjurney And if you edit tsfresh/utilities/distribution.py and replace

from tqdm import tqdm

by

from tqdm.auto import tqdm

does it fix your issues?

[gunjan-solanki]

It's an old issue with tqdm on jupyter . see here : tqdm/tqdm#375

Workaround is to clear tqdm instance with

tqdm._instances.clear()

whenever it occurs.

tqdm/tqdm#375 (comment)

[dbarbier]

Different workarounds have been suggested in this issue; il looks like main tqdm developer advocates importing tqdm.auto, thus it is a better option IMO if it works, this is why I requested feedback.

[nils-braun]

The problem I have with the tqdm.auto is that you might fall into

ImportError: IntProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html

as we are note requiring ipywidgets.
What do people think?
I mean it looks a lot more beautiful with the ipywidgets no question.

[rjurney]

This is a common enough problem that it seems like it should ‘just work’ in both a console and a Jupyter Notebook. People don’t always know the context in which their code will be run and requiring every library to determine its execution context and use a different API seems an enormous duplication of effort that could be done once by making the default method for tqdm work in both console and Jupyter.

Am I right?

[nils-braun]

I am closing this issue as it has nothing to do with tsfresh, but is an upstream issue for tqdm (where it was addressed or at least discussed already).

[rjurney]

@nils-braun this breaks your library on the most popular platform where it is used. This attitude is unfortunate and impacts many (most?) users of the library.

[MaxBenChrist]

What about detecting the environment where tsfresh is run and use the respective Bar? See also https://stackoverflow.com/questions/15411967/how-can-i-check-if-code-is-executed-in-the-ipython-notebook

[nils-braun]

@rjurney sorry did not want to be rude! :-)
I think then I misunderstood the problem.

I thought it is a problem in the tqdm library (and if we can fix it there much more people can benefit from this) - but of course we can also do a workaround in tsfresh (which will obviously not fix the root cause). Do you already have a solution in mind and are you willing to do a PR with it? We are always happy for contributors!

[casperdcl]

fixed in tqdm>=4.45.0, which should not require any special handling on exceptions (no need to catch and clear instances). If it does, please post on tqdm/tqdm#548.

Also if you can't use tqdm.auto.tqdm/tqdm.notebook.tqdm and still have this issue even when no exceptions are triggered then it's probably just your env not supporting \r. https://github.com/tqdm/tqdm/#faq-and-known-issues

[casperdcl]

btw @nils-braun the expected behaviour is:

from tqdm import tqdm  # plain-text version; relies on `\r`
from tqdm.notebook import tqdm  # notebook version; relies on `ipywidgets`
from tqdm.autonotebook import tqdm  # auto-detect frontend, printing warning
from tqdm.auto import tqdm  # auto-detect frontend, not printing warning

so basically from tqdm.auto import tqdm should fallback to just the plain-text version if ipywidgets is not found, without printing any warnings. Is this not the case?

[nils-braun]

@casperdcl Thanks for the very nice update and thanks for reporting it here :-)
(by the way, thanks for the super nice library!)
The from tqdm.auto import tqdm work as expected in console mode as well as in jupyter notebooks, as long as the ipywidgets package is installed. If it is not installed and you call it in a jupyter notebook, it will first fail with

NameError: name 'IProgress' is not defined

and the error handling of that will fail in

ImportError: FloatProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html

I guess this is expected (from looking at your code you are trying out different combinations of IPython.html.widgets and ipywidgets) and you are printing a very nice error message, so in my opinion this is fine.

[casperdcl]

Hmm. I'm not sure about the case of running in a notebook sans ipywidgets when using tqdm.auto. Perhaps it's best to raise an ImportWarning and fallback to the CLI version rather than raising an ImportError?

[nils-braun]

@casperdcl I guess it depends on what message you want to communicate: is tqdm well tested and well working in CLI mode even in notebooks (e.g. no bugs like the one described here) or do you assume that notebooks work best with the widgets?
The former means more testing cases you need to cover and the latter means more restrictions for the users.
(In my opinion tqdm looks really good with widgets :-))

[casperdcl]

is tqdm well tested and well working in CLI mode even in notebooks (e.g. no bugs like the one described here) or do you assume that notebooks work best with the widgets?

Both :)