Each iteration of progressbar starts a new line in Jupyter

[jolespin]

If I'm running tqdm on a cell in Jupyter and cancel it, when I run tqdm again it prints on a new line for each iteration. Is this a common problem? I have seen this happen to others online but haven't seen a solution. I checked https://github.com/tqdm/tqdm/#help and didn't see the issue. If there is a solution please share. Thanks

Example below:

 0%|          | 1/3542 [00:00<18:27,  3.20it/s]

  0%|          | 16/3542 [00:00<13:08,  4.47it/s]

  1%|          | 24/3542 [00:00<09:31,  6.15it/s]

  1%|          | 32/3542 [00:00<06:56,  8.42it/s]

  1%|          | 40/3542 [00:00<05:07, 11.37it/s]

  1%|▏         | 48/3542 [00:01<03:50, 15.17it/s]

  2%|▏         | 56/3542 [00:01<02:54, 20.00it/s]

  2%|▏         | 65/3542 [00:01<02:13, 26.00it/s]

  2%|▏         | 80/3542 [00:01<01:40, 34.47it/s]

  3%|▎         | 102/3542 [00:01<01:15, 45.69it/s]

  3%|▎         | 115/3542 [00:01<01:03, 53.89it/s]

  4%|▎         | 127/3542 [00:01<00:56, 60.65it/s]

  4%|▍         | 138/3542 [00:01<00:50, 67.34it/s]

  4%|▍         | 152/3542 [00:02<00:44, 76.91it/s]

  5%|▍         | 168/3542 [00:02<00:38, 87.94it/s]

  5%|▌         | 184/3542 [00:02<00:34, 96.67it/s]

  6%|▌         | 200/3542 [00:02<00:31, 106.87it/s]

  6%|▌         | 216/3542 [00:02<00:28, 115.95it/s]

  7%|▋         | 232/3542 [00:02<00:26, 124.56it/s]

  7%|▋         | 248/3542 [00:02<00:25, 129.90it/s]

  7%|▋         | 264/3542 [00:02<00:24, 133.62it/s]

  8%|▊         | 280/3542 [00:02<00:23, 138.38it/s]

  8%|▊         | 296/3542 [00:03<00:22, 143.22it/s]

  9%|▉         | 314/3542 [00:03<00:22, 143.28it/s]

  9%|▉         | 332/3542 [00:03<00:21, 151.39it/s]

 11%|█         | 376/3542 [00:03<00:17, 186.09it/s]

 11%|█▏        | 399/3542 [00:03<00:17, 175.56it/s]

 12%|█▏        | 420/3542 [00:03<00:18, 165.79it/s]

 13%|█▎        | 452/3542 [00:03<00:17, 172.49it/s]

 14%|█▎        | 484/3542 [00:03<00:17, 179.45it/s]

 15%|█▍        | 516/3542 [00:04<00:16, 185.41it/s]

 15%|█▌        | 548/3542 [00:04<00:16, 185.36it/s]

 16%|█▋        | 580/3542 [00:04<00:15, 193.92it/s]

 17%|█▋        | 612/3542 [00:04<00:14, 203.12it/s]

 18%|█▊        | 644/3542 [00:04<00:14, 205.14it/s]

 19%|█▉        | 676/3542 [00:04<00:13, 210.39it/s]

 20%|█▉        | 708/3542 [00:05<00:13, 217.00it/s]

 21%|██        | 740/3542 [00:05<00:12, 222.86it/s]

 22%|██▏       | 772/3542 [00:05<00:12, 228.65it/s]

 23%|██▎       | 804/3542 [00:05<00:11, 234.10it/s]

 24%|██▎       | 836/3542 [00:05<00:11, 238.16it/s]

 25%|██▍       | 868/3542 [00:05<00:11, 236.76it/s]

 25%|██▌       | 900/3542 [00:05<00:10, 240.95it/s]

 26%|██▋       | 932/3542 [00:05<00:10, 244.96it/s]

 27%|██▋       | 964/3542 [00:06<00:10, 249.78it/s]

 28%|██▊       | 996/3542 [00:06<00:09, 257.60it/s]

 29%|██▉       | 1028/3542 [00:06<00:09, 255.52it/s]

 30%|██▉       | 1060/3542 [00:06<00:09, 256.75it/s]

 31%|███       | 1092/3542 [00:06<00:09, 259.42it/s]

 32%|███▏      | 1124/3542 [00:06<00:09, 263.77it/s]

 33%|███▎      | 1156/3542 [00:06<00:08, 270.32it/s]

 34%|███▎      | 1188/3542 [00:06<00:08, 267.43it/s]

 34%|███▍      | 1220/3542 [00:07<00:08, 274.86it/s]

 35%|███▌      | 1252/3542 [00:07<00:08, 278.29it/s]

 36%|███▋      | 1288/3542 [00:07<00:07, 293.28it/s]

 37%|███▋      | 1324/3542 [00:07<00:07, 306.54it/s]

 38%|███▊      | 1355/3542 [00:07<00:07, 301.84it/s]

 39%|███▉      | 1386/3542 [00:07<00:07, 295.60it/s]

 40%|███▉      | 1416/3542 [00:07<00:07, 286.25it/s]
 41%|████      | 1448/3542 [00:07<00:07, 282.87it/s]
 42%|████▏     | 1484/3542 [00:07<00:06, 298.21it/s]
 43%|████▎     | 1520/3542 [00:08<00:06, 297.00it/s]
 44%|████▍     | 1556/3542 [00:08<00:06, 303.63it/s]
 46%|████▌     | 1620/3542 [00:08<00:05, 360.27it/s]
 48%|████▊     | 1684/3542 [00:08<00:05, 348.07it/s]
 49%|████▉     | 1748/3542 [00:08<00:05, 347.18it/s]

[falexwolf]

Same in Rodeo.

I think in Jupyter it can be solved via http://stackoverflow.com/questions/42212810/tqdm-in-jupyter-notebook. In Rodeo though, one gets a Widget Javascript not detected. error, which I couldn't resolve.

[hadim]

Same problem here. Any idea @casperdcl ?

[casperdcl]

Sounds like you need to catch the keyboard interrupt and close the bar? Don't use notebooks so can't say... @lrq3000?

[shivam-kotwalia]

Instead of tqdm.tqdm use tqdm.notebook.tqdm or tqdm.auto.tqdm

from tqdm.auto import tqdm, trange
from time import sleep

for i in trange(10, desc='1st loop'):
    for j in tqdm(xrange(100), desc='2nd loop'):
        sleep(0.01)

https://github.com/tqdm/tqdm/#ipython-jupyter-integration

(edited by @casperdcl seeing as this is the top comment and people may not bother scrolling further for updates)

[casperdcl]

Ah yes ofc.

[mohitlamba2302]

Since tqdm uses a simple printing mechanism to display progress bars, you should not write any message in the terminal using print() while a progress bar is open.

[gaursurya]

Import below library instead of default one.

from tqdm import tqdm_notebook as tqdm

[chengs]

@gaursurya a better way from tqdm.auto import tqdm

[casperdcl]

auto suppresses autonotebook's warning

[munael]

tqdm_notebook doesn't solve the problem for me when the code is in a module imported and used by a notebook.

And tqdm.auto apparently doesn't exist in the latest version on pypi?

[wookayin]

Is there any way to fix this issue without using tqdm_notebook? For example, some third-party libraries (e.g. moviepy) that we don't have control about its tqdm import uses the raw tqdm instead of tqdm_notebook, unless we do a monkey-patching. After an autoreload, tqdm starts to emit newline at every iteration.

[ajaymaity]

Also, if I use tqdm_notebook instead of tqdm, tqdm.pandas() does not work. Any solution?

[morawi]

I had the same problem, but using Spyder ... not in Jupyter.
I've solved it by putting:
pbar.close(); del pbar
after the loop that has pbar.update(10)
I think this tweak will work for Jupyter as well.

[BlackArbsCEO]

@morawi can you give an example? When I tried your proposed solution it still printed on a new line each time.

def mp_main(window=None, df=None,):
    range_ = range(window, df.shape[0])
    pool = mp.Pool(10)
    res_list = []
    func = partial(mp_kca_features, df=df, t=range_, window=window)
    with tqdm(total=len(range_)) as pbar:
        for i,res in tqdm(enumerate(pool.imap_unordered(func, range_))):
            pbar.update()
            res_list.append(res)
    pool.close()
    pool.join()
    return res_list

##
result = mp_main(df=df, window=window)

[morawi]

@BlackArbsCEO
I needed to do some extra stuff to make it 100% work, here is the code:

import time
...

        print('\n Storing for ', mode  )
        time.sleep(1) 
        pbar = tqdm(total=len(self.file_label))
        time.sleep(1)
        for i in range(len(self.file_label)):
            word = self.file_label[i]  
            word_str = word[1].lower() 
            self.out_vector[i] = torch.from_numpy(self.some_function(word_str)))
            pbar.update(1)        
        pbar.close();   del pbar ''' not sure if this del pbar is necessary, but I'm keeping it'''

I needed to call several instances of the tqdm in some other functions, similar to the above, and all went fine.

[sugatoray]

Is there any way to fix this issue without using tqdm_notebook? For example, some third-party libraries (e.g. moviepy) that we don't have control about its tqdm import uses the raw tqdm instead of tqdm_notebook, unless we do a monkey-patching. After an autoreload, tqdm starts to emit newline at every iteration.

Try using tnrange instead (as mentioned by @shivam-kotwalia in this thread)

from tqdm import tnrange
import time

print('\n\nExample: tnrange')
for i, first, last in zip(tnrange(6, desc='Main Loop'), first_name, last_name):
  time.sleep(0.3)
  #pass

If you would like to see examples on tqdm, tnrange and tqdm_notebook checkout this jupyter notebook on GitHub.

Refer to: https://pypi.python.org/pypi/tqdm#ipython-jupyter-integration

[casperdcl]

if you don't know about the frontend, use from tqdm.auto import tqdm, trange

[RoohollahH]

from tqdm import tqdm_notebook as tqdm
import time
for i1 in tqdm(range(5)):
for i2 in tqdm(range(300)):
# do something, e.g. sleep
time.sleep(0.01)

[kim-sunghoon]

when using the shell instead of jupyter notebook, same newline error happened.... would you like help me?

[casperdcl]

looks like your terminal has broken widechars support. try using ascii=True

[kriventsov]

For me tqdm_notebook does not work.
The following is hacky, but seems to work reasonably well to reset tqdm:

from tqdm import tqdm as tqdm_base
def tqdm(*args, **kwargs):
    if hasattr(tqdm_base, '_instances'):
        for instance in list(tqdm_base._instances):
            tqdm_base._decr_instances(instance)
    return tqdm_base(*args, **kwargs)

Sometimes previous output is printed at the start (which I am not sure how to remove), but I find it much less annoying than newlines (especially in long loops).

[rjurney]

This happens for me with the Jupyter client whether I’ve stopped it or not.

[chbrown]

Still an issue with tqdm==4.41.1 ... so I've combined the workarounds/hacks described above (❤️ the community support here 😜).
Assuming usage like:

from tqdm import tqdm
list(tqdm(bad_iterator))  # where bad_iterator raises an exception at some point

You can ensure tqdm is reset and properly reinitialized like so:

from tqdm import tqdm
getattr(tqdm, '_instances', {}).clear()  # ⬅ add this line
list(tqdm(bad_iterator))

The additional getattr(...).clear() call won't raise an exception on the first pass (as @rjurney lamented), but still fix the problem (as @valgur described and @roastbeeef seconded) on subsequent executions.

[emigre459]

@chbrown Awesome solution, I threw that right up in my module after from tqdm import tqdm and worked flawlessly!

[Evidlo]

If you're using ipython, you can use a startup script to clear tqdm instances after each command execution:

~/.ipython/profile_default/startup/tqdm.py

from IPython import get_ipython

def tqdm_clear(*args, **kwargs):
    from tqdm import tqdm
    getattr(tqdm, '_instances', {}).clear()

get_ipython().events.register('post_execute', tqdm_clear)

[Santosh-16k]

Faced the same issue, got it resolved by importing the following in my jupyter notebook:
from tqdm.notebook import tqdm

[casperdcl]

@Santosh-16k from tqdm.auto import tqdm is more portable

[KranthiKumar28]

Use tqdm_notebook

from tqdm import tqdm_notebook as tqdm

x=[1,2,3,4,5]

for i in tqdm(len(x)):

print(x[i])

[casperdcl]

There are 2 different issues:

1. some environments not supporting text-based progress bars well
   • solution: use tqdm.auto or tqdm.notebook if possible (following https://github.com/tqdm/tqdm/#ipython-jupyter-integration and https://ipywidgets.readthedocs.io/en/stable/user_install.html, i.e. pip install ipywidgets && jupyter nbextension enable --py widgetsnbextension && jupyter labextension install @jupyter-widgets/jupyterlab-manager), or
   • open an issue with your environment maintainers
2. incorrectly used/unterminated/non-close()d bars exception and thread safety #548
   • should be (mostly) fixed in tqdm>=4.45.0 (Exceptions #926 -> next release v4.45.0 #929)

[hzitoun]

Instead of tqdm use tqdm_notebook

from tqdm import tnrange, tqdm_notebook
from time import sleep

for i in tnrange(10, desc='1st loop'):
    for j in tqdm_notebook(xrange(100), desc='2nd loop'):
        sleep(0.01)

https://pypi.python.org/pypi/tqdm#ipython-jupyter-integration

Now you have to use from tqdm.notebook import tqdm

[casperdcl]

Yes updated #375 (comment) which will hopefully stop people constantly posting this :)

@hadim and everyone else I think #548 is mostly fixed in tqdm>=4.45.0 (regarding no longer having to del/clear instances/reimport).

[rkadam]

Since I suffered for last few hours, thought it's worth while to make a note here how I fixed the issue.

I'm using tqdm = 4.45.0 so this is issue still happening. @casperdcl #375 (comment) didn't help either. What helped me is this link: https://towardsdatascience.com/progress-bars-in-python-4b44e8a4c482 in which Author suggests installing

pip install ipywidgets
jupyter nbextension enable --py widgetsnbextension
jupyter labextension install @jupyter-widgets/jupyterlab-manager

After that I was able to run this nested loop without any problem.

from tqdm.auto import tqdm, trange
from time import sleep

for i in trange(10, desc='1st loop'):
    for j in trange(100, desc = "2nd loop"):
        sleep(0.01)

Hoping this will help for folks who are still struggling. Thank you for such awesome module!
Raju

Development Environment: Mac OS Catalina (10.15.4), Python 3.8.1, Tqdm 4.45.0, jupterlab 2.1.1

[casperdcl]

Just updated #375 (comment) further. Going to close this; please re-open if there are any other issues.

[mylyu]

There are 2 different issues:

1. some environments not supporting text-based progress bars well

   • solution: use tqdm.auto or tqdm.notebook if possible (following https://github.com/tqdm/tqdm/#ipython-jupyter-integration and https://ipywidgets.readthedocs.io/en/stable/user_install.html, i.e. pip install ipywidgets && jupyter nbextension enable --py widgetsnbextension && jupyter labextension install @jupyter-widgets/jupyterlab-manager), or
   • open an issue with your environment maintainers

2. incorrectly used/unterminated/non-close()d bars exception and thread safety #548

   • should be (mostly) fixed in tqdm>=4.45.0 (Exceptions #926 -> next release v4.45.0 #929)

Oh my, thanks. finally it is fixed.

[seyeeet]

I am writing the output to a txt files and this problem still happens there and none of the above suggestions work for that case :|

[casperdcl]

@seyeeet that's #506 (comment)

[FlorinAndrei]

For future reference, in addition to everything else said above, it is worth checking whether it's other software components that cause the issue. In my case, I've spent a frustrating couple hours yesterday trying to fix something that looked just like this issue, only to realize today that it was actually a VSCode bug.

microsoft/vscode#184657

I should have tested it in a classic Jupyter Notebook + browser setup before going down the rabbit hole of finding a problem with TQDM when the problem was somewhere else.

It got fixed today after VSCode upgraded itself to 1.79.2