Multiprocessing and tqdm in jupyter notebooks.

[vyasr]

I'm trying to use tqdm along with multiprocessing.Pool in a notebook, and it doesn't quite seem to render correctly. The general problem appears to be well documented in Issue #407 and Issue #329, but neither of the fixes appear to have percolated to the notebook code. In particular, the "canonical example" in Issue #407 works for me on the command line, but when I move to a jupyter notebook and replace from tqdm import tqdm with from tqdm import tqdm_notebook as tqdm I get something like the following.

Changing the number of workers in the Pool yields different results, but the results are consistent (e.g. running with Pool(4) always shows progressers 1, 6, and 7).

During times when no progress bar is updating, I see the following error message repeatedly appearing on the terminal where the jupyter notebook is running.

[IPKernelApp] WARNING | WARNING: attempted to send message from fork
{'metadata': {}, 'msg_id': 'b58be6b1-80db-4a2d-9699-b3a4f3ddae8e', 'header': {'username': 'vramasub', 'version': '5.0', 'session': '667381c2-d872-4fc2-b2b5-06e33
957b17bb', 'msg_id': 'b58be6b1-80db-4a2d-9699-b3a4f3ddae8e', 'msg_type': 'comm_msg', 'date': datetime.datetime(2017, 12, 6, 15, 44, 13, 453842)}, 'content': {'dd
ata': {'state': {'value': '100% 5000/5000 [00:05<00:00, 877.55it/s]'}, 'buffers': [], 'method': 'update'}, 'comm_id': '486f6112cd1c496b95c4b25670f73786'}, 'mm
sg_type': 'comm_msg', 'parent_header': {'username': 'username', 'msg_id': 'FF1406FB5DC044BC98FFEDFF244B9E13', 'session': 'E79AAA8C69DA4475BAACFAD1D0EC9DAF', 'vee
rsion': '5.0', 'msg_type': 'execute_request', 'date': datetime.datetime(2017, 12, 6, 15, 43, 39, 135203)}}

For reference, this is the same issue noted here, a different project that is making use of tqdm internally. The terminal printing stops when progressbars are successfully printing in the notebook.

I have tested with both tqdm version 4.19.4 (the current version on pip) and the current master (installed using pip install -e git+https://github.com/tqdm/tqdm.git@master#egg=tqdm). I have tested on both Linux (4.9.34-gentoo) and OS X (High Sierra 10.13.1). My jupyter version is 4.4.0 both on Linux and on OS X.

[lrq3000]

I'm not sure what the culprit is but parallel bars are quite tricky. On Linux, it is usually transparent because tqdm can provide a lock by default, but that's not the case on Windows, the user must define one in the parent app and then provide it to tqdm. In your case, using Jupyter instead of the Python interpreter, maybe the "default lock" does not exist.

What you can try is to use a lock you define in your Jupyter cell, and provide it to tqdm, if this fixes the issue then this would confirm my hypothesis above.

[wmayner]

Can confirm that the same problem arises when explicitly providing a lock, using the Windows-supported code from tqmd/examples/parallel_bars.py:

from __future__ import print_function
from time import sleep
from tqdm import tqdm_notebook, trange
from multiprocessing import Pool, freeze_support, RLock


L = list(range(9))


def progresser(n):
    interval = 0.001 / (len(L) - n + 2)
    total = 5000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    # NB: ensure position>0 to prevent printing '\n' on completion.
    # `tqdm` can't autmoate this since this thread
    # may not know about other bars in other threads #477.
    for _ in tqdm_notebook(range(total), desc=text, position=n + 1):
        sleep(interval)


if __name__ == '__main__':
    freeze_support()  # for Windows support
    p = Pool(len(L),
             initializer=tqdm.set_lock,
             initargs=(RLock(),))  # Provide a lock explicitly
    p.map(progresser, L)
    print('\n' * len(L))

[casperdcl]

Is this related to https://stackoverflow.com/questions/37103243/multiprocessing-pool-in-jupyter-notebook-works-on-linux-but-not-windows ?

[wmayner]

I don't think so; I ran the above code on Linux (in a Jupyter notebook).

[nalepae]

I found a strange hack to workaround your issue:
Just add a print statement in the progresser function.

Example here:

from time import sleep
from tqdm import tqdm_notebook as tqdm
from multiprocessing import Pool, freeze_support

def progresser(n):
    # This line is the strange hack
    print(' ', end='', flush=True)

    text = "progresser #{}".format(n)
    for i in tqdm(range(5000), desc=text, position=n):
         sleep(0.001)
        
if __name__ == '__main__':
    freeze_support()
    L = list(range(10))
    print()
    Pool(2).map(progresser, L)

Actually, you can put any print you want, but you have to print something:
print() or print('') won't work.

The line I added is the line I found which works and which modifies as few as possible the output.

And here is my output:

My configuration:
Linux Ubuntu 16.04, running on Chrome.

[codejedi365]

I can't believe that hack works but it does for jupyter! Wish I had found this thread 6 hours ago!
Thanks @nalepae !

[nunesgh]

I'm sorry if I'm off topic here, but is it possible to have just one progress bar that accounts for the execution of all of the workers?

For example, my pool would have more than 65000 workers and I would like to have my progress bar updated every time a worker finishes executing. Unfortunately, each worker creates its own progress bar on top of the other and what I see is a progress bar stuck on 1/65000.

[casperdcl]

btw check out tqdm.contrib.concurrent (process_map and thread_map)

[imneonizer]

I'm sorry if I'm off topic here, but is it possible to have just one progress bar that accounts for the execution of all of the workers?

For example, my pool would have more than 65000 workers and I would like to have my progress bar updated every time a worker finishes executing. Unfortunately, each worker creates its own progress bar on top of the other and what I see is a progress bar stuck on 1/65000.

Use interprocess communication to collect progress from all process and show i in the main progressbar.
I achieved this using redis, in each process i create a hash of input data and send it to redis,
on progressbar side i get all the progress by calculating same hash for input data and collecting their current status and summing them up to form final progress status.

[Berowne]

I found a strange hack to workaround your issue:
Just add a print statement in the progresser function.

...

def progresser(n):
# This line is the strange hack
print(' ', end='', flush=True)
....

Excellent hack thanks!

[chengs]

Seems position is ignored now.
#1133