The canonical multiprocessing example displays only a single bar

[mojones]

I'm trying to use tqdm to monitor multiple FTP downloads using ftplib. Browsing around the documentation led me to this "canonical" example:

from time import sleep
from tqdm import tqdm
from multiprocessing import Pool, freeze_support

def progresser(n):
    text = "progresser #{}".format(n)
    for i in tqdm(range(5000), desc=text, position=n):
        sleep(0.001)

if __name__ == '__main__':
    freeze_support()  # for Windows support
    L = list(range(10))
    Pool(len(L)).map(progresser, L)

but on my system (Ubuntu 14.04, Python 3.4.3, tqdm 4.14.0) I get a mixture of empty lines, non-updating progress bars, and a single updating progress bar at the bottom:

If I watch the bottom progress bar carefully I can see that the name occasionally changes, so I think that the progress bars are being drawn on top of one another. I've searched the issues and the closest thing I can find is this:

#285

but I don't think it's the same problem, as that issue reported multiple updating progress bars visible and I am only seeing one (the screenshots are very different). Let me know if I can provide any other info - I really want to use tqdm for this project but am out of depth when it comes to trying to figure out what's wrong.

[casperdcl]

This is annoying. It works fine on py27... There is indeed something weird with py3. Have you seen #329?

[mojones]

I did have a look at #329, but quickly reached the limit of my understanding when looking at the code examples :-). Entertainingly, when I run the canonical code under Python 2 on my system, I get a different result, though still not quite what is expected:

Which seems to me like the progress bars are still overwriting each other, just in a different pattern.

[NoAnyLove]

The initialization of tdqm, update method and close method are not thread-safe. I searchd the issue, and found that pull#291 is trying to solve the parallelism output, but that code is not in current release, nor in master branch. You can use your own synchronization mechanism to solve this problem.

Besides, the progressbar at position 0 will have an extra line return appended when close it, as @lrq3000 mentioned in issue #329. An easy workaround is, starting from 1. I think it would be better to have an optinal parameter to determine this behavior.

A sample code

from multiprocessing.pool import ThreadPool
import time
import threading
from tqdm import tqdm


def demo(lock, position, total):
    text = "progresser #{}".format(position)
    with lock:
        progress = tqdm(
            total=total,
            position=position,
            desc=text,
        )
    for _ in range(0, total, 5):
        with lock:
            progress.update(5)
        time.sleep(0.1)
    with lock:
        progress.close()


pool = ThreadPool(5)
tasks = range(50)
lock = threading.Lock()
for i, url in enumerate(tasks, 1):
    pool.apply_async(demo, args=(lock, i, 100))
pool.close()
pool.join()

[tomplex]

@NoAnyLove, I've been trying to sort this issue out for a couple of days. None of the stackexchange questions I found were helpful at all..this is PERFECT and exactly my use case. Thank you for posting this!!

[casperdcl]

Thanks for this. I recall writing pretty much exactly this snippet ages ago but it must be lost in another issue somewhere.

[casperdcl]

Ah yes of course good ol' #329

[casperdcl]

Could you check out the updated branch at #329, and verify it works with:

from multiprocessing.pool import ThreadPool
import time
import threading
from tqdm import tqdm


def demo(position, total):
    text = "progresser #{}".format(position)
    progress = tqdm(
        total=total,
        position=position,
        desc=text,
    )
    for _ in range(0, total, 5):
        progress.update(5)
        time.sleep(0.1)
    progress.close()


pool = ThreadPool(5)
tasks = range(50)
for i, url in enumerate(tasks, 1):
    pool.apply_async(demo, args=(i, 100))
pool.close()
pool.join()

or even the original (canonical) example:

from time import sleep
from tqdm import tqdm
from multiprocessing import Pool, freeze_support

def progresser(n):
    text = "progresser #{}".format(n)
    for i in tqdm(range(5000), desc=text, position=n):
        sleep(0.001)

if __name__ == '__main__':
    freeze_support()  # for Windows support
    L = list(range(10))
    Pool(len(L)).map(progresser, L)

[Guo-Zhang]

It still does not work on my Mac. The result is as follows:

[decewei]

multiprocessing library updates Pool to class, I used this, but there's still bug:

from multiprocessing import Pool, Lock
import time
from tqdm import tqdm
lock = Lock()

def demo(position):
    text = "progresser #{}".format(position)
    total = 100
    with lock:
        progress = tqdm(total=total, position=position, desc=text)
    for _ in range(0, total, 5):
        with lock:
            progress.update(5)
        time.sleep(0.1)
    with lock:
        progress.close()

args = []
for i in range(50):
    args.append((i,))

with Pool(processes=5) as pool:
    pool.starmap(demo, args)

[pronkinnikita]

from fastprogress import progress_bar
from multiprocessing import Pool

def process(n: int) -> int:
    return n ** 2

numbers = [1, 2, 3, 4]

with Pool(3) as p:
    processed_numbers = list(tqdm(p.imap(process, numbers), total=len(numbers))

[casperdcl]

@pronkinnikita are you saying that if you replace from fastprogress import progress_bar with from tqdm.auto import tqdm as progress_bar your code stops working properly?

[pronkinnikita]

@pronkinnikita are you saying that if you replace from fastprogress import progress_bar with from tqdm.auto import tqdm as progress_bar your code stops working properly?

Sorry, updated comment, now it seems fine. Nope, i think is works the same way with vanila tqdm.

[Vampire-Vx]

still have bug on canonical multiprocessing example...

[casperdcl]

are you sure the current version of https://github.com/tqdm/tqdm#nested-progress-bars doesn't run out-of-the box for you?

[aktiver]

@NoAnyLove great solution! how does one add a custom function to executed while TQDM tracks the functions execution process?