Progressbars always on newline

[GilJ]

When running multiple progress bars from parallel processes, they print progress on new lines, with a random amount of new lines in between. I've thought to notice that when a single worked is updating his progressbar a few times before the others processors do, that it's own bar gets updated, so this seems to be an issue with multiprocessing support. I also have the issue with the multiprocessing example from https://github.com/tqdm/tqdm#nested-progress-bars .

I don't think this is a duplicate of other issues, as the ones I found are set in a Notebook.

Sample output:

Worker 8: 0it [00:00, ?it/s]
Worker 2: 0it [00:00, ?it/s]




Worker 6: 0it [00:00, ?it/s]
Worker 2: 1it [00:00,  4.37it/s]



Worker 1: 0it [00:00, ?it/s]






Worker 8: 1it [00:00,  2.36it/s]

Worker 3: 0it [00:00, ?it/s]
[...]

My versions are:

• tqdm: 4.23.4
• Python: 3.6.5
• Platform: Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] win32

My code is set up as:
0. The freeze_support() method from the multiprocessing module is called

1. The main process creates a set of workers, each is assigned an index. Workers are a subclass of multiprocessing.Process. They are also passed a lock for the tqdm.set_lock() call.
2. The .start() method of the workers is called, they all initialize their progress bar, and use their index for position argument of tqdm.
3. The workers accept work in a while True loop, from a thread safe Queue. After each task is done, they .update() their progress bar.

If any other information is required, I'd gladly provide it.

[chengs]

can you give a small code to reproduce this?

[casperdcl]

might be the environment not supporting nested bars. Do nested bars work without multiprocessing?

[emgre]

I get a similar issue with tqdm 4.27.0, Python 3.7.0 on 64-bit Windows 10. The code is executed from PowerShell in VS Code, but I also tested it in cmd.exe and in PowerShell standalone (not in VSCode I mean) and I get the same result.

The following program (taken from here) produces the issue.

from multiprocessing.pool import ThreadPool
import time
import threading
from tqdm import tqdm


def demo(lock, position, total):
    text = "progresser #{}".format(position)
    with lock:
        progress = tqdm(
            total=total,
            position=position,
            desc=text,
        )
    for _ in range(0, total, 5):
        with lock:
            progress.update(5)
        time.sleep(0.1)
    with lock:
        progress.close()


pool = ThreadPool(5)
tasks = range(50)
lock = threading.Lock()
for i, url in enumerate(tasks, 1):
    pool.apply_async(demo, args=(lock, i, 100))
pool.close()
pool.join()

I'm running the code through pipenv shell, but I don't think it's really important to know that.

[casperdcl]

@emgre Can you substitute threading.Lock() -> multiprocessing.RLock()

[emgre]

@casperdcl I can, but the issue persists.

[emgre]

I also tried the example given in the README, and it produces the same issue.

from time import sleep
from tqdm import trange, tqdm
from multiprocessing import Pool, freeze_support, RLock

L = list(range(9))

def progresser(n):
    interval = 0.001 / (n + 2)
    total = 5000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    for i in trange(total, desc=text, position=n):
        sleep(interval)

if __name__ == '__main__':
    freeze_support()  # for Windows support
    p = Pool(len(L),
             # again, for Windows support
             initializer=tqdm.set_lock, initargs=(RLock(),))
    p.map(progresser, L)
    print("\n" * (len(L) - 2))

[alessio-greco]

This same issue is produced when you interrupt an executing script that contains tqdm on Spyder.
Precisely:

1. Write any code (section) using tqdm on Spyder
2. Execute the code
3. Stop the execution of the code section while it is running
4. After that, execute it another time. Any progress will be printed on a new line.

[casperdcl]

@Arakkun that's #548

[ghost]

I copied the various snippets in #407 and I'm getting the same result as well

macOS High Sierra (Pycharm)
Python 3.7
tqdm 4.28.1

Edit:
works fine when launching from terminal.... any ideas?

[robertschulze]

I can confirm the behaviour for Win 7 + PyDev (Eclipse Luna) + Anaconda 3.6 x64

[jhmenke]

Still happens (Linux + Pycharm). sleep is the problem, otherwise it works fine

    for _ in trange(3600, desc="Waiting"):
        sleep(1)

[casperdcl]

pycharm? might be #203 and https://github.com/tqdm/tqdm/#faq-and-known-issues

[jhmenke]

In my example it is not nested, but this seems to be the issue (https://youtrack.jetbrains.com/issue/PY-33061). alright, thanks!

[raff7]

interestingly, i get the same problem with a simple loop over a list of json files:
for nc,j in tqdm.tqdm(enumerate(jf)): stuff

but only if i run it in iterm2 on mac with zsh, does not happen with simple bash and the default terminal...
Weird

[casperdcl]

@raff7 you should see https://github.com/tqdm/tqdm#faq-and-known-issues if you're enumerating... essentially use tqdm.contrib.tenumerate

[gordonwatts]

Is this issue here related to this stackoverflow question I asked?

[HassanNaseri]

Same issue here, when trying to update the progress bar inside a parallel process. Updating the bar after completion of the process is not an option for me, as I need finer updates within the process. Is there any way to get this functionality without issues?

from tqdm import tqdm
from time import sleep
import multiprocessing as mp

def my_proc(pid):
    my_list = list(range(100))
    with tqdm(total=len(my_list), desc=f'Process {pid}', position=pid+1) as pbar:
        for x in my_list:
           sleep(0.1)
           pbar.update(1)
    return 1

if __name__ == '__main__':
    pool_size = 4
    pool = mp.Pool(pool_size)
    resultsAsync = [pool.apply_async(my_proc, args=(i,)) for i in range(pool_size)]        
    
    for resAsync in resultsAsync:
        res_batch = resAsync.get() 
        
    pool.close()

[mjkvaak]

The issue of tracking multiple sub-processes has been solved very elegentaly here. In particular, @HassanNaseri example would read:

from time import sleep
import multiprocessing as mp
from atpbar import atpbar, register_reporter, find_reporter, flush

def my_proc(pid):
    my_list = list(range(100))
    for _ in atpbar(my_list, name='inner {}'.format(pid)):
        sleep(0.1)
    return 1

if __name__ == '__main__':
    pool_size = 4
    reporter = find_reporter()
    with mp.Pool(pool_size, register_reporter, [reporter]) as p:
        ret = p.map(my_proc, range(pool_size))
        flush()

Please do not take this cross-referencing to another repo as an offense. I would have rather posted my answer in a StackExchange thread but I couldn't find one on this topic.