Threading | Multiprocessing Visual Error

[trifwn]

• I have marked all applicable categories:
  • exception-raising bug
  • [ +] visual output bug
• [+ ] I have visited the source website, and in particular
  read the known issues
• [+] I have searched through the issue tracker for duplicates
• [+] I have mentioned version numbers, operating system and
  environment, where applicable:

  import tqdm, sys
  print(tqdm.__version__, sys.version, sys.platform)
  4.65.0 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:08:06) [GCC 11.3.0] linux

I have stumbled upon a case where tqdm bars need to be updated manually. Specifically I am trying to track a process that is launched by Fortran library. In the code below I have reproduced the behavior sparing the case-oriented details. The problem arises when a process is completed. Specifically when a process finished, the cursor position changes and therefore the bars move relative to their initial position leaving behind the previous iteration. Also, it happens, that if a bar tries to update itself, when another bar is changing, its position is recalculated from there leaving a mess. I believe this behavior has to do with the absence of lock, or a way to lock the changes in the bars. I haven't found anything on the repository so I am filing this issue! It should be possible to solve the issue with queues but i would prefer not to pass tqdm objects around since they are unpickable.

from concurrent.futures import ThreadPoolExecutor, as_completed
import random
from tqdm.auto import tqdm
from time import sleep 
from threading import Thread

def latest_time(dir:str):
    return random.randint(0,100), 'a', False
    
def monitor_run(
        dir:str,
        name: str,
        position: int,
        max_iter: int,
        refresh_progress: float=2,
) -> None:
    sleep(1 + (position+1)/10)
    with tqdm(
        total = max_iter,
        bar_format="{l_bar}{bar:30}{r_bar}",
        desc=f"\t\t{name}: 0.0 Progress",
        position= position,
        ncols = 100,
        leave = True,
        ascii= True,
        colour ='#00ff00',

    ) as pbar:
        desc_prev: float = 0
        while True:
            sleep(refresh_progress)
            time, desc, error = latest_time(dir)
            
            if desc is None:
                desc: float | None = desc_prev
            else:
                desc_prev = desc
                pbar.desc = f"\t\t {name}: {desc} Progress"
            
            if error:
                pbar.write(f"Encountered Error at {desc}")
                break
            
            if time is None:
                continue

            pbar.n = int(time)
            pbar.refresh(nolock=True)

            if time>=max_iter:
                pbar.close()
                break

def serial_monitor(
    dir:str,
    position: int,
    max_iter: int,
    refresh_progress: float = 2,
)-> None:
    monitor_run(dir,str(position), position ,max_iter, refresh_progress)


def serial_monitor_star(args)-> None:
    serial_monitor(*args)

def parallel_monitor(
        dirs: list[str],
        max_iter: int,
        refresh_progress: float =2,
) -> None:
    args_list = [
        [
            dir, position+1, max_iter, refresh_progress
        ] for position, dir in enumerate(dirs)
    ]
    # tqdm.write("\t\tStarting:")
    # thread_map(
    #     serial_monitor_star, args_list, tqdm_class = tqdm ,max_workers = len(reynolds)
    # )
    # tqdm.write("\t\tCompleted")
    with tqdm(total=2*len(dirs)):
        with ThreadPoolExecutor(max_workers= len(dirs)) as ex:
            futures = [
                ex.submit(
                    serial_monitor_star,
                    args
                ) for args in args_list
            ]
            for future in as_completed(futures):
                result = future.result()


if __name__ =="__main__":
    parallel_monitor(
        ['','','','','','',''],
        100,
        2
    )

[mjpieters]

I strongly suspect that this is the same issue as #1496. Could you try:

pip install git+https://github.com/tqdm/tqdm.git@refs/pull/1502/merge

locally and see if that fixes your test output? That line tells pip to install the code from my PR as the tqdm project.

[zfb132]

See here for a demo using multiprocessing.Manager to sync the progress bars, maybe you can try it
#1000 (comment)