TypeError at tqdm_utils

[pemm8]

Hi there,

I'm getting an issue when setting progress_bar = True.

Traceback:

Traceback (most recent call last):
  File "/Users/user/Projects/project/hk_analysis/matching.py", line 74, in <module>
    process_matches_multi()
  File "/Users/user/Projects/project/hk_analysis/matching.py", line 49, in process_matches_multi
    matching_results = pool.map(atomic_matching, [{'item': item} for item in updated[:10]], progress_bar=True)
  File "/Users/user/.virtualenvs/pdf_analysis/lib/python3.9/site-packages/mpire/pool.py", line 265, in map
    results = self.map_unordered(func, ((args_idx, args) for args_idx, args in enumerate(iterable_of_args)),
  File "/Users/user/.virtualenvs/pdf_analysis/lib/python3.9/site-packages/mpire/pool.py", line 324, in map_unordered
    return list(self.imap_unordered(func, iterable_of_args, iterable_len, max_tasks_active, chunk_size,
  File "/Users/user/.virtualenvs/pdf_analysis/lib/python3.9/site-packages/mpire/pool.py", line 490, in imap_unordered
    tqdm_manager_owner = TqdmManager.start_manager() if progress_bar else False
  File "/Users/user/.virtualenvs/pdf_analysis/lib/python3.9/site-packages/mpire/tqdm_utils.py", line 121, in start_manager
    cls.MANAGER_HOST.value = cls.MANAGER.address[1:]
  File "<string>", line 11, in setvalue
TypeError: bytes expected instead of str instance

Snapshot of code:

from mpire import WorkerPool
from mpire.dashboard import connect_to_dashboard

connect_to_dashboard(8080)

def atomic_matching(shared_objects, item):
    ### ... match processing
    return data

def process_matches_multi():
    ### ... get data
    ### ... set choices and previous

    with WorkerPool(n_jobs=5) as pool:
        pool.set_shared_objects((choices, previous))
        matching_results = pool.map(atomic_matching, [{'item': item} for item in updated])
    
    ### ... output results

If I change tqdm_utils as follows (line 121), it disappears but no progress bar appears and the dashboard does not connect

cls.MANAGER_HOST.value = cls.MANAGER.address[1:].encode()

I've checked tqdm and it is working in isolation.

Versions are:
mpire==2.3.0
tqdm==4.62.1 (Also tried tqdm==4.62.3)
Python 3.9.7

Love the project and thanks for all the hard work :)

Many thanks,
Paul

[sybrenjansen]

Hmm, that's weird. I tested it just now on a fresh venv with the same versions you're using and it doesn't produce an error on my end.

That said, I can try to make this part more robust by not checking the Python version, but simply by using try/except clauses and see what works. I will start working on it probably next week.

For now, as a workaround, you can simply comment the following in mpire/tqdm_utils.py:

        # cls.MANAGER_STARTED.set()

        # Set host so other processes know where to connect to. On Linux, since Python 3.9, address is a bytes object
        # and is prefixed by a null byte which needs to be removed (null byte doesn't work with Array). Before 3.9, this
        # was a string, which we need to transform to bytes
        # if RUNNING_WINDOWS or (sys.version_info[0] == 3 and sys.version_info[1] < 9):
        #     cls.MANAGER_HOST.value = cls.MANAGER.address.encode()
        # else:
        #     cls.MANAGER_HOST.value = cls.MANAGER.address[1:]

and you should be good. This part is only used when using start methods other than fork.

[pemm8]

Thank you @sybrenjansen - the workaround worked fine!

[vlomshakov]

Hey! I've faced the problem too under Mac OS with python version 3.9.7.
With another setups (windows and ubuntu with python 3.7.6) everything works perfect.

[sybrenjansen]

Ok, thnx for letting me know. I'm a bit busy at the moment, but I'll get it to it soon

[sybrenjansen]

@pemm8 , @vlomshakov can anyone of you check if the linked PR fixes your issues?

[vlomshakov]

@sybrenjansen thanks! Everything works now (with 2.3.1 version).