Dashboard progressbar improvements

[harmenwassenaar]

No description provided.

[harmenwassenaar]

Also, if you have any suggestions how to add another package-name in this line
https://github.com/sybrenjansen/mpire/blob/master/mpire/dashboard/utils.py#L69
that would be very welcome. Because We're wrapping our own small framework around mpire, and it would be great if we could filter it out and show the line on the stack before it.

[sybrenjansen]

Also, if you have any suggestions how to add another package-name in this line https://github.com/sybrenjansen/mpire/blob/master/mpire/dashboard/utils.py#L69 that would be very welcome. Because We're wrapping our own small framework around mpire, and it would be great if we could filter it out and show the line on the stack before it.

I see what you mean.

If you have a code example that I can tinker with that would be great, then I can have a look. Please add the expected output and the output you're currently getting

[harmenwassenaar]

Also, if you have any suggestions how to add another package-name in this line https://github.com/sybrenjansen/mpire/blob/master/mpire/dashboard/utils.py#L69 that would be very welcome. Because We're wrapping our own small framework around mpire, and it would be great if we could filter it out and show the line on the stack before it.

I see what you mean.

If you have a code example that I can tinker with that would be great, then I can have a look. Please add the expected output and the output you're currently getting

Here's a simplified version of what I'm using:

executor.py

import inspect
from mpire import WorkerPool
from mpire import WorkerPool
from mpire.exception import remove_highlighting

class MpireExecutor:
    def __init__(self, n_jobs: int = 1, **kwargs):
        mpire_pool_args = set(inspect.signature(WorkerPool).parameters.keys())
        mpire_imap_args = set(list(inspect.signature(WorkerPool.imap).parameters.keys())[3:])

        self.pool_args = {'n_jobs': n_jobs, **{k: v for k, v in kwargs.items() if k in mpire_pool_args}}
        self.imap_args = {'chunk_size': 1, **{k: v for k, v in kwargs.items() if k in mpire_imap_args}}

    def __call__(self, func, data):
        with WorkerPool(**self.pool_args) as pool:
            try:
                yield from pool.imap_unordered(func, [(p,) for p in data], **self.imap_args)
            except Exception as e:
                # strip highlighting so it's readable when logged to file
                if getattr(e, '__cause__', None):
                    e.__cause__ = e.__cause__.__class__(remove_highlighting(str(e.__cause__)))
                raise

And then if we run

from executor import MpireExecutor
from mpire.dashboard import start_dashboard
start_dashboard()
mpe = MpireExecutor(n_jobs=1, progress_bar=True)
res = list(mpe(lambda x: x**2, range(10)))

The task details on the dashboard show

Function: <lambda>, on line 1 of /home/harmen.local/TMP/<ipython-input-3-b3ee4196616e>

Invoked on line 17 of /home/harmen.local/TMP/executor.py, through yield from pool.imap_unordered(func, [(p,) for p in data], **self.imap_args)

The Invoked on line is the same for anything you execute, so it doesn't help narrowing down what code is actually running.
If I change line 69 mpire/dashboard/utils.py to also exclude executor, then it works, but monkey-patching the function or hacking the file in the venv is not really ideal.

[sybrenjansen]

Also, if you have any suggestions how to add another package-name in this line
https://github.com/sybrenjansen/mpire/blob/master/mpire/dashboard/utils.py#L69
that would be very welcome. Because We're wrapping our own small framework around mpire, and it would be great if we could filter it out and show the line on the stack before it.

I implemented a similar system to what the warnings module of Python does with the stacklevel. The code snippet below shows what's possible:

from mpire.dashboard import get_stacklevel, set_stacklevel, start_dashboard

print(get_stacklevel())  # defaults to 1
set_stacklevel(2)

start_dashboard()

mpe = MpireExecutor(n_jobs=1, progress_bar=True)
res = list(mpe(lambda x: x**2, range(10)))

On the dashboard, this now shows:

Function: <lambda>, on line 1 of /Users/sybren/repos/mpire/<ipython-input-9-b3ee4196616e>

Invoked on line 1 of <ipython-input-9-b3ee4196616e>, through res = list(mpe(lambda x: x**2, range(10)))

Does this suffice for your usecase?

[harmenwassenaar]

Does this suffice for your usecase?

Yes, I think so. Thanks!