Progress bar tqdm wrapper, and manual control #105

tlambert03 · 2021-01-17T03:30:35Z

this PR adds a basic tqdm_mgui iterator that updates a progress bar (from #104), with the internal state of the tqdm iterator. allowing iterator-based progress bars ( a step towards #25)

for i in tqdm_mgui(range(100)):
    time.sleep(0.1)

tqdm.mov

codecov · 2021-01-17T03:32:10Z

Codecov Report

Merging #105 (79093ac) into master (8855f45) will increase coverage by 0.26%.
The diff coverage is 94.04%.

@@            Coverage Diff             @@
##           master     #105      +/-   ##
==========================================
+ Coverage   90.51%   90.78%   +0.26%     
==========================================
  Files          25       26       +1     
  Lines        2435     2517      +82     
==========================================
+ Hits         2204     2285      +81     
- Misses        231      232       +1

Impacted Files	Coverage Δ
magicgui/widgets/_concrete.py	`82.15% <70.00%> (-0.47%)`	⬇️
magicgui/tqdm.py	`96.96% <96.96%> (ø)`
magicgui/type_map.py	`89.67% <100.00%> (+0.13%)`	⬆️
magicgui/types.py	`100.00% <100.00%> (ø)`
magicgui/widgets/_bases/value_widget.py	`97.82% <100.00%> (+0.04%)`	⬆️
magicgui/widgets/_function_gui.py	`96.59% <100.00%> (+0.04%)`	⬆️
magicgui/backends/_qtpy/widgets.py	`85.19% <0.00%> (-0.26%)`	⬇️
magicgui/widgets/_bases/ranged_widget.py	`84.26% <0.00%> (+2.24%)`	⬆️
magicgui/backends/_qtpy/application.py	`97.22% <0.00%> (+8.33%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8855f45...709f07a. Read the comment docs.

tlambert03 · 2021-01-17T16:02:56Z

this is getting kinda cool!
If you use tqdm_mgui (or the convenience tmgrange shortcut) inside of a @magicgui-decorated function, it will add the progressbar to the gui. It also supports nested progressbars. For instance:

import random
from time import sleep
from magicgui import magicgui
from magicgui.tqdm import tmgrange, tqdm_mgui

@magicgui(call_button=True, layout="vertical")
def long_function(
    steps=20, repeats=4, choices="ABCDEFGHIJKLMNOP12345679", char="", delay=0.1
):
    # using the `tmgrange` shortcut instead of range()
    for r in tmgrange(repeats, label="repeats"):
        letters = [random.choice(choices) for _ in range(steps)]
        # `tqdm_gui`, like `tqdm`, accepts any iterable
        # this progress bar is nested and will be run & reset multiple times
        for letter in tqdm_mgui(letters, label="steps"):
            long_function.char.value = letter
            sleep(delay)

long_function.show(run=True)

Untitled.mov

tlambert03 · 2021-01-17T16:05:06Z

cc @GenevieveBuckley

tlambert03 · 2021-01-17T19:20:27Z

progress bar can now also be controlled by annotating a parameter as magicgui.widgets.ProgressBar:

@magicgui(call_button="tick", pbar={"min": 0, "step": 2, "max": 20, "value": 0})
def manual(pbar: ProgressBar, increment: bool = True):
    """Example of manual progress bar control."""
    if increment:
		# can also use pbar.value, pbar.min, pbar.max, etc...
        pbar.increment()
    else:
        pbar.decrement()

sofroniewn · 2021-01-18T01:41:40Z

Wow, this is looking really cool!! I can give a proper review later, but overall the functionality looks great. I'll need to do a little investigation to understand when I'd use tqdm_gui, tmgrange, and magicgui.widgets.ProgressBar. I've not used tqdm itself much, so maybe that's why it's not immediately obvious to me, but I'm excited to give this a go!

sofroniewn

Ok, this is looking good i left some comments/ questions, most are for my clarification/ naming ideas that could be ignored - probably the only thing i see as critical is enable/ disable idea for fixing the segfault after clicking run twice.

examples/progress.py

sofroniewn · 2021-01-23T21:57:54Z

examples/progress.py

+        sleep(delay)
+
+
+long_running.show(run=True)


I noticed that if I click the run button while the thing is running i get a seg fault and quit - we should probably just disable the button while the loop starts and then enable it when done, possibly with a context manager so that it re-enables even with an error during the iteration

(base) czirwc1macos2701:magicgui nsofroniew$ python examples/progress.py Traceback (most recent call last): File "/Users/nsofroniew/GitHub/magicgui/magicgui/widgets/_bases/value_widget.py", line 44, in <lambda> lambda *x: self.changed(value=x[0] if x else None) File "/Users/nsofroniew/GitHub/magicgui/magicgui/events.py", line 576, in __call__ raise RuntimeError("EventEmitter loop detected!") RuntimeError: EventEmitter loop detected! Abort trap: 6

yeah I noticed that too. That's a larger issue having to do with long running functions that doesn't have to do with this PR ... will make a new issue

sounds good

that issue might need to get fixed first, but thinking about it more I think i'd like the behavior to toggle run/ pause which would start and stop progress. could come in later PRs, but noting it now

sofroniewn · 2021-01-23T22:06:21Z

magicgui/tqdm.py

@@ -0,0 +1,151 @@
+"""A wrapper around the tqdm.tqdm iterator that adds a ProgressBar to a magicgui."""


Just wondering if there is another place this could live that is not at the top level, i.e. a grouping of a larger thing that it might be an example of? I guess though there isn't anything else quite like in magicgui right now as it's not a widget per se (that is ProgressBar itself, this is the iterator) but something that if found inside magic functions with generate a widget?

I guess though there isn't anything else quite like in magicgui right now as it's not a widget per se

exactly. this is just the iterator magic. the main reason I put it at the top level is that I don't want to actually depend on tqdm directly (it's available as magicgui[tqdm]), and I want people to use from magicgui.tqdm import tqdm.... As you point out, you can still use the manual progressbar API without this (magicgui.widgets.ProgresBar)

magicgui/tqdm.py

examples/progress.py

sofroniewn · 2021-01-23T22:11:56Z

magicgui/tqdm.py

+}
+
+
+class tqdm_mgui(tqdm):


Curious why not just have this class be called tqdm, to use it you're importing from magicgui import tqdm already, especially if this is a drop in replacement for tqdm / behavior is identical if not inside a magicgui decorated function

because it's not yet a drop-in replacement :/

ok now it is! I liked this idea... so now it's just from magicgui.tqdm import tqdm, trange ... and if it's not inside of a @magicgui it will work as usual (like in IPython)

love it so much!!

sofroniewn · 2021-01-23T22:14:32Z

magicgui/tqdm.py

+            self._mgui._tqdm_depth -= 1
+
+
+def tmgrange(*args, **kwargs) -> tqdm_mgui:


I guess this is analogous to trange, might be nice to link to that https://tqdm.github.io/docs/tqdm/#trange somehow. I'm also wondering if the above import were to change to from magicgui import tqdm would this be from magicgui import trange, maybe not, that's a little awkward?

see above about not depending directly on tqdm. I wanted the import error to be raised on accessing magicgui.tqdm

tlambert03

thanks a lot!

examples/progress.py

tlambert03 · 2021-01-23T23:32:53Z

examples/progress.py

+        sleep(delay)
+
+
+long_running.show(run=True)


yeah I noticed that too. That's a larger issue having to do with long running functions that doesn't have to do with this PR ... will make a new issue

tlambert03 · 2021-01-23T23:37:21Z

magicgui/tqdm.py

@@ -0,0 +1,151 @@
+"""A wrapper around the tqdm.tqdm iterator that adds a ProgressBar to a magicgui."""


I guess though there isn't anything else quite like in magicgui right now as it's not a widget per se

exactly. this is just the iterator magic. the main reason I put it at the top level is that I don't want to actually depend on tqdm directly (it's available as magicgui[tqdm]), and I want people to use from magicgui.tqdm import tqdm.... As you point out, you can still use the manual progressbar API without this (magicgui.widgets.ProgresBar)

tlambert03 · 2021-01-23T23:38:07Z

magicgui/tqdm.py

+}
+
+
+class tqdm_mgui(tqdm):


because it's not yet a drop-in replacement :/

tlambert03 · 2021-01-23T23:39:27Z

magicgui/tqdm.py

+            self._mgui._tqdm_depth -= 1
+
+
+def tmgrange(*args, **kwargs) -> tqdm_mgui:


see above about not depending directly on tqdm. I wanted the import error to be raised on accessing magicgui.tqdm

sofroniewn

Ok, I love it - looks great to me. I made an additional comment on the run/ pause toggle, but sounds like that might have to come in a new PR. I think that's fine, but depending on how much work it is might try and get it in before the next release too. I find it very tempting to click on the run button again and imagine others will too :-)

jni

@tlambert03 pretty happy with this. I'm not 100% about "overwriting" the tqdm name, but I see the argument in favour, so let's give it a go.

Overall, I was kind of envisioning a different, more magical API using generators. Half baked idea:

@magicgui(
    progress=[argname],  # where to get the total length
)
def generator(argname : List):
    for elem in list:
        process(elem)
        yield
    return

There's every chance this current API is better though; For one, my suggestion can't do nested iterations, or resetting, or any of that. But, maybe it could be extended (multiple args, yielded values). At any rate, these are all things that can be explored in the future.

tlambert03 · 2021-01-24T15:14:59Z

Overall, I was kind of envisioning a different, more magical API using generators. Half baked idea:

(as discussed on zulip), I like this idea too, and agree that it isn't mutually exclusive. I like that the current pattern lets someone simply wrap the iterator itself (i.e. it doesn't require you to specify the parameters of the iterable in two places), but I also like thinking about what we could do with generators. so I'll merge this, and keep thinking about that as a complementary pattern

tlambert03 added 4 commits January 16, 2021 16:35

add progress bar

bdfb388

add to docs

8759723

basic tqdm functionality

a5c986e

remove print

aa14036

tlambert03 added 2 commits January 16, 2021 22:33

add tmgrange shortcut

7ac5fe4

Merge branch 'master' into iterator-progress

63c5803

tlambert03 added 5 commits January 17, 2021 11:59

enable nesting

3316089

Merge branch 'master' into iterator-progress

d39bb51

better leave=False behavior

4254c38

fix cleanup

9f47f71

add manual control example

54d2c77

tlambert03 changed the title ~~tqdm wrapper~~ Progress bar tqdm wrapper, and manual control Jan 17, 2021

tlambert03 added 2 commits January 17, 2021 14:21

add tqdm to test requirements

03e4870

shorten pbar tmgrange example

7c38178

tlambert03 marked this pull request as ready for review January 17, 2021 19:25

tlambert03 added 3 commits January 17, 2021 16:33

typo

9eb424f

remove methods from functiongui

b501fc0

complete test coverage

2e39407

tlambert03 requested review from jni and sofroniewn January 18, 2021 00:44

tlambert03 added 3 commits January 17, 2021 19:46

typo

ef2afa0

add docstring

54694bd

add incremet decrement tests

14f7716

tlambert03 added 2 commits January 19, 2021 10:15

Merge branch 'master' into iterator-progress

7b69ca7

Merge branch 'master' into iterator-progress

c31c2d0

tlambert03 added 5 commits January 20, 2021 13:08

Merge branch 'master' into iterator-progress

adb44d9

Merge branch 'master' into iterator-progress

13cff0f

fix import

f623eb9

Merge branch 'master' into iterator-progress

0a6a120

Merge branch 'master' into iterator-progress

2a21553

sofroniewn reviewed Jan 23, 2021

View reviewed changes

tlambert03 commented Jan 23, 2021

View reviewed changes

tlambert03 added 2 commits January 23, 2021 19:28

rename functions to bare tqdm

0812582

split apart examples

0142ee9

sofroniewn approved these changes Jan 24, 2021

View reviewed changes

tlambert03 mentioned this pull request Jan 24, 2021

EventEmitter loop when running an already-running function #128

Closed

Merge branch 'master' into iterator-progress

709f07a

jni approved these changes Jan 24, 2021

View reviewed changes

tlambert03 merged commit d13a829 into pyapp-kit:master Jan 24, 2021

tlambert03 deleted the iterator-progress branch January 24, 2021 18:30

This was referenced Jan 25, 2021

Progress bar widget for long-running computations #25

Closed

v0.2.6 release summary #135

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Progress bar tqdm wrapper, and manual control #105

Progress bar tqdm wrapper, and manual control #105

tlambert03 commented Jan 17, 2021

codecov bot commented Jan 17, 2021 •

edited

Loading

tlambert03 commented Jan 17, 2021

tlambert03 commented Jan 17, 2021

tlambert03 commented Jan 17, 2021

sofroniewn commented Jan 18, 2021

sofroniewn left a comment

sofroniewn Jan 23, 2021

tlambert03 Jan 23, 2021

sofroniewn Jan 24, 2021

sofroniewn Jan 24, 2021

sofroniewn Jan 23, 2021

tlambert03 Jan 23, 2021

sofroniewn Jan 23, 2021

tlambert03 Jan 23, 2021

tlambert03 Jan 24, 2021

sofroniewn Jan 24, 2021

sofroniewn Jan 23, 2021

tlambert03 Jan 23, 2021

tlambert03 left a comment

tlambert03 Jan 23, 2021

tlambert03 Jan 23, 2021

tlambert03 Jan 23, 2021

tlambert03 Jan 23, 2021

sofroniewn left a comment

jni left a comment

tlambert03 commented Jan 24, 2021

		@@ -0,0 +1,151 @@
		"""A wrapper around the tqdm.tqdm iterator that adds a ProgressBar to a magicgui."""

		self._mgui._tqdm_depth -= 1


		def tmgrange(args, *kwargs) -> tqdm_mgui:

		sleep(delay)


		long_running.show(run=True)

		sleep(delay)


		long_running.show(run=True)

Progress bar tqdm wrapper, and manual control #105

Progress bar tqdm wrapper, and manual control #105

Conversation

tlambert03 commented Jan 17, 2021

codecov bot commented Jan 17, 2021 • edited Loading

Codecov Report

tlambert03 commented Jan 17, 2021

tlambert03 commented Jan 17, 2021

tlambert03 commented Jan 17, 2021

sofroniewn commented Jan 18, 2021

sofroniewn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlambert03 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sofroniewn left a comment

Choose a reason for hiding this comment

jni left a comment

Choose a reason for hiding this comment

tlambert03 commented Jan 24, 2021

codecov bot commented Jan 17, 2021 •

edited

Loading