Widget progress bar missing in Notebook

[jakirkham]

Not sure if things are just finishing so fast on my trivial test data or what, but I'm not seeing the widget progress bar show up. If I switch to the text one it shows up fine, but is complete immediately. Is the widget progress bar not being drawn on completion or is something else going on?

[mrocklin]

My first guesses are that ipywidgets isn't installed and working properly or that no feedback has come back yet from the scheduler.

[jakirkham]

Considering I have my own custom progress bars in some places still that work fine, I don't think that is it. I'd be happy to provide more info if you can tell me what it is you need.

[mrocklin]

I don't have any other requests for information. I don't know how to diagnose this issue.

[jakirkham]

In an effort to make some progress on this issue, do you have a simple example that works for you?

[mrocklin]

Pretty much all examples work fine for me. Here is a trivial one.

futures = client.map(sleep, [1] * 100)
progress(futures)

[jakirkham]

I'm assuming in your example that you have the following imports and client is the distributed client.

from time import sleep
from distributed.diagnostics import progress

That said, this just returns for me immediately with no progress bar. Doesn't this need some sort of compute step? If so, where does that go? If not, I don't understand how this works.

[mrocklin]

This uses the concurrent.futures API. http://distributed.readthedocs.io/en/latest/quickstart.html#map-and-submit-functions

[jakirkham]

Ok, so that's still not totally clear.

Which line are you doing the computation on? Before calling progress? After? AFAICT progress doesn't do the computation itself.

Also if you are using submit, how are you calling it? Looks like it takes a function, what are you giving it? An identity function? Would gather work here without submit?

[mrocklin]

Sorry about that. Here is a more complete self-contained example:

https://gist.github.com/c4e3bfbd449f544a4a3275188c290b9d

Map or submit send computations immediately. They are like compute. http://distributed.readthedocs.io/en/latest/memory.html#creating-futures

[jakirkham]

Thanks. All is clear now. Was able to reproduce that without issue. 😄

So I think the problem is I would like to block further execution until the tasks are complete. However given the current implementation of these progress bars, this will just block them from ever being drawn, which won't work. Is there some standard way to block further execution and let the progress bar be drawn?

[mrocklin]

Call progress on your thing. Then call dask.distributed.wait on your thing

…

On Tue, May 30, 2017 at 3:29 PM, jakirkham ***@***.***> wrote: Thanks. All is clear now. Was able to reproduce that without issue. 😄 So I think the problem is I would like to block further execution until the tasks are complete. However given the current implementation of these progress bars, this will just block them from ever being drawn, which won't work. Is there some standard way to block further execution and let the progress bar be drawn? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#926 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszMHWwex8Y51v7vfviY20jvqkZiUAks5r_G4rgaJpZM4MRhMu> .

[jakirkham]

I've tried wait, looping through as_completed, manually checking for all futures to be done/sleeping if not. In all cases, they block the Jupyter Notebook's ability to draw. Also tried to run wait in another thread, but that ran into different issues. Any other ideas?

[mrocklin]

Nope. Other than to use something outside of Jupyter notebooks, like the Bokeh dashboard.

…

On Tue, May 30, 2017 at 3:53 PM, jakirkham ***@***.***> wrote: I've tried wait, looping through as_completed, manually checking for all futures to be done and sleeping if not. In all cases, this results in blocking the Jupyter Notebook's ability to draw. Also tried to run wait in another thread, but that ran into different issues. Any other ideas? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#926 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszJl5A5XgZ0Yvz2oRae8DuKycdZSMks5r_HPBgaJpZM4MRhMu> .

[mrocklin]

Although I guess we could put some sort of wait keyword in the notebook progressbar? On Tue, May 30, 2017 at 4:45 PM, Matthew Rocklin <mrocklin@continuum.io> wrote:

…

Nope. Other than to use something outside of Jupyter notebooks, like the Bokeh dashboard. On Tue, May 30, 2017 at 3:53 PM, jakirkham ***@***.***> wrote: > I've tried wait, looping through as_completed, manually checking for all > futures to be done and sleeping if not. In all cases, this results in > blocking the Jupyter Notebook's ability to draw. Also tried to run wait > in another thread, but that ran into different issues. Any other ideas? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#926 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AASszJl5A5XgZ0Yvz2oRae8DuKycdZSMks5r_HPBgaJpZM4MRhMu> > . >