add sugar methods/properties to AsyncResult #1548
Conversation
|
At @fperez request, I had a go at addressing the flicker when using clear_output in the notebook. I used a simple timeout, which gets cleared/flushed immediately on the next output-related action. I won't object to moving that to a separate PR, but I did it here as it is related, and this new code makes significant use of the |
|
this notebook shows a few examples of print, the new AR.wait_interactive, and plotting with clear_output, all of which seem better behaved than previously. |
|
This PR now depends on PR #1563 |
|
And as I commented in that other one, might as well make the notebook with examples part of the PR, so we beef up the notebooks we provide with the system for users. That will become even more useful once we start including their html for in the docs, along with a link to the original .ipynb file, as we'll be able to point users to those pages in the docs. |
* ar.wall_time = received - submitted * ar.serial_time = sum of serial computation time * ar.elapsed = time since submission (wall_time if done) * ar.progress = (int) number of sub-tasks that have completed * ar.wait_interactive(): prints progress * len(ar) = # of tasks
|
rebased - I'll add the notebook in the morning, probably. |
|
Progress notebook added, demos:
|
| "collapsed": false, | ||
| "input": [ | ||
| "from IPython.core.display import clear_output", | ||
| "for i in range(10):", |
There was a problem hiding this comment.
It's missing an import time here.
There was a problem hiding this comment.
ah, yes. I have import os,sys,time in my startup files, so I always forget to import them.
|
Minor comments inline, easy to fix up and we'll be good to go. This is awesome. |
demos intermediate progress with: * clear_output/print/flush * AsyncResult.wait_interactive * clear_output/display(Figure) * HTML/JS progress bar * PyMC ProgressBar class
|
I made your edits to the notebook, and started a doc for the AsyncResult object. I really want to make it clear that the AsyncResult object itself is where most of our API lives. At least the nifty parts. I have one question we should resolve before merging. In wall_time, I use received - submitted, which is the actual roundtrip time for the Client. The only problem with this one is that the This is a larger issue for AsyncHubResults (those fetched from the Hub), where the Possible partial solutions to this problem:
I'm personally inclined towards either one of the last two, but I want there to be a convenient There are a couple of other timings that might be useful to add:
I don't know how many of these I should actually include. Perhaps I should just make a method that makes it easy to do any of the |
appease crappy tools that can't deal with reasonable filenames.
allows 'wall_time' to make sense in cases other than simple waiting AsyncResult.
for computing various comparisons of timestamps in AsyncResults
|
Okay, review addressed I think:
Did I miss anything else? |
runs Client.spin() in a background thread at a set interval
|
Added |
|
Great, merging now. Awesome job, thanks! |
Add sugar methods/properties to AsyncResult that are generically useful: * `ar.wall_time` = received - submitted * `ar.serial_time` = sum of serial computation time * `ar.elapsed` = time since submission (wall_time if done) * `ar.progress` = (int) number of sub-tasks that have completed * `len(ar)` = # of tasks * `ar.wait_interactive()`: prints progress These are simple methods derived from the metadata/timestamps already created. But I've been persuaded by @wesm's practice of including simple methods that do useful (and/or cool) things. This also required/revealed some minor fixes/cleanup to clear_output in some cases: * dedent base `core.displaypub.clear_output`, so it's actually defined in the class * clear_output publishes `'\r\b'`, so it will clear terminal-like frontends that don't understand full clear_output behavior. * `core.display.clear_output()` still works, even outside an IPython session. Added a new notebook that shows how to use these new methods and how to do simple animations/progress bars using `clear_output()`. Added `Client.spin_thread(interval)` / `stop_spin_thread()` for running spin in a background thread, to keep zmq queue clear. This can be used to ensure that timing information is as accurate as possible (at the cost of having a background thread active).
Add sugar methods/properties to AsyncResult that are generically useful: * `ar.wall_time` = received - submitted * `ar.serial_time` = sum of serial computation time * `ar.elapsed` = time since submission (wall_time if done) * `ar.progress` = (int) number of sub-tasks that have completed * `len(ar)` = # of tasks * `ar.wait_interactive()`: prints progress These are simple methods derived from the metadata/timestamps already created. But I've been persuaded by @wesm's practice of including simple methods that do useful (and/or cool) things. This also required/revealed some minor fixes/cleanup to clear_output in some cases: * dedent base `core.displaypub.clear_output`, so it's actually defined in the class * clear_output publishes `'\r\b'`, so it will clear terminal-like frontends that don't understand full clear_output behavior. * `core.display.clear_output()` still works, even outside an IPython session. Added a new notebook that shows how to use these new methods and how to do simple animations/progress bars using `clear_output()`. Added `Client.spin_thread(interval)` / `stop_spin_thread()` for running spin in a background thread, to keep zmq queue clear. This can be used to ensure that timing information is as accurate as possible (at the cost of having a background thread active).
Adds the following attributes / methods:
ar.wall_time= received - submittedar.serial_time= sum of serial computation timear.elapsed= time since submission (wall_time if done)ar.progress= (int) number of sub-tasks that have completedlen(ar)= # of tasksar.wait_interactive(): prints progressThese are simple methods derived from the metadata/timestamps already created. But I've been persuaded by @wesm's practice of including simple methods that do useful (and/or cool) things.
This also required/revealed some minor fixes/cleanup to clear_output in some cases:
core.displaypub.clear_output, so it's actually defined in the class'\r\b', so it will clear terminal-like frontends that don't understand full clear_output behavior.core.display.clear_output()still works, even outside an IPython session.