Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Update Parallel Magics and Exception Display #1893

Merged
merged 27 commits into from

4 participants

Min RK Fernando Perez Thomas Kluyver Brian E. Granger
Min RK
Owner

Based on feedback from @fperez, a few small changes to parallel exception handling and magics:

Exception changes:

  • apply_requests trigger showtraceback machinery, so apply errors are as pretty as execute ones
  • InteractiveShell.showtraceback handles RemoteErrors, so it only draws the remote traceback, rather than the unhelpful local one.

Magics changes:

  • removed parallelmagic extension
  • creating a Client implies activate of a lazily-evaluated directview on all engines
  • can activate Magics on multiple views with different suffixes:
eall = rc.activate('all', 'all')
e0 = rc.activate(0, '0')
%pxall a=5
%px0 print a
  • add %pxconfig magic for changing default block/targets for a collection of magics
  • add targets arg to %%px cell magic
  • %result renamed to %pxresult for consistency (%result kept for bw compat)
  • %pxresult now only draws most recent result, but accepts all the output-formatting args of %%px
  • add --out arg to %%px for storing the AsyncResult object in the user_ns

I need to do another pass on docs and tests, but I think it's mostly there.

Fernando Perez
Owner

Sweet!! Reviewing now... @ellisonbg and @jonathan-taylor, you'll want to have a look at this. During an intense session this week with @jonathan-taylor using the notebook to debug some parallel statistical code, I kept making mental notes and thinking about how we could make the process smoother and easier. Min and I then had a very productive, long discussion on Thursday where we bashed out all these ideas and this is the result. I'm digging into the review now, but having your thoughts would be great too.

Overall I think Min and I felt these changes would make big improvements to the fluidity of the experience and would open up good patterns for using the system, let us know what you think.

Fernando Perez fperez commented on the diff
IPython/extensions/parallelmagic.py
((309 lines not shown))
def load_ipython_extension(ip):
- """Load the extension in IPython."""
- global _loaded
- if not _loaded:
- ip.register_magics(ParallelMagics)
- _loaded = True
+ warn("Parallel Magics are no longer defined in an extension", DeprecationWarning)
Fernando Perez Owner
fperez added a note

This should also say, "use import ..." so that users know what the new API is. Just a bit of user-friendly sugar.

Min RK Owner
minrk added a note

Sure, though I should note that the API hasn't changed. Nobody should have ever explicitly called %load_ext parallelmagic at any point, as it's always been totally redundant with activate. The only difference now is that activate is unnecessary as well, rather than being the only necessary step.

Fernando Perez Owner
fperez added a note

Agreed, I'm just being 'extra friendly' here with errors. It's fine if it stays as-is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner

This looks very solid to me, awesome job. But I'm obviously biased, since we went over the design together, so I'll hold off until others have a chance to give their opinons as well...

We can't run test_pr right now b/c master has an actual failure, I'm going to try to fix that urgently so we can get back to a working master.

Fernando Perez
Owner

Test results for commit b7dc35d (can't merge cleanly)
Platform: linux2

  • python2.7: OK

Not available for testing:

Min RK
Owner

Test results for commit b7dc35d merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: Failed, log at https://gist.github.com/2909198 (libraries not available: rpy2 wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib pymongo qt rpy2 wx wx.aui)

Not available for testing:

Fernando Perez
Owner

Odd, I don't see the any of the 2.7 failures, these aren't all reload ones, are they?

Min RK
Owner

Yes, they are all autoreload - they are all the same error, where distutils does an if foo is Extension test, but foo is the autoreloaded Extension, and Extension is the original, so is fails.

Fernando Perez
Owner

Do you want to disable those tests as a separate commit? That could go into master quickly, so we can really trust test_pr everywhere... I'd rather have a few less tests than a recurrent failure...

Min RK
Owner

Done: #1911

Fernando Perez
Owner

OK, merged and made #1912 to track their isolated re-enabling. Thanks!

Thomas Kluyver
Owner

Test results for commit b7dc35d merged into master
Platform: linux2

  • python2.7: OK (libraries not available: rpy2 tornado)
  • python3.1: OK (libraries not available: cython matplotlib numpy pymongo qt rpy2 wx wx.aui zmq)
  • python3.2: OK (libraries not available: cython pymongo rpy2 wx wx.aui)

Not available for testing: python2.6

Thomas Kluyver
Owner

(was just using this to test the post_pr_test.py script)

Fernando Perez
Owner

This is such a big improvement to the parallel machinery that I'm tempted to merge it now, so that we can get more feedback on the whole thing as soon as possible in master. We can always refine a bit later, but the code looks clean, the tests all pass and I'm pretty convinced the API and experience are a big step forward. Thoughts?

Min RK
Owner
Fernando Perez
Owner

Great, ping when ready. I'm really excited to see this go in.

minrk added some commits
Min RK minrk Only draw remote tracebacks for IPython.parallel remote exceptions 9c66d11
Min RK minrk use IPython traceback formatting in apply requests ad82d87
Min RK minrk update CompositeError docs with recent changes 2013f1b
Min RK minrk update parallel magic test for exception output with recent change 40ea872
Min RK minrk change %result and add %%px --out foo
%result now only displays the last result, but accepts the display-formatting options.

%%px --out=foo

stores the AsyncResult object for the execution in the user_ns.
8df45e0
Min RK minrk parallel magics are no longer an extension 018b580
Min RK minrk move RemoteError import into showtraceback
avoids circular imports
273f3c7
Min RK minrk remove parallelmagic ext from cluster default config 8293ed3
Min RK minrk update parallel magics
* no longer in extension
* can have multiple active sets of magics with optional suffix
* add pxconfig
* %result only fetches last result, but accepts display-order args
* use magic_arguments decorators
ad8073a
Min RK minrk update Parallel Magics notebook 172fb11
Min RK minrk add note about sync_imports
closes #1598
5fbdbb2
Min RK minrk parallel magics get their own doc file 2b4ceca
Min RK minrk add RMS example for iterating over MapResult bfc0d55
Min RK minrk fill out client.shutdown docstring
closes #1801
fb2dd8e
Min RK minrk View no longer has separate results dict
Only makes it harder to cleanup cache, serves no purpose.
66af5f1
Min RK minrk include errors in IPython.parallel namespace 5887d58
Min RK minrk test %%px --block / --targets 87d1fa8
Min RK minrk add a few pxconfig tests 6036c86
Min RK minrk remove parallelmagic extension from docs d7c3a8d
Min RK minrk publish busy/idle for apply_requests 5f7fdbd
Min RK minrk use idle status message to signal when outputs are complete b49ad41
Min RK minrk update parallel magics docstring d3f1753
Min RK minrk remove now-obsolete output-waits in tests 59a136b
Min RK minrk update Parallel Magics notebook 58b1ac3
Min RK minrk use finite wait_for_outputs d87dacc
Min RK
Owner

Okay, should be ready. Running test_pr now.

Further changes:

  • apply_requests also trigger busy/idle status messages
  • idle message is used to signal that outputs are ready (should help some of the occasionally-failing tests)
Min RK
Owner

added %pxconfig --[no-]verbose, per ML discussion.

Running test_pr, which will certainly fail due to #1916, but these tests should be okay

Brian E. Granger
Owner
Min RK
Owner

Test results for commit 59f329c merged into master
Platform: darwin

Not available for testing: python3.1

Min RK
Owner

@ellisonbg I don't know what you mean - there's no difference in size between 8 engines and 800. The "Parallel execution on..." wil always fit comfortably on one line. But I added the verbose flag, and all parallel tests pass, despite the appearance of the above report from test_pr.

Fernando Perez
Owner

Is no-verbose now the default, or verbose?

Min RK
Owner
Fernando Perez
Owner

Great, just checking. My vote is then to merge this. It's very valuable and I think from now on it's diminishing returns to keep it in review... What do you think?

Min RK
Owner
Fernando Perez
Owner

OK, let's go ahead, so we have a chance to see its effect on master for a day or so before the beta. The closer the beta is to a real RC, the better off we'll be.

Awesome job, and thanks for the patience with all this went through!

Fernando Perez
Owner

Merging now.

Fernando Perez fperez merged commit 60e6629 into from
Min RK
Owner
Min RK minrk deleted the branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 12, 2012
  1. Min RK
  2. Min RK
  3. Min RK
  4. Min RK
  5. Min RK

    change %result and add %%px --out foo

    minrk authored
    %result now only displays the last result, but accepts the display-formatting options.
    
    %%px --out=foo
    
    stores the AsyncResult object for the execution in the user_ns.
  6. Min RK
  7. Min RK

    move RemoteError import into showtraceback

    minrk authored
    avoids circular imports
  8. Min RK
  9. Min RK

    update parallel magics

    minrk authored
    * no longer in extension
    * can have multiple active sets of magics with optional suffix
    * add pxconfig
    * %result only fetches last result, but accepts display-order args
    * use magic_arguments decorators
  10. Min RK

    update Parallel Magics notebook

    minrk authored
  11. Min RK

    add note about sync_imports

    minrk authored
    closes #1598
  12. Min RK
  13. Min RK
  14. Min RK

    fill out client.shutdown docstring

    minrk authored
    closes #1801
  15. Min RK

    View no longer has separate results dict

    minrk authored
    Only makes it harder to cleanup cache, serves no purpose.
  16. Min RK
  17. Min RK

    test %%px --block / --targets

    minrk authored
  18. Min RK

    add a few pxconfig tests

    minrk authored
  19. Min RK
  20. Min RK
  21. Min RK
  22. Min RK

    update parallel magics docstring

    minrk authored
  23. Min RK
  24. Min RK

    update Parallel Magics notebook

    minrk authored
  25. Min RK

    use finite wait_for_outputs

    minrk authored
  26. Min RK
  27. Min RK

    add %pxconfig --[no-]verbose

    minrk authored
Something went wrong with that request. Please try again.