Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Update Parallel Magics and Exception Display #1893

Merged
merged 27 commits into from

4 participants

@minrk
Owner

Based on feedback from @fperez, a few small changes to parallel exception handling and magics:

Exception changes:

  • apply_requests trigger showtraceback machinery, so apply errors are as pretty as execute ones
  • InteractiveShell.showtraceback handles RemoteErrors, so it only draws the remote traceback, rather than the unhelpful local one.

Magics changes:

  • removed parallelmagic extension
  • creating a Client implies activate of a lazily-evaluated directview on all engines
  • can activate Magics on multiple views with different suffixes:
eall = rc.activate('all', 'all')
e0 = rc.activate(0, '0')
%pxall a=5
%px0 print a
  • add %pxconfig magic for changing default block/targets for a collection of magics
  • add targets arg to %%px cell magic
  • %result renamed to %pxresult for consistency (%result kept for bw compat)
  • %pxresult now only draws most recent result, but accepts all the output-formatting args of %%px
  • add --out arg to %%px for storing the AsyncResult object in the user_ns

I need to do another pass on docs and tests, but I think it's mostly there.

@fperez
Owner

Sweet!! Reviewing now... @ellisonbg and @jonathan-taylor, you'll want to have a look at this. During an intense session this week with @jonathan-taylor using the notebook to debug some parallel statistical code, I kept making mental notes and thinking about how we could make the process smoother and easier. Min and I then had a very productive, long discussion on Thursday where we bashed out all these ideas and this is the result. I'm digging into the review now, but having your thoughts would be great too.

Overall I think Min and I felt these changes would make big improvements to the fluidity of the experience and would open up good patterns for using the system, let us know what you think.

@fperez fperez commented on the diff
IPython/extensions/parallelmagic.py
((309 lines not shown))
def load_ipython_extension(ip):
- """Load the extension in IPython."""
- global _loaded
- if not _loaded:
- ip.register_magics(ParallelMagics)
- _loaded = True
+ warn("Parallel Magics are no longer defined in an extension", DeprecationWarning)
@fperez Owner
fperez added a note

This should also say, "use import ..." so that users know what the new API is. Just a bit of user-friendly sugar.

@minrk Owner
minrk added a note

Sure, though I should note that the API hasn't changed. Nobody should have ever explicitly called %load_ext parallelmagic at any point, as it's always been totally redundant with activate. The only difference now is that activate is unnecessary as well, rather than being the only necessary step.

@fperez Owner
fperez added a note

Agreed, I'm just being 'extra friendly' here with errors. It's fine if it stays as-is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fperez
Owner

This looks very solid to me, awesome job. But I'm obviously biased, since we went over the design together, so I'll hold off until others have a chance to give their opinons as well...

We can't run test_pr right now b/c master has an actual failure, I'm going to try to fix that urgently so we can get back to a working master.

@fperez
Owner

Test results for commit b7dc35d (can't merge cleanly)
Platform: linux2

  • python2.7: OK

Not available for testing:

@minrk
Owner

Test results for commit b7dc35d merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: Failed, log at https://gist.github.com/2909198 (libraries not available: rpy2 wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib pymongo qt rpy2 wx wx.aui)

Not available for testing:

@fperez
Owner

Odd, I don't see the any of the 2.7 failures, these aren't all reload ones, are they?

@minrk
Owner

Yes, they are all autoreload - they are all the same error, where distutils does an if foo is Extension test, but foo is the autoreloaded Extension, and Extension is the original, so is fails.

@fperez
Owner

Do you want to disable those tests as a separate commit? That could go into master quickly, so we can really trust test_pr everywhere... I'd rather have a few less tests than a recurrent failure...

@minrk
Owner

Done: #1911

@fperez
Owner

OK, merged and made #1912 to track their isolated re-enabling. Thanks!

@takluyver
Owner

Test results for commit b7dc35d merged into master
Platform: linux2

  • python2.7: OK (libraries not available: rpy2 tornado)
  • python3.1: OK (libraries not available: cython matplotlib numpy pymongo qt rpy2 wx wx.aui zmq)
  • python3.2: OK (libraries not available: cython pymongo rpy2 wx wx.aui)

Not available for testing: python2.6

@takluyver
Owner

(was just using this to test the post_pr_test.py script)

@fperez
Owner

This is such a big improvement to the parallel machinery that I'm tempted to merge it now, so that we can get more feedback on the whole thing as soon as possible in master. We can always refine a bit later, but the code looks clean, the tests all pass and I'm pretty convinced the API and experience are a big step forward. Thoughts?

@minrk
Owner
@fperez
Owner

Great, ping when ready. I'm really excited to see this go in.

minrk added some commits
@minrk minrk Only draw remote tracebacks for IPython.parallel remote exceptions 9c66d11
@minrk minrk use IPython traceback formatting in apply requests ad82d87
@minrk minrk update CompositeError docs with recent changes 2013f1b
@minrk minrk update parallel magic test for exception output with recent change 40ea872
@minrk minrk change %result and add %%px --out foo
%result now only displays the last result, but accepts the display-formatting options.

%%px --out=foo

stores the AsyncResult object for the execution in the user_ns.
8df45e0
@minrk minrk parallel magics are no longer an extension 018b580
@minrk minrk move RemoteError import into showtraceback
avoids circular imports
273f3c7
@minrk minrk remove parallelmagic ext from cluster default config 8293ed3
@minrk minrk update parallel magics
* no longer in extension
* can have multiple active sets of magics with optional suffix
* add pxconfig
* %result only fetches last result, but accepts display-order args
* use magic_arguments decorators
ad8073a
@minrk minrk update Parallel Magics notebook 172fb11
@minrk minrk add note about sync_imports
closes #1598
5fbdbb2
@minrk minrk parallel magics get their own doc file 2b4ceca
@minrk minrk add RMS example for iterating over MapResult bfc0d55
@minrk minrk fill out client.shutdown docstring
closes #1801
fb2dd8e
@minrk minrk View no longer has separate results dict
Only makes it harder to cleanup cache, serves no purpose.
66af5f1
@minrk minrk include errors in IPython.parallel namespace 5887d58
@minrk minrk test %%px --block / --targets 87d1fa8
@minrk minrk add a few pxconfig tests 6036c86
@minrk minrk remove parallelmagic extension from docs d7c3a8d
@minrk minrk publish busy/idle for apply_requests 5f7fdbd
@minrk minrk use idle status message to signal when outputs are complete b49ad41
@minrk minrk update parallel magics docstring d3f1753
@minrk minrk remove now-obsolete output-waits in tests 59a136b
@minrk minrk update Parallel Magics notebook 58b1ac3
@minrk minrk use finite wait_for_outputs d87dacc
@minrk
Owner

Okay, should be ready. Running test_pr now.

Further changes:

  • apply_requests also trigger busy/idle status messages
  • idle message is used to signal that outputs are ready (should help some of the occasionally-failing tests)
@minrk
Owner

added %pxconfig --[no-]verbose, per ML discussion.

Running test_pr, which will certainly fail due to #1916, but these tests should be okay

@ellisonbg
Owner
@minrk
Owner

Test results for commit 59f329c merged into master
Platform: darwin

Not available for testing: python3.1

@minrk
Owner

@ellisonbg I don't know what you mean - there's no difference in size between 8 engines and 800. The "Parallel execution on..." wil always fit comfortably on one line. But I added the verbose flag, and all parallel tests pass, despite the appearance of the above report from test_pr.

@fperez
Owner

Is no-verbose now the default, or verbose?

@minrk
Owner
@fperez
Owner

Great, just checking. My vote is then to merge this. It's very valuable and I think from now on it's diminishing returns to keep it in review... What do you think?

@minrk
Owner
@fperez
Owner

OK, let's go ahead, so we have a chance to see its effect on master for a day or so before the beta. The closer the beta is to a real RC, the better off we'll be.

Awesome job, and thanks for the patience with all this went through!

@fperez
Owner

Merging now.

@fperez fperez merged commit 60e6629 into ipython:master
@minrk
Owner
@minrk minrk deleted the minrk:compositeerr branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 12, 2012
  1. @minrk
  2. @minrk
  3. @minrk
  4. @minrk
  5. @minrk

    change %result and add %%px --out foo

    minrk authored
    %result now only displays the last result, but accepts the display-formatting options.
    
    %%px --out=foo
    
    stores the AsyncResult object for the execution in the user_ns.
  6. @minrk
  7. @minrk

    move RemoteError import into showtraceback

    minrk authored
    avoids circular imports
  8. @minrk
  9. @minrk

    update parallel magics

    minrk authored
    * no longer in extension
    * can have multiple active sets of magics with optional suffix
    * add pxconfig
    * %result only fetches last result, but accepts display-order args
    * use magic_arguments decorators
  10. @minrk

    update Parallel Magics notebook

    minrk authored
  11. @minrk

    add note about sync_imports

    minrk authored
    closes #1598
  12. @minrk
  13. @minrk
  14. @minrk

    fill out client.shutdown docstring

    minrk authored
    closes #1801
  15. @minrk

    View no longer has separate results dict

    minrk authored
    Only makes it harder to cleanup cache, serves no purpose.
  16. @minrk
  17. @minrk

    test %%px --block / --targets

    minrk authored
  18. @minrk

    add a few pxconfig tests

    minrk authored
  19. @minrk
  20. @minrk
  21. @minrk
  22. @minrk

    update parallel magics docstring

    minrk authored
  23. @minrk
  24. @minrk

    update Parallel Magics notebook

    minrk authored
  25. @minrk

    use finite wait_for_outputs

    minrk authored
  26. @minrk
  27. @minrk

    add %pxconfig --[no-]verbose

    minrk authored
Something went wrong with that request. Please try again.