Update Parallel Magics and Exception Display #1893

Merged
merged 27 commits into from Jun 12, 2012

Projects

None yet

4 participants

@minrk
IPython member

Based on feedback from @fperez, a few small changes to parallel exception handling and magics:

Exception changes:

  • apply_requests trigger showtraceback machinery, so apply errors are as pretty as execute ones
  • InteractiveShell.showtraceback handles RemoteErrors, so it only draws the remote traceback, rather than the unhelpful local one.

Magics changes:

  • removed parallelmagic extension
  • creating a Client implies activate of a lazily-evaluated directview on all engines
  • can activate Magics on multiple views with different suffixes:
eall = rc.activate('all', 'all')
e0 = rc.activate(0, '0')
%pxall a=5
%px0 print a
  • add %pxconfig magic for changing default block/targets for a collection of magics
  • add targets arg to %%px cell magic
  • %result renamed to %pxresult for consistency (%result kept for bw compat)
  • %pxresult now only draws most recent result, but accepts all the output-formatting args of %%px
  • add --out arg to %%px for storing the AsyncResult object in the user_ns

I need to do another pass on docs and tests, but I think it's mostly there.

@fperez
IPython member

Sweet!! Reviewing now... @ellisonbg and @jonathan-taylor, you'll want to have a look at this. During an intense session this week with @jonathan-taylor using the notebook to debug some parallel statistical code, I kept making mental notes and thinking about how we could make the process smoother and easier. Min and I then had a very productive, long discussion on Thursday where we bashed out all these ideas and this is the result. I'm digging into the review now, but having your thoughts would be great too.

Overall I think Min and I felt these changes would make big improvements to the fluidity of the experience and would open up good patterns for using the system, let us know what you think.

@fperez fperez commented on the diff Jun 10, 2012
IPython/extensions/parallelmagic.py
def load_ipython_extension(ip):
- """Load the extension in IPython."""
- global _loaded
- if not _loaded:
- ip.register_magics(ParallelMagics)
- _loaded = True
+ warn("Parallel Magics are no longer defined in an extension", DeprecationWarning)
@fperez
fperez Jun 10, 2012

This should also say, "use import ..." so that users know what the new API is. Just a bit of user-friendly sugar.

@minrk
minrk Jun 10, 2012

Sure, though I should note that the API hasn't changed. Nobody should have ever explicitly called %load_ext parallelmagic at any point, as it's always been totally redundant with activate. The only difference now is that activate is unnecessary as well, rather than being the only necessary step.

@fperez
fperez Jun 11, 2012

Agreed, I'm just being 'extra friendly' here with errors. It's fine if it stays as-is.

@fperez
IPython member

This looks very solid to me, awesome job. But I'm obviously biased, since we went over the design together, so I'll hold off until others have a chance to give their opinons as well...

We can't run test_pr right now b/c master has an actual failure, I'm going to try to fix that urgently so we can get back to a working master.

@fperez
IPython member

Test results for commit b7dc35d (can't merge cleanly)
Platform: linux2

  • python2.7: OK

Not available for testing:

@minrk
IPython member

Test results for commit b7dc35d merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: Failed, log at https://gist.github.com/2909198 (libraries not available: rpy2 wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib pymongo qt rpy2 wx wx.aui)

Not available for testing:

@fperez
IPython member

Odd, I don't see the any of the 2.7 failures, these aren't all reload ones, are they?

@minrk
IPython member

Yes, they are all autoreload - they are all the same error, where distutils does an if foo is Extension test, but foo is the autoreloaded Extension, and Extension is the original, so is fails.

@fperez
IPython member

Do you want to disable those tests as a separate commit? That could go into master quickly, so we can really trust test_pr everywhere... I'd rather have a few less tests than a recurrent failure...

@minrk
IPython member

Done: #1911

@fperez
IPython member

OK, merged and made #1912 to track their isolated re-enabling. Thanks!

@takluyver
IPython member

Test results for commit b7dc35d merged into master
Platform: linux2

  • python2.7: OK (libraries not available: rpy2 tornado)
  • python3.1: OK (libraries not available: cython matplotlib numpy pymongo qt rpy2 wx wx.aui zmq)
  • python3.2: OK (libraries not available: cython pymongo rpy2 wx wx.aui)

Not available for testing: python2.6

@takluyver
IPython member

(was just using this to test the post_pr_test.py script)

@fperez
IPython member

This is such a big improvement to the parallel machinery that I'm tempted to merge it now, so that we can get more feedback on the whole thing as soon as possible in master. We can always refine a bit later, but the code looks clean, the tests all pass and I'm pretty convinced the API and experience are a big step forward. Thoughts?

@minrk
IPython member
@fperez
IPython member

Great, ping when ready. I'm really excited to see this go in.

minrk added some commits Jun 8, 2012
@minrk minrk Only draw remote tracebacks for IPython.parallel remote exceptions 9c66d11
@minrk minrk use IPython traceback formatting in apply requests ad82d87
@minrk minrk update CompositeError docs with recent changes 2013f1b
@minrk minrk update parallel magic test for exception output with recent change 40ea872
@minrk minrk change %result and add %%px --out foo
%result now only displays the last result, but accepts the display-formatting options.

%%px --out=foo

stores the AsyncResult object for the execution in the user_ns.
8df45e0
@minrk minrk parallel magics are no longer an extension 018b580
@minrk minrk move RemoteError import into showtraceback
avoids circular imports
273f3c7
@minrk minrk remove parallelmagic ext from cluster default config 8293ed3
@minrk minrk update parallel magics
* no longer in extension
* can have multiple active sets of magics with optional suffix
* add pxconfig
* %result only fetches last result, but accepts display-order args
* use magic_arguments decorators
ad8073a
@minrk minrk update Parallel Magics notebook 172fb11
@minrk minrk add note about sync_imports
closes #1598
5fbdbb2
@minrk minrk parallel magics get their own doc file 2b4ceca
@minrk minrk add RMS example for iterating over MapResult bfc0d55
@minrk minrk fill out client.shutdown docstring
closes #1801
fb2dd8e
@minrk minrk View no longer has separate results dict
Only makes it harder to cleanup cache, serves no purpose.
66af5f1
@minrk minrk include errors in IPython.parallel namespace 5887d58
@minrk minrk test %%px --block / --targets 87d1fa8
@minrk minrk add a few pxconfig tests 6036c86
@minrk minrk remove parallelmagic extension from docs d7c3a8d
@minrk minrk publish busy/idle for apply_requests 5f7fdbd
@minrk minrk use idle status message to signal when outputs are complete b49ad41
@minrk minrk update parallel magics docstring d3f1753
@minrk minrk remove now-obsolete output-waits in tests 59a136b
@minrk minrk update Parallel Magics notebook 58b1ac3
@minrk minrk use finite wait_for_outputs d87dacc
@minrk
IPython member

Okay, should be ready. Running test_pr now.

Further changes:

  • apply_requests also trigger busy/idle status messages
  • idle message is used to signal that outputs are ready (should help some of the occasionally-failing tests)
@minrk
IPython member

added %pxconfig --[no-]verbose, per ML discussion.

Running test_pr, which will certainly fail due to #1916, but these tests should be okay

@ellisonbg
IPython member
@minrk
IPython member

Test results for commit 59f329c merged into master
Platform: darwin

Not available for testing: python3.1

@minrk
IPython member

@ellisonbg I don't know what you mean - there's no difference in size between 8 engines and 800. The "Parallel execution on..." wil always fit comfortably on one line. But I added the verbose flag, and all parallel tests pass, despite the appearance of the above report from test_pr.

@fperez
IPython member

Is no-verbose now the default, or verbose?

@minrk
IPython member
@fperez
IPython member

Great, just checking. My vote is then to merge this. It's very valuable and I think from now on it's diminishing returns to keep it in review... What do you think?

@minrk
IPython member
@fperez
IPython member

OK, let's go ahead, so we have a chance to see its effect on master for a day or so before the beta. The closer the beta is to a real RC, the better off we'll be.

Awesome job, and thanks for the patience with all this went through!

@fperez
IPython member

Merging now.

@fperez fperez merged commit 60e6629 into ipython:master Jun 12, 2012
@minrk
IPython member
@minrk minrk deleted the minrk:compositeerr branch Mar 31, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment