Skip to content
This repository

Update Parallel Magics and Exception Display #1893

Merged
merged 27 commits into from almost 2 years ago

4 participants

Min RK Fernando Perez Thomas Kluyver Brian E. Granger
Min RK
Owner
minrk commented June 09, 2012

Based on feedback from @fperez, a few small changes to parallel exception handling and magics:

Exception changes:

  • apply_requests trigger showtraceback machinery, so apply errors are as pretty as execute ones
  • InteractiveShell.showtraceback handles RemoteErrors, so it only draws the remote traceback, rather than the unhelpful local one.

Magics changes:

  • removed parallelmagic extension
  • creating a Client implies activate of a lazily-evaluated directview on all engines
  • can activate Magics on multiple views with different suffixes:
eall = rc.activate('all', 'all')
e0 = rc.activate(0, '0')
%pxall a=5
%px0 print a
  • add %pxconfig magic for changing default block/targets for a collection of magics
  • add targets arg to %%px cell magic
  • %result renamed to %pxresult for consistency (%result kept for bw compat)
  • %pxresult now only draws most recent result, but accepts all the output-formatting args of %%px
  • add --out arg to %%px for storing the AsyncResult object in the user_ns

I need to do another pass on docs and tests, but I think it's mostly there.

Fernando Perez
Owner
fperez commented June 09, 2012

Sweet!! Reviewing now... @ellisonbg and @jonathan-taylor, you'll want to have a look at this. During an intense session this week with @jonathan-taylor using the notebook to debug some parallel statistical code, I kept making mental notes and thinking about how we could make the process smoother and easier. Min and I then had a very productive, long discussion on Thursday where we bashed out all these ideas and this is the result. I'm digging into the review now, but having your thoughts would be great too.

Overall I think Min and I felt these changes would make big improvements to the fluidity of the experience and would open up good patterns for using the system, let us know what you think.

Fernando Perez fperez commented on the diff June 09, 2012
IPython/extensions/parallelmagic.py
((309 lines not shown))
337 18
 
338 19
 def load_ipython_extension(ip):
339  
-    """Load the extension in IPython."""
340  
-    global _loaded
341  
-    if not _loaded:
342  
-        ip.register_magics(ParallelMagics)
343  
-        _loaded = True
  20
+    warn("Parallel Magics are no longer defined in an extension", DeprecationWarning)
3
Fernando Perez Owner
fperez added a note June 09, 2012

This should also say, "use import ..." so that users know what the new API is. Just a bit of user-friendly sugar.

Min RK Owner
minrk added a note June 09, 2012

Sure, though I should note that the API hasn't changed. Nobody should have ever explicitly called %load_ext parallelmagic at any point, as it's always been totally redundant with activate. The only difference now is that activate is unnecessary as well, rather than being the only necessary step.

Fernando Perez Owner
fperez added a note June 10, 2012

Agreed, I'm just being 'extra friendly' here with errors. It's fine if it stays as-is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner
fperez commented June 09, 2012

This looks very solid to me, awesome job. But I'm obviously biased, since we went over the design together, so I'll hold off until others have a chance to give their opinons as well...

We can't run test_pr right now b/c master has an actual failure, I'm going to try to fix that urgently so we can get back to a working master.

Fernando Perez
Owner
fperez commented June 11, 2012

Test results for commit b7dc35d (can't merge cleanly)
Platform: linux2

  • python2.7: OK

Not available for testing:

Min RK
Owner
minrk commented June 11, 2012

Test results for commit b7dc35d merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: Failed, log at https://gist.github.com/2909198 (libraries not available: rpy2 wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib pymongo qt rpy2 wx wx.aui)

Not available for testing:

Fernando Perez
Owner
fperez commented June 11, 2012

Odd, I don't see the any of the 2.7 failures, these aren't all reload ones, are they?

Min RK
Owner
minrk commented June 11, 2012

Yes, they are all autoreload - they are all the same error, where distutils does an if foo is Extension test, but foo is the autoreloaded Extension, and Extension is the original, so is fails.

Fernando Perez
Owner
fperez commented June 11, 2012

Do you want to disable those tests as a separate commit? That could go into master quickly, so we can really trust test_pr everywhere... I'd rather have a few less tests than a recurrent failure...

Min RK
Owner
minrk commented June 11, 2012

Done: #1911

Fernando Perez
Owner
fperez commented June 11, 2012

OK, merged and made #1912 to track their isolated re-enabling. Thanks!

Thomas Kluyver
Collaborator

Test results for commit b7dc35d merged into master
Platform: linux2

  • python2.7: OK (libraries not available: rpy2 tornado)
  • python3.1: OK (libraries not available: cython matplotlib numpy pymongo qt rpy2 wx wx.aui zmq)
  • python3.2: OK (libraries not available: cython pymongo rpy2 wx wx.aui)

Not available for testing: python2.6

Thomas Kluyver
Collaborator

(was just using this to test the post_pr_test.py script)

Fernando Perez
Owner
fperez commented June 11, 2012

This is such a big improvement to the parallel machinery that I'm tempted to merge it now, so that we can get more feedback on the whole thing as soon as possible in master. We can always refine a bit later, but the code looks clean, the tests all pass and I'm pretty convinced the API and experience are a big step forward. Thoughts?

Min RK
Owner
minrk commented June 11, 2012
Fernando Perez
Owner
fperez commented June 11, 2012

Great, ping when ready. I'm really excited to see this go in.

added some commits June 08, 2012
Min RK Only draw remote tracebacks for IPython.parallel remote exceptions 9c66d11
Min RK use IPython traceback formatting in apply requests ad82d87
Min RK update CompositeError docs with recent changes 2013f1b
Min RK update parallel magic test for exception output with recent change 40ea872
Min RK change %result and add %%px --out foo
%result now only displays the last result, but accepts the display-formatting options.

%%px --out=foo

stores the AsyncResult object for the execution in the user_ns.
8df45e0
Min RK parallel magics are no longer an extension 018b580
Min RK move RemoteError import into showtraceback
avoids circular imports
273f3c7
Min RK remove parallelmagic ext from cluster default config 8293ed3
Min RK update parallel magics
* no longer in extension
* can have multiple active sets of magics with optional suffix
* add pxconfig
* %result only fetches last result, but accepts display-order args
* use magic_arguments decorators
ad8073a
Min RK update Parallel Magics notebook 172fb11
Min RK add note about sync_imports
closes #1598
5fbdbb2
Min RK parallel magics get their own doc file 2b4ceca
Min RK add RMS example for iterating over MapResult bfc0d55
Min RK fill out client.shutdown docstring
closes #1801
fb2dd8e
Min RK View no longer has separate results dict
Only makes it harder to cleanup cache, serves no purpose.
66af5f1
Min RK include errors in IPython.parallel namespace 5887d58
Min RK test %%px --block / --targets 87d1fa8
Min RK add a few pxconfig tests 6036c86
Min RK remove parallelmagic extension from docs d7c3a8d
Min RK publish busy/idle for apply_requests 5f7fdbd
Min RK use idle status message to signal when outputs are complete b49ad41
Min RK update parallel magics docstring d3f1753
Min RK remove now-obsolete output-waits in tests 59a136b
Min RK update Parallel Magics notebook 58b1ac3
Min RK use finite wait_for_outputs d87dacc
Min RK
Owner
minrk commented June 11, 2012

Okay, should be ready. Running test_pr now.

Further changes:

  • apply_requests also trigger busy/idle status messages
  • idle message is used to signal that outputs are ready (should help some of the occasionally-failing tests)
Min RK
Owner
minrk commented June 11, 2012

added %pxconfig --[no-]verbose, per ML discussion.

Running test_pr, which will certainly fail due to #1916, but these tests should be okay

Brian E. Granger
Owner
Min RK
Owner
minrk commented June 11, 2012

Test results for commit 59f329c merged into master
Platform: darwin

Not available for testing: python3.1

Min RK
Owner
minrk commented June 11, 2012

@ellisonbg I don't know what you mean - there's no difference in size between 8 engines and 800. The "Parallel execution on..." wil always fit comfortably on one line. But I added the verbose flag, and all parallel tests pass, despite the appearance of the above report from test_pr.

Fernando Perez
Owner
fperez commented June 11, 2012

Is no-verbose now the default, or verbose?

Min RK
Owner
minrk commented June 11, 2012
Fernando Perez
Owner
fperez commented June 11, 2012

Great, just checking. My vote is then to merge this. It's very valuable and I think from now on it's diminishing returns to keep it in review... What do you think?

Min RK
Owner
minrk commented June 11, 2012
Fernando Perez
Owner
fperez commented June 11, 2012

OK, let's go ahead, so we have a chance to see its effect on master for a day or so before the beta. The closer the beta is to a real RC, the better off we'll be.

Awesome job, and thanks for the patience with all this went through!

Fernando Perez
Owner
fperez commented June 11, 2012

Merging now.

Fernando Perez fperez merged commit 60e6629 into from June 11, 2012
Fernando Perez fperez closed this June 11, 2012
Min RK
Owner
minrk commented June 11, 2012
Min RK minrk deleted the branch March 31, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 27 unique commits by 1 author.

Jun 11, 2012
Min RK Only draw remote tracebacks for IPython.parallel remote exceptions 9c66d11
Min RK use IPython traceback formatting in apply requests ad82d87
Min RK update CompositeError docs with recent changes 2013f1b
Min RK update parallel magic test for exception output with recent change 40ea872
Min RK change %result and add %%px --out foo
%result now only displays the last result, but accepts the display-formatting options.

%%px --out=foo

stores the AsyncResult object for the execution in the user_ns.
8df45e0
Min RK parallel magics are no longer an extension 018b580
Min RK move RemoteError import into showtraceback
avoids circular imports
273f3c7
Min RK remove parallelmagic ext from cluster default config 8293ed3
Min RK update parallel magics
* no longer in extension
* can have multiple active sets of magics with optional suffix
* add pxconfig
* %result only fetches last result, but accepts display-order args
* use magic_arguments decorators
ad8073a
Min RK update Parallel Magics notebook 172fb11
Min RK add note about sync_imports
closes #1598
5fbdbb2
Min RK parallel magics get their own doc file 2b4ceca
Min RK add RMS example for iterating over MapResult bfc0d55
Min RK fill out client.shutdown docstring
closes #1801
fb2dd8e
Min RK View no longer has separate results dict
Only makes it harder to cleanup cache, serves no purpose.
66af5f1
Min RK include errors in IPython.parallel namespace 5887d58
Min RK test %%px --block / --targets 87d1fa8
Min RK add a few pxconfig tests 6036c86
Min RK remove parallelmagic extension from docs d7c3a8d
Min RK publish busy/idle for apply_requests 5f7fdbd
Min RK use idle status message to signal when outputs are complete b49ad41
Min RK update parallel magics docstring d3f1753
Min RK remove now-obsolete output-waits in tests 59a136b
Min RK update Parallel Magics notebook 58b1ac3
Min RK use finite wait_for_outputs d87dacc
Min RK minor tweak to wait_for_idle in tests 56135a5
Min RK add %pxconfig --[no-]verbose 59f329c
Something went wrong with that request. Please try again.