FIx Parallel Magics for newparallel #341

Closed
minrk opened this Issue Apr 8, 2011 · 8 comments

Comments

Projects
None yet
3 participants
@minrk
Member

minrk commented Apr 8, 2011

The parallel magics did work with the new code, but I didn't touch the magic, as they were still for IPython.kernel. Now that IPython.kernel is out of the way, I should give the parallelmagic an update, to make it better match newparallel.

Also, %autopx is now broken, I believe due to the recent AST merge. %autopx hijacked run_source, but as far as I can tell, that's never called anymore.

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver Apr 8, 2011

Member

Yes, the chain of calls for executing user code now goes run_cell ->
run_ast_nodes -> run_code. I think it's run_cell that you'll need to set up
a modified version of - that's the only one that expects code as a string.

Member

takluyver commented Apr 8, 2011

Yes, the chain of calls for executing user code now goes run_cell ->
run_ast_nodes -> run_code. I think it's run_cell that you'll need to set up
a modified version of - that's the only one that expects code as a string.

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez Apr 8, 2011

Member

On Fri, Apr 8, 2011 at 12:26 PM, minrk
reply@reply.github.com
wrote:

Also, %autopx is now broken, I believe due to the recent AST merge.  %autopx hijacked run_source, but as far as I can tell, that's never called anymore.

I think you can use run_cell now. It didn't exist before but it's
actually much more robust than run_source. That should do the trick.

Member

fperez commented Apr 8, 2011

On Fri, Apr 8, 2011 at 12:26 PM, minrk
reply@reply.github.com
wrote:

Also, %autopx is now broken, I believe due to the recent AST merge.  %autopx hijacked run_source, but as far as I can tell, that's never called anymore.

I think you can use run_cell now. It didn't exist before but it's
actually much more robust than run_source. That should do the trick.

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 8, 2011

Member

Thanks, I think I've figured it out enough for interactive use. However, it doesn't work from inside %run or irunner.

If I have a script containing:

%autopx
a=5
%autopx

%autopx should hijack execution until %autopx is called again, but this doesn't happen. I believe this is because the whole file gets run inside a single run_cell. Is there not a way to hijack individual lines anymore?

It looks like hijacking run_code is my only option, but I'm pretty sure I need the source, not the code objects.

This is a general issue. I expect there are several behaviors no longer consistent between %run/irunner and interactive execution, when manipulating the ipython object.

Member

minrk commented Apr 8, 2011

Thanks, I think I've figured it out enough for interactive use. However, it doesn't work from inside %run or irunner.

If I have a script containing:

%autopx
a=5
%autopx

%autopx should hijack execution until %autopx is called again, but this doesn't happen. I believe this is because the whole file gets run inside a single run_cell. Is there not a way to hijack individual lines anymore?

It looks like hijacking run_code is my only option, but I'm pretty sure I need the source, not the code objects.

This is a general issue. I expect there are several behaviors no longer consistent between %run/irunner and interactive execution, when manipulating the ipython object.

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver Apr 8, 2011

Member

Yes, I think it is because it's all getting run in one call. Although I
don't think that's just a result of the AST stuff - we were already grouping
together blocks to send to run_source. When we're %run-ing a script, we
really need to treat each node as a separate cell, so that they affect the
execution of nodes after them. Can you send AST nodes over the wire? A quick
test suggests that they are pickle-able. Unfortunately, I think the
alternative is to resurrect split_blocks...

Member

takluyver commented Apr 8, 2011

Yes, I think it is because it's all getting run in one call. Although I
don't think that's just a result of the AST stuff - we were already grouping
together blocks to send to run_source. When we're %run-ing a script, we
really need to treat each node as a separate cell, so that they affect the
execution of nodes after them. Can you send AST nodes over the wire? A quick
test suggests that they are pickle-able. Unfortunately, I think the
alternative is to resurrect split_blocks...

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 9, 2011

Member

If they are pickleable, then I think I should be fine. Of course, it also requires that I can execute code that would raise NameErrors locally.

Member

minrk commented Apr 9, 2011

If they are pickleable, then I think I should be fine. Of course, it also requires that I can execute code that would raise NameErrors locally.

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver Apr 9, 2011

Member

OK, so then you (hopefully) just need to override run_ast_nodes. We can
generate nodes regardless of the names in the code. I'll have a think about
the best way to get .ipy scripts running line-by-line.

Member

takluyver commented Apr 9, 2011

OK, so then you (hopefully) just need to override run_ast_nodes. We can
generate nodes regardless of the names in the code. I'll have a think about
the best way to get .ipy scripts running line-by-line.

@ghost ghost assigned minrk Apr 10, 2011

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 10, 2011

Member

Splitting run_ast_nodes so that it calls run_code for each node, rather than for the whole set, seems to fix much of the issue. Then, all I need to do is override run_code, and it works.

See here:
minrk/ipython@ipython:master...minrk:parallelmagicdiff-0

Member

minrk commented Apr 10, 2011

Splitting run_ast_nodes so that it calls run_code for each node, rather than for the whole set, seems to fix much of the issue. Then, all I need to do is override run_code, and it works.

See here:
minrk/ipython@ipython:master...minrk:parallelmagicdiff-0

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 11, 2011

Member

closed by e68cf3a

Member

minrk commented Apr 11, 2011

closed by e68cf3a

@minrk minrk closed this Apr 11, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment