add `%%px --local` for local execution #2753

Merged
merged 4 commits into from Jan 14, 2013

Conversation

Projects
None yet
4 participants
Owner

minrk commented Jan 6, 2013

So that a cell can be run on engines as well as the local namespace.

Prompted by SO Question 14184621.

add `%%px --local` for local execution
So that a cell can be run on engines *as well as* the local namespace.

Prompted by SO Question [14184621](http://stackoverflow.com/questions/14184621).
Contributor

bfroehle commented Jan 6, 2013

Wow, neat idea.

  1. If the command is slow to execute, might it be worthwhile to start processing on the remotes before running locally?
  2. Or, in the current implementation, should we leave a note saying that the command will be first executed locally and will run on the engines only if the local execution succeeds?
Contributor

ahmadia commented Jan 6, 2013

Agreed with @bfroehle. Ideally, the command might return a future from remote execution, starts local execution in a new thread, then waits on both.

Owner

minrk commented Jan 6, 2013

There are advantages to both:

  1. local first - errors will raise immediately, preventing the much more disruptive remote errors
  2. remote first - everything is executing concurrently

@ahmadia - there is really no reason at all to do the local execution in a thread. If we do local execution after remote, all n+1 execuctions will be concurrent.

I chose 1., since I really intended this for interactive definitions, where the first choice is always preferable,
but if people are goind to use it for execution (would be weird, but not unlikely), then I suppose I should change to 2.

%%px --local runs local last
ensures concurrent execution of all n+1 executions
Owner

minrk commented Jan 6, 2013

local is now run after async submission of remote

Contributor

ahmadia commented Jan 6, 2013

@minrk - I suggested that you execute locally in a thread so that in the case the local execution spinlocks but the remote execution does not, you will be able to retrieve the remote results in a "good" state even if you have to signal/interrupt local execution. If you are not using threads to protect against this sort of behavior elsewhere in IPython I agree there's not much point in using one here.

Owner

minrk commented Jan 6, 2013

@ahmadia - yes, it makes sense in the abstract, just not in the Context of IPython, where there is no async local execution.

Contributor

ogrisel commented Jan 9, 2013

+1 for merging the current impl from my point of view.

minrk added a commit that referenced this pull request Jan 14, 2013

Merge pull request #2753 from minrk/pxlocal
add `%%px --local` for local execution

local execution is concurrent with remote.

@minrk minrk merged commit 38c31f8 into ipython:master Jan 14, 2013

1 check passed

default The Travis build passed
Details

@minrk minrk deleted the minrk:pxlocal branch Mar 31, 2014

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request #2753 from minrk/pxlocal
add `%%px --local` for local execution

local execution is concurrent with remote.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment