Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

add `%%px --local` for local execution #2753

Merged
merged 4 commits into from

4 participants

@minrk
Owner

So that a cell can be run on engines as well as the local namespace.

Prompted by SO Question 14184621.

@minrk minrk add `%%px --local` for local execution
So that a cell can be run on engines *as well as* the local namespace.

Prompted by SO Question [14184621](http://stackoverflow.com/questions/14184621).
c2fa6e5
@bfroehle

Wow, neat idea.

  1. If the command is slow to execute, might it be worthwhile to start processing on the remotes before running locally?
  2. Or, in the current implementation, should we leave a note saying that the command will be first executed locally and will run on the engines only if the local execution succeeds?
@ahmadia

Agreed with @bfroehle. Ideally, the command might return a future from remote execution, starts local execution in a new thread, then waits on both.

@minrk
Owner

There are advantages to both:

  1. local first - errors will raise immediately, preventing the much more disruptive remote errors
  2. remote first - everything is executing concurrently

@ahmadia - there is really no reason at all to do the local execution in a thread. If we do local execution after remote, all n+1 execuctions will be concurrent.

I chose 1., since I really intended this for interactive definitions, where the first choice is always preferable,
but if people are goind to use it for execution (would be weird, but not unlikely), then I suppose I should change to 2.

@minrk minrk %%px --local runs local last
ensures concurrent execution of all n+1 executions
753c10d
@minrk
Owner

local is now run after async submission of remote

@ahmadia

@minrk - I suggested that you execute locally in a thread so that in the case the local execution spinlocks but the remote execution does not, you will be able to retrieve the remote results in a "good" state even if you have to signal/interrupt local execution. If you are not using threads to protect against this sort of behavior elsewhere in IPython I agree there's not much point in using one here.

@minrk
Owner

@ahmadia - yes, it makes sense in the abstract, just not in the Context of IPython, where there is no async local execution.

@ogrisel

+1 for merging the current impl from my point of view.

@minrk minrk merged commit 38c31f8 into from
@minrk minrk deleted the branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 6, 2013
  1. @minrk

    add `%%px --local` for local execution

    minrk authored
    So that a cell can be run on engines *as well as* the local namespace.
    
    Prompted by SO Question [14184621](http://stackoverflow.com/questions/14184621).
  2. @minrk

    %%px --local runs local last

    minrk authored
    ensures concurrent execution of all n+1 executions
  3. @minrk
Commits on Jan 10, 2013
  1. @minrk
This page is out of date. Refresh to see the latest.
View
19 IPython/parallel/client/magics.py
@@ -70,6 +70,10 @@ def exec_args(f):
magic_arguments.argument('-t', '--targets', type=str,
help="specify the targets on which to execute",
),
+ magic_arguments.argument('--local', action="store_const",
+ const=True, dest="local",
+ help="also execute the cell in the local namespace",
+ ),
magic_arguments.argument('--verbose', action="store_const",
const=True, dest="set_verbose",
help="print a message at each execution",
@@ -290,14 +294,27 @@ def cell_px(self, line='', cell=None):
if args.targets:
save_targets = self.view.targets
self.view.targets = self._eval_target_str(args.targets)
+ # if running local, don't block until after local has run
+ block = False if args.local else args.block
try:
- return self.parallel_execute(cell, block=args.block,
+ ar = self.parallel_execute(cell, block=block,
groupby=args.groupby,
save_name=args.save_name,
)
finally:
if args.targets:
self.view.targets = save_targets
+
+ # run locally after submitting remote
+ block = self.view.block if args.block is None else args.block
+ if args.local:
+ self.shell.run_cell(cell)
+ # now apply blocking behavor to remote execution
+ if block:
+ ar.get()
+ ar.display_outputs(args.groupby)
+ if not block:
+ return ar
@skip_doctest
def autopx(self, line=''):
View
30 docs/examples/parallel/Parallel Magics.ipynb
@@ -36,7 +36,8 @@
],
"language": "python",
"metadata": {},
- "outputs": []
+ "outputs": [],
+ "prompt_number": 2
},
{
"cell_type": "markdown",
@@ -438,12 +439,35 @@
"outputs": []
},
{
+ "cell_type": "heading",
+ "level": 2,
+ "metadata": {},
+ "source": [
+ "Local Execution"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "As of IPython 0.14, you can instruct `%%px` to also execute the cell locally.\n",
+ "This is useful for interactive definitions,\n",
+ "or if you want to load a data source everywhere,\n",
+ "not just on the engines."
+ ]
+ },
+ {
"cell_type": "code",
"collapsed": false,
- "input": [],
+ "input": [
+ "%%px --local\n",
+ "thispid = os.getpid()\n",
+ "print thispid"
+ ],
"language": "python",
"metadata": {},
- "outputs": []
+ "outputs": [],
+ "prompt_number": 5
}
],
"metadata": {}
Something went wrong with that request. Please try again.