Skip to content
This repository

Add -l option to %R magic to allow passing in of local namespace #2130

Merged
merged 1 commit into from almost 2 years ago

4 participants

Guy Haskin Fernald Thomas Kluyver Bradley M. Froehle Don't Add Me To Your Organization a.k.a The Travis Bot
Guy Haskin Fernald
guyhf commented

Feature request for rmagic: It would be great to be able to pass in local variables to the %R magic so that you can write functions that invoke R with values from the local scope (not just from user_ns). For example:

%load_ext rmagic
import numpy as np
def fun(values):
    for v in values:
        %R -i v -o prob prob <- pnorm(v)
        print prob
fun(np.arange(-1,1,.25))

Right now with %R -i magic you can only get variables from user_ns, so the above code won't work. I have added a %R -l argument that works just like %R -i except that variables are taken from the local scope. So the above code would be written:

%load_ext rmagic
import numpy as np
def fun(values):
    for v in values:
        %R -l v -o prob prob <- pnorm(v)
        print prob
fun(np.arange(-1,1,.25))

And the result would be:

[ 0.15865525]
[ 0.22662735]
[ 0.30853754]
[ 0.40129367]
[ 0.5]
[ 0.59870633]
[ 0.69146246]
[ 0.77337265]

The following changes to rmagic.py would make this work.

guyhf@5448c93

Thomas Kluyver
Collaborator

Rather than adding a separate option, why not make -i input variables look in the local scope first? That's simpler for the user and fits in with how we already use variables.

P.S. It looks like your editor is deleting trailing spaces. If you want to make a pull request, can you disable that, so that all the extra changes don't clutter up the diff? Thanks.

Thomas Kluyver
Collaborator

Ah, I see that you did those things. Github oddity: we get notified of comments, but not of new commits on a pull request, so remember to say something when you update it ;-)

I think this should also have a test - look in IPython.extensions.tests.test_rmagic for examples. It will be a little more complex than that, because it needs to define a function within IPython. This might be one time when it's easiest to use a doctest - see e.g. IPython.core.tests.test_magic.doctest_time.

Guy Haskin Fernald
guyhf commented
Thomas Kluyver
Collaborator

Test results for commit 1dea0ba merged into master
Platform: linux2

  • python2.7: OK (libraries not available: oct2py rpy2)
  • python3.2: OK (libraries not available: cython oct2py pymongo rpy2 wx wx.aui)

Not available for testing: python2.6

Bradley M. Froehle
Collaborator

Looks pretty good to me, but I don't use R regularly enough to test this.

Outside of the scope of this pull request, I must comment that the @needs_local_scope decorator is a little clumsy. It'd be nicer if it just passed in local_ns as a keyword argument. However at least it doesn't directly conflict with the cell magics (as I guess there isn't a sense of a local namespace in the cell magics).

Thomas Kluyver
Collaborator
Bradley M. Froehle
Collaborator

Yes, I don't think we should parse the function signature, just that the namespace should be passed in the kwargs instead of the args, making usage a simpler:

@needs_local_scope 
...
def magic_function(line, cell=None, local_ns=None):
    ...

But this is just bike shedding and not related to this issue.

Thomas Kluyver
Collaborator

Ah, that makes sense, then. Yes, that's probably a good idea.

Guy Haskin Fernald

I modified the needs_local_scope decorator so that it passes in a local_ns kwarg instead. This makes for a cleaner interface. I tested it with R, but I don't use octave so I couldn't run those tests.

IPython/extensions/octavemagic.py
((8 lines not shown))
248 253 if args.input:
249 254 for input in ','.join(args.input).split(','):
250 255 input = unicode_to_str(input)
251   - self._oct.put(input, self.shell.user_ns[input])
  256 + self._oct.put(input, local_ns.get(input, self.shell.user_ns.get(input, None)))
1
Thomas Kluyver Collaborator
takluyver added a note

There's a change in behaviour here if the user specifies an input variable that doesn't exist - before it would fail with a KeyError, now it will use None (or the octave equivalent) instead. I think failing is right, because it probably means the user mistyped something. It might take a couple of extra lines to get it to fail properly, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/extensions/rmagic.py
((8 lines not shown))
496 502 if args.input:
497 503 for input in ','.join(args.input).split(','):
498   - self.r.assign(input, self.pyconverter(self.shell.user_ns[input]))
  504 + self.r.assign(input, self.pyconverter(local_ns.get(input, self.shell.user_ns.get(input, None))))
1
Thomas Kluyver Collaborator
takluyver added a note

As above, I think it should fail if the input variable doesn't exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Guy Haskin Fernald

That makes sense. I just switched it from user_ns.get() to back to user_ns[] will cause the same KeyError as before.

Thomas Kluyver
Collaborator

Except I think that will throw the error if you pass a variable that only exists in the local namespace, because the second argument to .get() is evaluated unconditionally. I still think it will take a couple of extra lines to do it correctly. Something like:

try:
    val = local_ns[input]
except KeyError:
    val = self.shell.user_ns[input]
self.r.assign(input, self.pyconverter(val))

(Python 3.3 has a handy ChainMap class that will make this sort of thing easier, but for now we have to do it the long way.)

Guy Haskin Fernald

Thanks. I fixed that and added some doctests to cover that case.

IPython/extensions/tests/test_rmagic.py
((18 lines not shown))
  77 + ...: %R -i u -o result result <- u+1
  78 + ...: return result[0]
  79 + ...:
  80 +
  81 +In [6]: test_rmagic_localscope(1) == 2
  82 +Out[6]: True
  83 +
  84 +In [7]: e = None
  85 +
  86 +In [8]: try:
  87 + ...: %R -i var_not_defined -o result result = var_not_defined
  88 + ...: except Exception as e:
  89 + ...: pass
  90 + ...:
  91 +
  92 +In [9]: isinstance(e, KeyError)
1
Thomas Kluyver Collaborator
takluyver added a note

I think this will fail in Python 3 - e only exists within the except clause, so it's a NameError to refer to it here. This is the sort of thing I don't like about doctests - they're quite brittle.

Ideally, we could convert this into a standard unit test - use ip.run_cell to define the function, ip.run_line_magic to call %R and nt.assert_raises to check that this bit throws a KeyError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/extensions/tests/test_octavemagic.py
((17 lines not shown))
  78 +In [5]: def test_octavemagic_localscope(u):
  79 + ...: %octave -i u -o result result = u + 1
  80 + ...: return result
  81 + ...:
  82 +
  83 +In [6]: result_ls = test_octavemagic_localscope(1)
  84 +result = 2
  85 +
  86 +In [7]: result_ls == 2
  87 +Out[7]: True
  88 +
  89 +In [8]: e = None
  90 +
  91 +In [9]: try:
  92 + ...: %octave -i var_not_defined -o result result = var_not_defined
  93 + ...: except Exception as e:
1
Bradley M. Froehle Collaborator
bfroehle added a note

Alternatively I think you can do this in a Python 3 portable way as:

except:
    e = sys.exc_info()[1]

(And add import sys to the top of the file if necessary.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Bradley M. Froehle
Collaborator

Pinging @guyhf. Just a few more quick changes here and I think this is ready to be committed.

Guy Haskin Fernald

Just updating it now. I'll convert the KeyError exception test to a standard unit test.

Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request passes (merged 80f3b364 into 3c8d448).

Guy Haskin Fernald

I changed the KeyError test to a unit test and tested both IPython.extensions.tests.test_rmagic and IPython.extensions.tests.test_octavemagic. I'm fine changing the other doctests to unit tests if that is what is preferred.

Thomas Kluyver
Collaborator

That looks fine.

I prefer unit tests because I've often had to fight with both doctests and the machinery that runs them (if you ever want a free headache, look at IPython/testing/plugin/ipdoctest.py). But I know they're easier to write, so as long as they work, there's no requirement to change them.

@bfroehle: I'll let you have another look at this, but if you're happy, you can merge it.

Guy Haskin Fernald

I went ahead and changed the rest of the doctests to unit tests and re-checked IPython.extensions.tests.test_rmagic and IPython.extensions.tests.test_octavemagic.

Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request passes (merged 5684165a into 3c7c235).

IPython/extensions/tests/test_octavemagic.py
... ... @@ -62,3 +62,24 @@ def test_octave_plot():
62 62 'plot([1, 2, 3]); figure; plot([4, 5, 6]);')
63 63
64 64 nt.assert_equal(test_octave_plot.svgs_generated, 2)
  65 +
  66 +def test_octavemagic_localscope():
  67 + ip = get_ipython()
  68 + ip.magic('load_ext octavemagic')
  69 + ip.push({'x':0})
  70 + ip.run_line_magic('octave', '-i x -o result result = x+1')
  71 + result = ip.user_ns['result']
  72 + np.testing.assert_equal(result, 1)
  73 +
  74 + ip.run_cell('''def test_octavemagic_localscope(u):
1
Thomas Kluyver Collaborator
takluyver added a note

Let's call this function something different from the test, to avoid confusion. Also, while we're at it, let's not have 'test' in the name, just in case nose somehow finds it (it attempts to run any function it finds named 'test').

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/extensions/tests/test_rmagic.py
... ... @@ -60,3 +60,23 @@ def test_cell_magic():
60 60 ip.run_cell_magic('R', '-i x,y -o r,xc a=lm(y~x)', snippet)
61 61 np.testing.assert_almost_equal(ip.user_ns['xc'], [3.2, 0.9])
62 62 np.testing.assert_almost_equal(ip.user_ns['r'], np.array([-0.2, 0.9, -1. , 0.1, 0.2]))
  63 +
  64 +
  65 +def test_rmagic_localscope():
  66 + ip.push({'x':0})
  67 + ip.run_line_magic('R', '-i x -o result result <-x+1')
  68 + result = ip.user_ns['result']
  69 + np.testing.assert_equal(result[0], 1)
  70 +
  71 + ip.run_cell('''def test_rmagic_localscope(u):
1
Thomas Kluyver Collaborator
takluyver added a note

Same with this one, please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Bradley M. Froehle
Collaborator

Test results for commit 5684165 merged into master (3c7c235)
Platform: linux2

  • python2.6: OK (libraries not available: azure matplotlib oct2py pymongo qt rpy2 wx wx.aui)
  • python2.7: OK (libraries not available: azure)
  • python3.2: OK (libraries not available: azure oct2py pymongo rpy2 wx wx.aui)

Not available for testing:

Bradley M. Froehle
Collaborator

With those small changes that @takluyver suggests in the tests, I think it's ready to go.

Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request passes (merged 7d7a7118 into 3c7c235).

Guy Haskin Fernald

I renamed the internal test functions.

Bradley M. Froehle bfroehle commented on the diff
IPython/extensions/octavemagic.py
((16 lines not shown))
248 251 if args.input:
249 252 for input in ','.join(args.input).split(','):
250 253 input = unicode_to_str(input)
251   - self._oct.put(input, self.shell.user_ns[input])
  254 + try:
  255 + val = local_ns[input]
  256 + except KeyError:
  257 + val = self.shell.user_ns[input]
1
Bradley M. Froehle Collaborator
bfroehle added a note

Just occurred to me that this is more-or-less equivalent to eval(input, self.shell.user_ns, local_ns) (with the exception that it'll raise a NameError instead of a KeyError if the key cannot be found).

But I think I like the extended try/except block better. Certainly it's faster to execute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Bradley M. Froehle bfroehle commented on the diff
IPython/extensions/tests/test_octavemagic.py
... ... @@ -62,3 +62,24 @@ def test_octave_plot():
62 62 'plot([1, 2, 3]); figure; plot([4, 5, 6]);')
63 63
64 64 nt.assert_equal(test_octave_plot.svgs_generated, 2)
  65 +
  66 +def test_octavemagic_localscope():
  67 + ip = get_ipython()
  68 + ip.magic('load_ext octavemagic')
1
Bradley M. Froehle Collaborator
bfroehle added a note

If we make any other changes this should probably be changed to ip.run_line_magic('load_ext', 'octavemagic'), but there are enough other instances of ip.magic in the test routines where it's clearly not actually an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/extensions/tests/test_octavemagic.py
... ... @@ -62,3 +62,24 @@ def test_octave_plot():
62 62 'plot([1, 2, 3]); figure; plot([4, 5, 6]);')
63 63
64 64 nt.assert_equal(test_octave_plot.svgs_generated, 2)
  65 +
  66 +def test_octavemagic_localscope():
  67 + ip = get_ipython()
  68 + ip.magic('load_ext octavemagic')
  69 + ip.push({'x':0})
  70 + ip.run_line_magic('octave', '-i x -o result result = x+1')
  71 + result = ip.user_ns['result']
  72 + np.testing.assert_equal(result, 1)
2
Bradley M. Froehle Collaborator
bfroehle added a note

Is there a reason why we are using numpy to do the equality testing?

Guy Haskin Fernald
guyhf added a note

I was following the lead of the earlier tests. This one doesn't need it though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Bradley M. Froehle
Collaborator

Great. I think this could be merged now. :)

Guy Haskin Fernald guyhf Added functionality to %R and %octave magics so that -i first looks i…
…n local

scope before looking in user_ns.  This allows the magics to be called from
within functions.

Also Modified @needs_local_scope decorator for magic extensions to use a
kwarg with the local namespace, so functions now get a separate kward
local_ns, e.g.

def magic_function(line, cell=None, local_ns=None):
    ...
99483a2
Guy Haskin Fernald

Switched np.testing.assert_equals to nt.assert_equals.

Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request passes (merged 99483a2 into 3c7c235).

Thomas Kluyver
Collaborator

Thanks, @guyhf .

@bfroehle : I'm happy with this. You can push the green button if you don't see any other issues.

Bradley M. Froehle bfroehle merged commit 5308c36 into from
Bradley M. Froehle
Collaborator

@guyhf Merged! Thanks for all your work and all your patience. I think we ended up with an excellent result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Aug 21, 2012
Guy Haskin Fernald guyhf Added functionality to %R and %octave magics so that -i first looks i…
…n local

scope before looking in user_ns.  This allows the magics to be called from
within functions.

Also Modified @needs_local_scope decorator for magic extensions to use a
kwarg with the local namespace, so functions now get a separate kward
local_ns, e.g.

def magic_function(line, cell=None, local_ns=None):
    ...
99483a2
This page is out of date. Refresh to see the latest.
5 IPython/core/interactiveshell.py
@@ -2088,11 +2088,12 @@ def run_line_magic(self, magic_name, line):
2088 2088 magic_arg_s = self.var_expand(line, stack_depth)
2089 2089 # Put magic args in a list so we can call with f(*a) syntax
2090 2090 args = [magic_arg_s]
  2091 + kwargs = {}
2091 2092 # Grab local namespace if we need it:
2092 2093 if getattr(fn, "needs_local_scope", False):
2093   - args.append(sys._getframe(stack_depth).f_locals)
  2094 + kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
2094 2095 with self.builtin_trap:
2095   - result = fn(*args)
  2096 + result = fn(*args,**kwargs)
2096 2097 return result
2097 2098
2098 2099 def run_cell_magic(self, magic_name, line, cell):
6 IPython/core/magics/execution.py
@@ -812,7 +812,7 @@ def timeit(self, line='', cell=None):
812 812 @skip_doctest
813 813 @needs_local_scope
814 814 @line_magic
815   - def time(self,parameter_s, user_locals):
  815 + def time(self,parameter_s, local_ns=None):
816 816 """Time execution of a Python statement or expression.
817 817
818 818 The CPU and wall clock times are printed, and the value of the
@@ -884,11 +884,11 @@ def time(self,parameter_s, user_locals):
884 884 wall_st = wtime()
885 885 if mode=='eval':
886 886 st = clock2()
887   - out = eval(code, glob, user_locals)
  887 + out = eval(code, glob, local_ns)
888 888 end = clock2()
889 889 else:
890 890 st = clock2()
891   - exec code in glob, user_locals
  891 + exec code in glob, local_ns
892 892 end = clock2()
893 893 out = None
894 894 wall_end = wtime()
17 IPython/extensions/octavemagic.py
@@ -45,7 +45,7 @@
45 45
46 46 from IPython.core.displaypub import publish_display_data
47 47 from IPython.core.magic import (Magics, magics_class, line_magic,
48   - line_cell_magic)
  48 + line_cell_magic, needs_local_scope)
49 49 from IPython.testing.skipdoctest import skip_doctest
50 50 from IPython.core.magic_arguments import (
51 51 argument, magic_arguments, parse_argstring
@@ -182,12 +182,13 @@ def octave_pull(self, line):
182 182 help='Plot format (png, svg or jpg).'
183 183 )
184 184
  185 + @needs_local_scope
185 186 @argument(
186 187 'code',
187 188 nargs='*',
188 189 )
189 190 @line_cell_magic
190   - def octave(self, line, cell=None):
  191 + def octave(self, line, cell=None, local_ns=None):
191 192 '''
192 193 Execute code in Octave, and pull some of the results back into the
193 194 Python namespace.
@@ -237,18 +238,24 @@ def octave(self, line, cell=None):
237 238 if cell is None:
238 239 code = ''
239 240 return_output = True
240   - line_mode = True
241 241 else:
242 242 code = cell
243 243 return_output = False
244   - line_mode = False
245 244
246 245 code = ' '.join(args.code) + code
247 246
  247 + # if there is no local namespace then default to an empty dict
  248 + if local_ns is None:
  249 + local_ns = {}
  250 +
248 251 if args.input:
249 252 for input in ','.join(args.input).split(','):
250 253 input = unicode_to_str(input)
251   - self._oct.put(input, self.shell.user_ns[input])
  254 + try:
  255 + val = local_ns[input]
  256 + except KeyError:
  257 + val = self.shell.user_ns[input]
  258 + self._oct.put(input, val)
252 259
253 260 # generate plots in a temporary directory
254 261 plot_dir = tempfile.mkdtemp()
18 IPython/extensions/rmagic.py
@@ -53,7 +53,7 @@
53 53
54 54 from IPython.core.displaypub import publish_display_data
55 55 from IPython.core.magic import (Magics, magics_class, cell_magic, line_magic,
56   - line_cell_magic)
  56 + line_cell_magic, needs_local_scope)
57 57 from IPython.testing.skipdoctest import skip_doctest
58 58 from IPython.core.magic_arguments import (
59 59 argument, magic_arguments, parse_argstring
@@ -344,8 +344,9 @@ def Rget(self, line):
344 344 'code',
345 345 nargs='*',
346 346 )
  347 + @needs_local_scope
347 348 @line_cell_magic
348   - def R(self, line, cell=None):
  349 + def R(self, line, cell=None, local_ns=None):
349 350 '''
350 351 Execute code in R, and pull some of the results back into the Python namespace.
351 352
@@ -482,7 +483,8 @@ def R(self, line, cell=None):
482 483
483 484 # arguments 'code' in line are prepended to
484 485 # the cell lines
485   - if not cell:
  486 +
  487 + if cell is None:
486 488 code = ''
487 489 return_output = True
488 490 line_mode = True
@@ -493,9 +495,17 @@ def R(self, line, cell=None):
493 495
494 496 code = ' '.join(args.code) + code
495 497
  498 + # if there is no local namespace then default to an empty dict
  499 + if local_ns is None:
  500 + local_ns = {}
  501 +
496 502 if args.input:
497 503 for input in ','.join(args.input).split(','):
498   - self.r.assign(input, self.pyconverter(self.shell.user_ns[input]))
  504 + try:
  505 + val = local_ns[input]
  506 + except KeyError:
  507 + val = self.shell.user_ns[input]
  508 + self.r.assign(input, self.pyconverter(val))
499 509
500 510 png_argdict = dict([(n, getattr(args, n)) for n in ['units', 'height', 'width', 'bg', 'pointsize']])
501 511 png_args = ','.join(['%s=%s' % (o,v) for o, v in png_argdict.items() if v is not None])
21 IPython/extensions/tests/test_octavemagic.py
@@ -62,3 +62,24 @@ def test_octave_plot():
62 62 'plot([1, 2, 3]); figure; plot([4, 5, 6]);')
63 63
64 64 nt.assert_equal(test_octave_plot.svgs_generated, 2)
  65 +
  66 +def test_octavemagic_localscope():
  67 + ip = get_ipython()
  68 + ip.magic('load_ext octavemagic')
  69 + ip.push({'x':0})
  70 + ip.run_line_magic('octave', '-i x -o result result = x+1')
  71 + result = ip.user_ns['result']
  72 + nt.assert_equal(result, 1)
  73 +
  74 + ip.run_cell('''def octavemagic_addone(u):
  75 + %octave -i u -o result result = u+1
  76 + return result''')
  77 + ip.run_cell('result = octavemagic_addone(1)')
  78 + result = ip.user_ns['result']
  79 + nt.assert_equal(result, 2)
  80 +
  81 + nt.assert_raises(
  82 + KeyError,
  83 + ip.run_line_magic,
  84 + "octave",
  85 + "-i var_not_defined 1+1")
20 IPython/extensions/tests/test_rmagic.py
@@ -60,3 +60,23 @@ def test_cell_magic():
60 60 ip.run_cell_magic('R', '-i x,y -o r,xc a=lm(y~x)', snippet)
61 61 np.testing.assert_almost_equal(ip.user_ns['xc'], [3.2, 0.9])
62 62 np.testing.assert_almost_equal(ip.user_ns['r'], np.array([-0.2, 0.9, -1. , 0.1, 0.2]))
  63 +
  64 +
  65 +def test_rmagic_localscope():
  66 + ip.push({'x':0})
  67 + ip.run_line_magic('R', '-i x -o result result <-x+1')
  68 + result = ip.user_ns['result']
  69 + nt.assert_equal(result[0], 1)
  70 +
  71 + ip.run_cell('''def rmagic_addone(u):
  72 + %R -i u -o result result <- u+1
  73 + return result[0]''')
  74 + ip.run_cell('result = rmagic_addone(1)')
  75 + result = ip.user_ns['result']
  76 + nt.assert_equal(result, 2)
  77 +
  78 + nt.assert_raises(
  79 + KeyError,
  80 + ip.run_line_magic,
  81 + "R",
  82 + "-i var_not_defined 1+1")

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.