Skip to content
This repository

Function annotation based hooks into the tab completion system #2636

Closed
wants to merge 22 commits into from

3 participants

Robert McGibbon Aaron Meurer Thomas Kluyver
Robert McGibbon

NOT READY FOR MERGING

This PR adds a way for user code to hook into the ipython tab completion system via function annotations PEP 3107. python 2.x support is provided via a decorator.

This feature was discussed on IPython-dev along with the use of function annotations for the feature on Python-ideas as it relates to interoperability between annotation schemes.

For example, using the glob tab completion, only files matching the appropriate ending (and directories) are shown:

>>> from IPython.extensions.annotations import tab_glob
>>> def f(x : tab_glob('*.txt')):
...    pass
>>> load(<TAB_CHARACTER>
'COPYING.txt'        'build/'             'ipython.egg-info/'  'setupext/'          
'IPython/'           'docs/'              'scripts/'           'tools/'   

Currently, the annotations provided are

  • tab_glob, which recommends string literals that are directories or files matching a pattern
  • tab_literal, which recommends from a set of enumerated literals
  • tab_instance, which recommends objects that exist in the current namespace that are instances of a supplied (set of) class(es).

The API for adding new annotations is pretty simple (the implementation of tab_instance is ~15 LOC), so users can easily both add these annotations to their functions or add new annotations that implement other types of matches.

I'm sure their are some bugs. One that I know about is poor handling by tab_glob of filenames with spaces, since the spaces cause the tokenizer to token-break where you don't want it to.

This feature is complicated enough to require a significant number of tests (which I haven't written yet -- I've just been testing it interactively). Testing it is a challenge in and of itself, since you need to somehow simulate the TAB key in the testing environment and parse the output of readline. If anyone has any ideas for how to test the final product automatically, that would be awesome. Obviously the methods can be tested individually, which is straightforward.

[Also, the completion machinery currently in ipython could use some serious refactoring, but I didn't try to tackle that here]


I'm not really sure what the next step forward on this is. It's a big enough change, IMO, that it deserves some serious discussion, user-facing documentation (IMHO I wrote good docstrings, but that's not enough), etc.

Also, I don't really like the names tab_glob, tab_literal, and tab_instance.

Thoughts?

Robert McGibbon

grrr. doctest doesn't like my examples that try to show what happens when you enter the tab key. I want to just write "TAB" to show what happens, but that doesn't work w.r.t doctests.

Aaron Meurer

Shouldn't they have IPython In[1]: instead of >>>?

Robert McGibbon

This is from the reply I sent to the Python-ideas thread that @takluyver started on function annotations, but it should probably go here too.

Here's the scheme that this code is using to check the annotations for tab completion information:

1) Check if the function in question has an annotation.

  • If not, bail out and skip this feature. No harm, no foul.
  • If so, go to (2).

2) Check if the annotation is one of our objects / implements our interface.

  • If so, fantastic! Start invoking this feature.
  • If not, go to (3).

3) Check if the annotation is iterable, and if it is iterable check if any of the items are one of our objects or implement our interface.

  • If so, fantastic! Start invoking this feature.
  • If not, bail out and skip this feature.

(An obvious optimization is skipping step (3) if the annotation is a string.)

One thing that I'm not sure about is whether the check for whether an annotations is ours is by looking for it to subclass an (abstract) baseclass extensions/annotations.py:AnnotationCompleterBase, by using hasattr to check for the method tab_matches implements the behavior one of these tab matching annotations needs to provide, or by simply trying to call obj.tab_matches inside of a try/catch block. I've read some discussion that hasattr is broken by design, but I don't know if that's relevant to this issue.

Aaron Meurer

I wouldn't call that broken by design. It was just originally written with the equivalent of except: instead of except AttributeError:. This was changed in Python 3. I don't think it affects us here. The worst that can happen is that it will mask an error in someone else's code, but actually, when tab completing, we might even want to mask all errors (or maybe not. It should be discussed).

Robert McGibbon

If you keep reading the thread, the discussion morphs into the fact that hasattr isn't really the static introspection that you might think it is (I did). (But I don't really care about hasattr, and I'd rather not get bogged down.)

Maybe the bigger issue is Nick Coglan's comments on python-ideas, saying that it would be better to use a decorator in addition to the annotation.

The syntax that he's recommending is something like:

>>> @tab_expansion
>>> def foo(filename : glob_expansion('*.txt')):
...    pass

Where the @tab_expansion decorator moves the glob_expansion information from __annotations__ into an ipython specific metadata location.

As Nick said:

Mixing annotations intended for different consumers is a fundamentally bad idea, as it encourages unreadable code and complex dances to avoid stepping on each other's toes. It's better to design a separate API that supports composition by passing the per-parameter details directly to a decorator factory (which then adds appropriate named attributes to the function), with annotations used just as syntactic sugar for simple cases where no composition is involved.

I am inclined to follow this advice, because he (1) makes a very good point and (2) is a python core developer.

Aaron Meurer

If you keep reading the thread, the discussion morphs into the fact that hasattr isn't really the static introspection that you might think it is (I did).

No, almost nothing in Python is static. But that's the way the language works.

Where the @tab_expansion decorator moves the glob_expansion information from annotations into an ipython specific metadata location.

We'll probably want to move it elsewhere at some point anyway, for caching purposes, because tab completion is one of those things that you want to be absolutely instantaneous, or else it significantly degrades the user experience.

I am inclined to follow this advice, because he (1) makes a very good point and (2) is a python core developer.

Yes, this is a good point.

Here are some other thoughts I've had:

  • We might want a way to "decorate a whole module", so to speak. That is to say, if you know that every function in a module acts the same way, you should be able to say that in one fell swoop.

  • Does your code work for classes? It will need to look at __new__ or __init__.

  • We also need to look at return types. There is annotations syntax for this as well. I think this case is easier, because we really just need to annotate the type of the object being returned. It is only needed for things like f(x).<TAB>. I guess eventually it can be made to be smarter for functions whose return type depends on the input.

  • Combining the last two points, return types for classes that define __new__.

My personal use case is SymPy. A lot of SymPy's API uses method chaining, which is annoying since it can't be tab completed unless you do it in steps (i.e., set the object to a variable, and then call the method on that variable). For my first point, almost all public facing SymPy functions and classes take in Expr objects as input, and give them as output. So it would be nice to just tell the completer to compete Exprs methods on basically everything, with as little work as possible on the part of me (the SymPy developer).

Aside from method chaining, the case where I personally find myself wishing for better tab completion is not in the types of the arguments, but in the names of the arguments themselves, i.e., keyword arguments. When the keyword arguments are explicitly named in the function definition, the tab completer should be able to get that information straightaway, using whatever functions from the inspect module. When they are given as **kwargs, they will have to be annotated.

Robert McGibbon

Thanks so much for looking over this closely. Your suggestions are top notch and really appreciated!

In no particular order:

  • It would be pretty straightforward to do a class decorator that would translate the annotations for every method. (Or a metaclass, but I prefer class decorators when the extra features of metaclasses are not required). I will add this.
  • It should be possible to run the "decorator" on every function in a module, either by putting something at the bottom of the file that references dir(sys.modules[__name__]) I've always felt like putting influential code at the bottom of a file is kind of hiding it -- it would be nicer at the top. Perhaps a sympy __init__.py could invoke the decorators before the functions are actually imported?

Thanks for mentioning SymPy and your use case, because I was thinking about the feature rather narrowly, since it was (in my head) just for my own uses.

  • f(x).<TAB> is a good idea. It would really be another feature, since the current PR only deals with lines that have an unclosed parentheses, but that's fine. Given that I've spent a few-tens-of-hours with the completer codebase, I know just how to do it (assuming that the syntax for passing in the type annotations is settled).
  • Building on the previous point, how should we handle foo(<TAB> when foo's first argument is known to take a certain type and a function bar is known to return instances of that same type. Currently only objects that are instances of that type would be recommended -- should bar and other functions known to return the type also be recommended?
  • The current code works for methods on classes, but not for class constructors/initializers. I forgot about that and will add it.

When the keyword arguments are explicitly named in the function definition, the tab completer should be able to get that information straightaway

  • I think this is already done (e.g. In[9), but I agree that it could be better. This is the current behavior.
In [7]: def foo(longname1=1, longname2=2):
   ...:     pass
   ...:    

In [8]: def bar(longname1, longname2):
   ...:     pass
   ...: 

In [9]: foo(longn<TAB>
longname1=  longname2= 

In [10]: bar(longn<TAB>
[system bell rings and no completions offered]

Perhaps we could do better by adding keyword arg complete such that In[10] would complete with longname1=?

Thomas Kluyver
Collaborator
Aaron Meurer
Robert McGibbon

Thanks so much for looking over this closely. Your suggestions are top notch and really appreciated!

Actually, I didn't look over the code closely. I'm leaving that to the
ipython devs, who know what to look for. I am more concerned about the
public API, as that is how I would be using it.

But frankly, the implementation is easy. Getting public API "right" is the hard part.


This just goes to show that we do need to register the completion metadata globally.

I'm not really sure what you mean by this. It seems logical to me that any return value annotation / tab completion info will be stored in the same location the argument annotation is -- that is, attached to the function. Perhaps in func.__annotations__, or perhaps in func.__tab_completions__ or perhaps in some other attribute.

Conceptually, I don't see much difference between the function arguments and the return value.


keyword argument tab completion has been the behavior since at least 0.12.1

$ ipython
Python 2.7.3 (default, Jun 29 2012, 17:56:12) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: def bar(longname1=1, longname2=2):
   ...:     pass
   ...: 

In [2]: bar(longn
longname1=  longname2=  
Robert McGibbon

@takluyver yeah, you're right. I wasn't trying to say that it should be part of the IPython API. I just meant that if @asmeurer wanted to decorate all of his functions at once, he'd have options.

The decorator will be conceptually like:

def tab_complete(func):
     # move the tab completion annotations into an ipython specific location
     func.__tab_completions = func.__annotations__

So that the normal usage is

from IPython.core.annotations import tab_complete, tab_instance
# I don't really like the name "tab_instance". I should think of something better
@tab_complete
def sympy_addition(x : tab_instance(Expr), y : tab_instance(Expr)):
    pass

But if @asmeurer wants to, he can avoid manually decorating all of his functions and do a "spooky" action at a distance decorating at the end of his module like

for fname in dir(sys.modules[__name__]):
    # you wound need some more checking in practice...
    # but you get the idea
    tab_complete(eval(f))
Aaron Meurer

keyword argument tab completion has been the behavior since at least 0.12.1

Did not know that. I guess I should try pressing tab more often, and see what works.

Robert McGibbon

I've got a lot of good ideas from this thread and from the python-ideas emails. But I don't think this code is really ready yet. I'm going to work on it more, and then I'll resubmit the PR when it's ready.

Robert McGibbon rmcgibbo closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 22 unique commits by 1 author.

Nov 06, 2012
Robert McGibbon Fixed rmagic/RPush issue #2550 32b56cd
Robert McGibbon added tests for new functionality 463df4b
Robert McGibbon changed to raw string in re.split e7c6f46
Nov 29, 2012
Robert McGibbon Merge branch 'master' of github.com:ipython/ipython 9e52d69
Robert McGibbon first commit 3507544
Nov 30, 2012
Robert McGibbon Code is working. Added documentation 4bec04f
Robert McGibbon added the user facing decorator to the extensions directory 4ba8f71
Robert McGibbon added comment with plans 70e9f9c
Robert McGibbon moved some of the functionality into the annotations 2774af3
Robert McGibbon isinstance tab completion working 3422400
Robert McGibbon I think its pretty much working. I just need to add some more docstrings 39f8544
Dec 01, 2012
Robert McGibbon Simplified code. Small bug remains with named args 4fae353
Robert McGibbon added more docstrings 74f3387
Robert McGibbon syntax fix 1fbff08
Robert McGibbon docstrings on tokenizer 0aa2bda
Robert McGibbon reverted changes to rmagic that are part of another PR and accidental…
…ly included in this branch
00cbe9a
Robert McGibbon reverted changes to rmagic that are part of another PR and accidental…
…ly included in this branch
f240661
Robert McGibbon Support for class methods and bugfix if the user enters more argument…
…s than the function actually takes
ea52ece
Robert McGibbon typo 0265de8
Robert McGibbon prevent doctest from running examples in annotations.py that show the…
… entering of the tab literal
be27ffe
Robert McGibbon Changed '>>>' to 'In[n]' in the docstrings's examples, per @asmeurer'…
…s comment
dc4804d
Dec 03, 2012
Robert McGibbon In the process of changing to decorator that wraps the annotations as…
… recommended by Nick on python-ideas
feba4db
This page is out of date. Refresh to see the latest.
406  IPython/core/completer.py
@@ -76,6 +76,9 @@
76 76
 import re
77 77
 import shlex
78 78
 import sys
  79
+import StringIO
  80
+import tokenize as _tokenizelib
  81
+import collections
79 82
 
80 83
 from IPython.config.configurable import Configurable
81 84
 from IPython.core.error import TryNext
@@ -178,6 +181,204 @@ def compress_user(path, tilde_expand, tilde_val):
178 181
         return path
179 182
 
180 183
 
  184
+def tokenize(src):
  185
+    """Tokenize a block of python source code using the stdlib's tokenizer.
  186
+
  187
+    Parameters
  188
+    ----------
  189
+    src : str
  190
+        A string of potential python source code. The code isn't evaled, it's
  191
+        just split into its representive tokens
  192
+
  193
+    Returns
  194
+    -------
  195
+    tokens : list of strings
  196
+        A list of tokens. Tokenizer errors from invalid python source (like
  197
+        unclosed string delimiters) are supressed.
  198
+
  199
+    Examples
  200
+    --------
  201
+    In [1]: tokenize('a + b = ["cdefg" + 10]')
  202
+    ['a', '+', 'b', '=', '[', '"cdefg"', '+', '10', ']']
  203
+
  204
+    Notes
  205
+    -----
  206
+    This serves a similar function to simply splitting the source on delmiters
  207
+    (as done by CompletionSplitter) but is slightly more sophisticated. In
  208
+    particular, characters that are delmiters are never returned in the tokens
  209
+    by CompletionSplitter (or its regular expression engine), so something
  210
+    like this happens:
  211
+
  212
+    In[2]: a = CompletionSplitter()._delim_re.split('a+ "hello')
  213
+    In[3]: b = CompletionSplitter()._delim_re.split('a+= hello')
  214
+    In[4]: a == b
  215
+    True
  216
+
  217
+    This makes it very tricky to do complicated types of tab completion.
  218
+
  219
+    This tokenizer instead uses the stdlib's tokenize, which is a little
  220
+    bit more knowledgeable about python syntax. In particular, string literals
  221
+    e.g. `tokenize("'a' + '''bc'''") == ["'a'", "+", "'''bc'''"]` get parsed
  222
+    as single tokens.
  223
+    """
  224
+    rawstr = StringIO.StringIO(src)
  225
+    iter_tokens = _tokenizelib.generate_tokens(rawstr.readline)
  226
+    def run():
  227
+        try:
  228
+            for toktype, toktext, (srow,scol), (erow,ecol), line  in iter_tokens:
  229
+                if toktype != _tokenizelib.ENDMARKER:
  230
+                    yield toktext
  231
+        except _tokenizelib.TokenError:
  232
+            pass
  233
+    tokens = list(run())
  234
+    return tokens
  235
+
  236
+
  237
+def open_iden_from_tokens(tokens):
  238
+    """Find the the nearest identifier (function/method/callable name)
  239
+    that comes before the last unclosed parentheses
  240
+
  241
+    Parameters
  242
+    ----------
  243
+    tokens : list of strings
  244
+        tokens should be a list of python tokens produced by splitting
  245
+        a line of input
  246
+
  247
+    Returns
  248
+    -------
  249
+    identifiers : list
  250
+        A list of tokens from `tokens` that are identifiers for the function
  251
+        or method that comes before an unclosed partentheses
  252
+    post_tokens : list
  253
+        The subset of the tokens that occur after `identifiers` in the input
  254
+
  255
+    Raises
  256
+    ------
  257
+    ValueError if the line doesn't match
  258
+
  259
+    See Also
  260
+    --------
  261
+    tokenize : to generate `tokens`
  262
+    current_cursor_arg
  263
+    """
  264
+
  265
+    # 1. pop off the tokens until we get to the first unclosed parens
  266
+    # as we pop them off, store them in a list
  267
+    iterTokens = iter(reversed(tokens))
  268
+    tokens_after_identifier = []
  269
+
  270
+    openPar = 0 # number of open parentheses
  271
+    for token in iterTokens:
  272
+        tokens_after_identifier.insert(0, token)
  273
+        if token == ')':
  274
+            openPar -= 1
  275
+        elif token == '(':
  276
+            openPar += 1
  277
+            if openPar > 0:
  278
+                # found the last unclosed parenthesis
  279
+                break
  280
+    else:
  281
+        raise ValueError
  282
+
  283
+    # 2. Concatenate dotted names ("foo.bar" for "foo.bar(x, pa" )
  284
+    identifiers = []
  285
+    isId = re.compile(r'\w+$').match
  286
+    while True:
  287
+        try:
  288
+            identifiers.append(next(iterTokens))
  289
+            if not isId(identifiers[-1]):
  290
+                identifiers.pop(); break
  291
+            if not next(iterTokens) == '.':
  292
+                break
  293
+        except StopIteration:
  294
+            break
  295
+
  296
+    return identifiers[::-1], tokens_after_identifier
  297
+
  298
+
  299
+def current_cursor_arg(post_tokens, obj):
  300
+    """Determine which argument the cursor is current entering
  301
+
  302
+    Parameters
  303
+    ----------
  304
+    post_tokens : str
  305
+        Tokens after the last unclosed parentheses on the current
  306
+        line, starting with an open parens
  307
+    obj : callable
  308
+        The function or method being called
  309
+
  310
+    Returns
  311
+    -------
  312
+    argname : str, None
  313
+        the name of one of the arguments to the function, or None
  314
+
  315
+    Examples
  316
+    --------
  317
+    In[5]: def foo(x, y, z):
  318
+    ...        pass
  319
+    In[6]: line = "a = foo(1, bar"
  320
+    In[7]: tokens = tokenize(line)
  321
+    In[8]: identifier, post_tokens = open_iden_from_tokens(tokens)
  322
+    In[9]: identifier == ['foo']
  323
+    True
  324
+    In[10]: post_tokens == ['(', '1', ',', 'bar']
  325
+    True
  326
+    In[11]: current_cursor_arg(post_tokens, foo)
  327
+    'y'
  328
+
  329
+    See ALso
  330
+    --------
  331
+    tokenize
  332
+    """
  333
+    try:
  334
+        argspec = inspect.getargspec(obj)
  335
+    except ValueError:
  336
+        # python3
  337
+        argspec = inspect.getfullargspec(obj)
  338
+
  339
+
  340
+    n_commas = 0
  341
+    n_open_parens = 0
  342
+    n_open_braces = 0
  343
+    n_open_brackets = 0
  344
+    iterTokens = iter(reversed(post_tokens))
  345
+    for token in iterTokens:
  346
+        if (n_open_parens >= 0) or (n_open_braces >= 0) or \
  347
+                (n_open_brackets >= 0):
  348
+
  349
+            if token == '=':
  350
+                # short circuit this stuff, they're using a named argument, so we'll
  351
+                # just take that name
  352
+                try:
  353
+                    return iterTokens.next()
  354
+                except:
  355
+                    return None
  356
+
  357
+            if token == ',':
  358
+                n_commas += 1
  359
+
  360
+        if token == '(':
  361
+            n_open_parens += 1
  362
+            if n_open_parens > 0:
  363
+                break
  364
+        elif token == '{':
  365
+            n_open_braces += 1
  366
+        elif token == '[':
  367
+            n_open_brackets += 1
  368
+        elif token == ')':
  369
+            n_open_parens -= 1
  370
+        elif token == '}':
  371
+            n_open_braces -= 1
  372
+        elif token == ']':
  373
+            n_open_brackets -= 1
  374
+
  375
+    if inspect.ismethod(obj):
  376
+        # need to ignore self
  377
+        return argspec.args[n_commas+1]
  378
+
  379
+    return argspec.args[n_commas]
  380
+
  381
+
181 382
 class Bunch(object): pass
182 383
 
183 384
 
@@ -232,6 +433,7 @@ def delims(self, delims):
232 433
     def split_line(self, line, cursor_pos=None):
233 434
         """Split a line of text with a cursor at the given position.
234 435
         """
  436
+
235 437
         l = line if cursor_pos is None else line[:cursor_pos]
236 438
         return self._delim_re.split(l)[-1]
237 439
 
@@ -245,7 +447,7 @@ class Completer(Configurable):
245 447
         but can be unsafe because the code is actually evaluated on TAB.
246 448
         """
247 449
     )
248  
-    
  450
+
249 451
 
250 452
     def __init__(self, namespace=None, global_namespace=None, config=None, **kwargs):
251 453
         """Create a new completer for the command line.
@@ -340,7 +542,7 @@ def attr_matches(self, text):
340 542
         #io.rprint('Completer->attr_matches, txt=%r' % text) # dbg
341 543
         # Another option, seems to work great. Catches things like ''.<tab>
342 544
         m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
343  
-    
  545
+
344 546
         if m:
345 547
             expr, attr = m.group(1, 3)
346 548
         elif self.greedy:
@@ -350,7 +552,7 @@ def attr_matches(self, text):
350 552
             expr, attr = m2.group(1,2)
351 553
         else:
352 554
             return []
353  
-    
  555
+
354 556
         try:
355 557
             obj = eval(expr, self.namespace)
356 558
         except:
@@ -361,7 +563,7 @@ def attr_matches(self, text):
361 563
 
362 564
         if self.limit_to__all__ and hasattr(obj, '__all__'):
363 565
             words = get__all__entries(obj)
364  
-        else: 
  566
+        else:
365 567
             words = dir2(obj)
366 568
 
367 569
         try:
@@ -384,7 +586,7 @@ def get__all__entries(obj):
384 586
         words = getattr(obj, '__all__')
385 587
     except:
386 588
         return []
387  
-    
  589
+
388 590
     return [w for w in words if isinstance(w, basestring)]
389 591
 
390 592
 
@@ -400,34 +602,34 @@ def _greedy_changed(self, name, old, new):
400 602
 
401 603
         if self.readline:
402 604
             self.readline.set_completer_delims(self.splitter.delims)
403  
-    
  605
+
404 606
     merge_completions = CBool(True, config=True,
405 607
         help="""Whether to merge completion results into a single list
406  
-        
  608
+
407 609
         If False, only the completion results from the first non-empty
408 610
         completer will be returned.
409 611
         """
410 612
     )
411 613
     omit__names = Enum((0,1,2), default_value=2, config=True,
412 614
         help="""Instruct the completer to omit private method names
413  
-        
  615
+
414 616
         Specifically, when completing on ``object.<tab>``.
415  
-        
  617
+
416 618
         When 2 [default]: all names that start with '_' will be excluded.
417  
-        
  619
+
418 620
         When 1: all 'magic' names (``__foo__``) will be excluded.
419  
-        
  621
+
420 622
         When 0: nothing will be excluded.
421 623
         """
422 624
     )
423 625
     limit_to__all__ = CBool(default_value=False, config=True,
424 626
         help="""Instruct the completer to use __all__ for the completion
425  
-        
  627
+
426 628
         Specifically, when completing on ``object.<tab>``.
427  
-        
  629
+
428 630
         When True: only those names in obj.__all__ will be included.
429  
-        
430  
-        When False [default]: the __all__ attribute is ignored 
  631
+
  632
+        When False [default]: the __all__ attribute is ignored
431 633
         """
432 634
     )
433 635
 
@@ -497,12 +699,13 @@ def __init__(self, shell=None, namespace=None, global_namespace=None,
497 699
             self.clean_glob = self._clean_glob
498 700
 
499 701
         # All active matcher routines for completion
500  
-        self.matchers = [self.python_matches,
  702
+        self.matchers = [self.annotations_argmatches,
  703
+                         self.python_matches,
501 704
                          self.file_matches,
502 705
                          self.magic_matches,
503 706
                          self.alias_matches,
504 707
                          self.python_func_kw_matches,
505  
-                         ]
  708
+                        ]
506 709
 
507 710
     def all_completions(self, text):
508 711
         """
@@ -611,7 +814,7 @@ def magic_matches(self, text):
611 814
         cell_magics = lsm['cell']
612 815
         pre = self.magic_escape
613 816
         pre2 = pre+pre
614  
-        
  817
+
615 818
         # Completion logic:
616 819
         # - user gives %%: only do cell magics
617 820
         # - user gives %: do both line and cell magics
@@ -641,7 +844,7 @@ def alias_matches(self, text):
641 844
 
642 845
     def python_matches(self,text):
643 846
         """Match attributes or global python names"""
644  
-        
  847
+
645 848
         #io.rprint('Completer->python_matches, txt=%r' % text) # dbg
646 849
         if "." in text:
647 850
             try:
@@ -684,65 +887,133 @@ def _default_arguments(self, obj):
684 887
         except TypeError: pass
685 888
         return []
686 889
 
687  
-    def python_func_kw_matches(self,text):
  890
+
  891
+    def python_func_kw_matches(self, text):
688 892
         """Match named parameters (kwargs) of the last open function"""
689 893
 
690 894
         if "." in text: # a parameter cannot be dotted
691 895
             return []
692  
-        try: regexp = self.__funcParamsRegex
693  
-        except AttributeError:
694  
-            regexp = self.__funcParamsRegex = re.compile(r'''
695  
-                '.*?(?<!\\)' |    # single quoted strings or
696  
-                ".*?(?<!\\)" |    # double quoted strings or
697  
-                \w+          |    # identifier
698  
-                \S                # other characters
699  
-                ''', re.VERBOSE | re.DOTALL)
700  
-        # 1. find the nearest identifier that comes before an unclosed
701  
-        # parenthesis before the cursor
702  
-        # e.g. for "foo (1+bar(x), pa<cursor>,a=1)", the candidate is "foo"
703  
-        tokens = regexp.findall(self.text_until_cursor)
704  
-        tokens.reverse()
705  
-        iterTokens = iter(tokens); openPar = 0
706  
-        for token in iterTokens:
707  
-            if token == ')':
708  
-                openPar -= 1
709  
-            elif token == '(':
710  
-                openPar += 1
711  
-                if openPar > 0:
712  
-                    # found the last unclosed parenthesis
713  
-                    break
714  
-        else:
  896
+
  897
+        try:
  898
+            tokens = tokenize(self.text_until_cursor)
  899
+            ids = open_iden_from_tokens(tokens)[0]
  900
+        except ValueError:
715 901
             return []
716  
-        # 2. Concatenate dotted names ("foo.bar" for "foo.bar(x, pa" )
717  
-        ids = []
718  
-        isId = re.compile(r'\w+$').match
719  
-        while True:
720  
-            try:
721  
-                ids.append(next(iterTokens))
722  
-                if not isId(ids[-1]):
723  
-                    ids.pop(); break
724  
-                if not next(iterTokens) == '.':
725  
-                    break
726  
-            except StopIteration:
727  
-                break
728  
-        # lookup the candidate callable matches either using global_matches
729  
-        # or attr_matches for dotted names
  902
+
730 903
         if len(ids) == 1:
731 904
             callableMatches = self.global_matches(ids[0])
732 905
         else:
733  
-            callableMatches = self.attr_matches('.'.join(ids[::-1]))
  906
+            callableMatches = self.attr_matches('.'.join(ids))
  907
+
734 908
         argMatches = []
735 909
         for callableMatch in callableMatches:
736 910
             try:
737 911
                 namedArgs = self._default_arguments(eval(callableMatch,
738  
-                                                         self.namespace))
  912
+                    self.namespace))
739 913
             except:
  914
+                print 'err'
740 915
                 continue
  916
+
741 917
             for namedArg in namedArgs:
742 918
                 if namedArg.startswith(text):
743 919
                     argMatches.append("%s=" %namedArg)
744 920
         return argMatches
745 921
 
  922
+    def annotations_argmatches(self, text):
  923
+        """Function specific matches based on the arguments of that function.
  924
+        For example, np.load(<tab> might only show files that match some glob
  925
+        pattern.
  926
+
  927
+        Note that if this sucessfully matches, it will be the only set of
  928
+        matches shown to the user. This behavior overrides the
  929
+        merge_completions config variable. This behavior is set by the attribute
  930
+        `exclusive_completions = True` which is set after the function. See
  931
+        extensions/annotations.py for the user-facing code to hook into this
  932
+        system
  933
+
  934
+        Parameters
  935
+        ----------
  936
+        text : str
  937
+            This argument is not used by this function, because we need to do
  938
+            most of the tokenizing and lexing custom.
  939
+
  940
+        Returns
  941
+        -------
  942
+        matches : list
  943
+            A list of strings to be displayed to the user via readline as
  944
+            possible tab completions. These completions are function/argument
  945
+            specific, and their appearance is based on decorators applied to the
  946
+            functions that annotate specific function arguments with possible
  947
+            tab completions.
  948
+        """
  949
+
  950
+        try:
  951
+            tokens = tokenize(self.text_until_cursor)
  952
+            ids, post_tokens = open_iden_from_tokens(tokens)
  953
+        except ValueError:
  954
+            return []
  955
+
  956
+        def match_object(obj_name):
  957
+            "Find an object by name using eval -- need exact match"
  958
+            try:
  959
+                return eval(obj_name, self.namespace)
  960
+            except:
  961
+                try:
  962
+                    return eval(obj_name, self.global_namespace)
  963
+                except:
  964
+                    return None
  965
+
  966
+        if len(ids) >= 1:
  967
+            try:
  968
+                obj = match_object('.'.join(ids))
  969
+                annotations = obj.__tab_completions__
  970
+                argname = current_cursor_arg(post_tokens, obj)
  971
+            except (AttributeError, TypeError, IndexError) as e:
  972
+                # the attribute error comes from obj not having an
  973
+                # __annotations__, the typeerror comes from it
  974
+                # potentially not being inspect.argspec-able
  975
+                # the indexerror comes if the user is trying to enter
  976
+                # the nth argument for a function that takes less than
  977
+                # n arguments
  978
+                return []
  979
+        else:
  980
+            # something wierd happened. this can happen for instance if the
  981
+            # line is f((<TAB>
  982
+            return []
  983
+
  984
+        try:
  985
+            attr = annotations[argname]
  986
+        except KeyError:
  987
+            # this indicates that the __annotations__ is malformed
  988
+            return []
  989
+
  990
+        # make the event namedtuple for the callback
  991
+        if not hasattr(self, '__func_argcomplete_Event'):
  992
+            # create the namedtuple type, and stash it in self
  993
+            self.__func_argcomplete_Event = collections.namedtuple('Event',
  994
+                ['text', 'tokens', 'line', 'ipcompleter'])
  995
+        event = self.__func_argcomplete_Event(text=text,
  996
+            tokens=tokens, line=self.text_until_cursor,
  997
+            ipcompleter=self)
  998
+
  999
+
  1000
+        matches = []
  1001
+        if hasattr(attr, 'tab_matches'):
  1002
+            m = attr.tab_matches(event)
  1003
+            matches.extend(m)
  1004
+        elif hasattr(attr, '__iter__'):
  1005
+            for item in attr:
  1006
+                if hasattr(item, 'tab_matches'):
  1007
+                    matches.extend(item.tab_matches(event))
  1008
+        # else we just pass and return an empty matches
  1009
+
  1010
+        return matches
  1011
+
  1012
+    # this is an extension to the API, where this method indicates
  1013
+    # that if it returns matches, they should be displayed to the user as
  1014
+    # the ONLY tab-completions
  1015
+    annotations_argmatches.exclusive_completions = True
  1016
+
746 1017
     def dispatch_custom_completer(self, text):
747 1018
         #io.rprint("Custom! '%s' %s" % (text, self.custom_completers)) # dbg
748 1019
         line = self.line_buffer
@@ -852,14 +1123,27 @@ def complete(self, text=None, line_buffer=None, cursor_pos=None):
852 1123
                 self.matches = []
853 1124
                 for matcher in self.matchers:
854 1125
                     try:
855  
-                        self.matches.extend(matcher(text))
  1126
+                        # this is an extension to the API, where this method
  1127
+                        # indicates that if it returns matches, they should be
  1128
+                        # displayed to the user as the ONLY tab-completions
  1129
+                        if hasattr(matcher, 'exclusive_completions') \
  1130
+                                and matcher.exclusive_completions is True:
  1131
+                            m = matcher(text)
  1132
+                            if m:
  1133
+                                self.matches = m
  1134
+                                break
  1135
+                        else:
  1136
+                            self.matches.extend(matcher(text))
856 1137
                     except:
857 1138
                         # Show the ugly traceback if the matcher causes an
858 1139
                         # exception, but do NOT crash the kernel!
859 1140
                         sys.excepthook(*sys.exc_info())
860 1141
             else:
861 1142
                 for matcher in self.matchers:
862  
-                    self.matches = matcher(text)
  1143
+                    try:
  1144
+                        self.matches = matcher(text)
  1145
+                    except:
  1146
+                        sys.excepthook(*sys.exc_info())
863 1147
                     if self.matches:
864 1148
                         break
865 1149
         # FIXME: we should extend our api to return a dict with completions for
340  IPython/extensions/annotations.py
... ...
@@ -0,0 +1,340 @@
  1
+# -*- coding: utf-8 -*-
  2
+"""
  3
+Interact with the IPython tab completion system via function annotations.
  4
+
  5
+For example (python3):
  6
+
  7
+In[1]: def load(filename : tab_glob('*.txt'), mode):
  8
+...        pass
  9
+
  10
+Will trigger the IPython tab completion system to recomment files ending in .txt
  11
+to you, when you're calling the load function interactively.
  12
+"""
  13
+
  14
+import inspect
  15
+import functools
  16
+import warnings
  17
+import types
  18
+import glob
  19
+import abc
  20
+try:
  21
+    # python3
  22
+    import builtins
  23
+    typetype = builtins.type
  24
+except ImportError:
  25
+    #python2
  26
+    typetype = types.TypeType
  27
+
  28
+from IPython.core.completer import has_open_quotes
  29
+
  30
+
  31
+__all__ = ['annotate', 'tab_glob', 'tab_instance', 'tab_literal']
  32
+
  33
+def tab_completion(*args, **kwargs):
  34
+    if (len(args) == 1 and len(kwargs.keys()) == 0 and
  35
+        hasattr(args[0], '__call__')):
  36
+        # this decorator was called with no arguments and is
  37
+        # wrapping a function where the annotations are defined
  38
+        # with py3k syntax
  39
+        f = args[0] # rename
  40
+        if not hasattr(f, '__annotations__'):
  41
+            raise ValueError('f is not annotated')
  42
+        f.__tab_completions__ = f.__annotations__
  43
+        return f
  44
+
  45
+    # otherwise, this decorator is being called with arguments, indicating
  46
+    # python2 syntax
  47
+    def wrapper(f):
  48
+        argspec = inspect.getargspec(f)
  49
+        for key in kwargs.keys():
  50
+            if (key not in argspec.args) and (key != argspec.varargs) \
  51
+                    and (key != argspec.kwargs):
  52
+                raise ValueError('%s is not an argument taken by %s' \
  53
+                    % (key, wrapped.__name__))
  54
+
  55
+            if hasattr(f, '__tab_completions__'):
  56
+                # check that the annotations being provided don't already exist
  57
+                # if they do, we warn and overwrite. this decorator does not
  58
+                # implement any scheme for composing annotations.
  59
+                if key in f.__annotations__:
  60
+                    warnings.warn('Overwriting tab completion on %s' % key,
  61
+                        RuntimeWarning)
  62
+
  63
+        try:
  64
+            f.__tab_completions__.update(kwargs)
  65
+        except AttributeError:
  66
+            f.__tab_completions__ = kwargs
  67
+
  68
+        return f
  69
+
  70
+    return wrapper
  71
+
  72
+
  73
+def tab_completion_return(return_value_annotation):
  74
+    """Register a tab completion annotation for the return value in python2.x
  75
+    
  76
+    Unfortunately, you won't be able to use
  77
+    @tab_completion(return=tab_instance(x)) as a decorator, since it's a
  78
+    syntax error. So use this instead
  79
+
  80
+    """
  81
+    def wrapper(f):
  82
+        argspec = inspect.getargspec(f)
  83
+        if hasattr(f, '__tab_completions__'):
  84
+            # check that the annotations being provided don't already exist
  85
+            # if they do, we warn and overwrite. this decorator does not
  86
+            # implement any scheme for composing annotations.
  87
+            if 'return' in f.__annotations__:
  88
+                warnings.warn('Overwriting tab completion on return',
  89
+                              RuntimeWarning)
  90
+
  91
+        try:
  92
+            f.__tab_completions__['return'] = return_value_annotation
  93
+        except AttributeError:
  94
+            f.__tab_completions__ = {'return' : return_value_annotation}
  95
+
  96
+        return f
  97
+
  98
+    return wrapper
  99
+
  100
+
  101
+def annotate(**kwargs):
  102
+    """Decorator to annotate function arguments in Python 2.x.
  103
+    Note, in Python3, this decorator is not required as the ability
  104
+    is built into the language. See PEP 3107 for details.
  105
+
  106
+    Examples
  107
+    --------
  108
+    #>>> @annotate(foo='bar', qux=str)
  109
+    #>>> def f(foo, qux='hello'):
  110
+    #...    pass
  111
+
  112
+    Is evalent to the following python3 code:
  113
+    In[2]: def f(foo : 'bar', qux : str = 'hello'):
  114
+    ...        pass
  115
+    """
  116
+
  117
+    def wrapper(f):
  118
+        argspec = inspect.getargspec(f)
  119
+        for key in kwargs.keys():
  120
+            if (key not in argspec.args) and (key != argspec.varargs) \
  121
+                    and (key != argspec.kwargs):
  122
+                raise ValueError('%s is not an argument taken by %s' \
  123
+                    % (key, wrapped.__name__))
  124
+
  125
+            if hasattr(f, '__annotations__'):
  126
+                # check that the annotations being provided don't already exist
  127
+                # if they do, we warn and overwrite. this decorator does not
  128
+                # implement any scheme for composing annotations.
  129
+                if key in f.__annotations__:
  130
+                    warnings.warn('Overwriting annotation on %s' % key,
  131
+                        RuntimeWarning)
  132
+
  133
+        f.__annotations__ = kwargs
  134
+
  135
+        return f
  136
+
  137
+    return wrapper
  138
+
  139
+
  140
+class AnnotationCompleterBase(object):
  141
+    """Abstract base class for IPython annotation based tab completion
  142
+    annotators
  143
+    """
  144
+    __metaclass__ = abc.ABCMeta
  145
+
  146
+    @abc.abstractmethod
  147
+    def tab_matches(self, event):
  148
+        """Callback for the function-annotation tab completion system.
  149
+
  150
+        If the user attempts to do a tab completion on an argument
  151
+        (to a function/method) that is annotated for tab completion,
  152
+        this callback will be executed.
  153
+
  154
+        Parameters
  155
+        ----------
  156
+        event : nametuple
  157
+            event is a namedtuple containing four keys: 'text', 'tokens',
  158
+            'line', and 'ipcompleter'
  159
+
  160
+        `Event` Attributes
  161
+        ------------------
  162
+        event.line : str
  163
+            event.line contains the full line entered by the IPython user.
  164
+        event.text : str
  165
+            event.text is the last portion of the line, formed by splitting
  166
+            the line on a set of python-language delimiters, and returning you
  167
+            the last portion.
  168
+        event.tokens : list of strings
  169
+            event.tokens is the result of running the python standard library
  170
+            tokenizer on event.line. This is similar to splitting on
  171
+            delimiters as above, but slightly different for string literals in
  172
+            particular, where, for instance, '''a''' would be a single token.
  173
+        event.ipcompleter : IPython.core.completer.IPCompleter
  174
+            This is a reference back to the caller's class. You can use this
  175
+            to get access to more functions to aid in parsing the line,
  176
+            namespaces, to get configuration options from the IPython
  177
+            configuration system, etc.
  178
+        """
  179
+        pass
  180
+
  181
+
  182
+class tab_literal(AnnotationCompleterBase):
  183
+    """Annotation for function arguments that recommends completions from a
  184
+    set of enumerated literals. This is useful if you have an argumnet for a
  185
+    function that is designed to be called only with a small handful of possible
  186
+    values"""
  187
+
  188
+    def __init__(self, *completions):
  189
+        """Set up a tab completion callback.
  190
+
  191
+        Example (python 3)
  192
+        ------------------
  193
+        In[3]: def f(x : tab_literal(100, 200, 300)):
  194
+        ...        pass
  195
+        In[4]: f(1<HIT_THE_TAB_KEY>
  196
+        will fill in the 100
  197
+        """
  198
+        self.completions = completions
  199
+
  200
+    def tab_matches(self, event):
  201
+        """Callback for the IPython annoation tab-completion system
  202
+        """
  203
+        # the complicated stuff here is handling string literals, because we want
  204
+        # to make readline see the quotation marks, which it usually thinks are
  205
+        # just delimiters and doesn't deal with.
  206
+        matches = []
  207
+        for cb in self.completions:
  208
+            if isinstance(cb, basestring):
  209
+                if event.tokens[-1] in [' ', '=', '(']:
  210
+                    matches.append("'%s'" % cb)
  211
+                elif event.tokens[-1] == ',':
  212
+                    matches.append(" '%s'" % cb)
  213
+                elif has_open_quotes(event.line) and cb.startswith(event.text):
  214
+                    matches.append(cb)
  215
+            else:
  216
+                str_cb = str(cb)
  217
+                if str_cb.startswith(event.text):
  218
+                    matches.append(str_cb)
  219
+
  220
+        return matches
  221
+
  222
+
  223
+class tab_glob(AnnotationCompleterBase):
  224
+    """Annotation for function arguments that recommends completions which are
  225
+    filenames matching a glob pattern.
  226
+    """
  227
+
  228
+    def __init__(self, glob_pattern):
  229
+        """Set up a tab completion callback with glob matching
  230
+
  231
+        Example (python 3)
  232
+        ------------------
  233
+        In[5]: def f(x : tab_glob("*.txt")):
  234
+        ...        pass
  235
+
  236
+        In[6]: f(<HIT_THE_TAB_KEY>
  237
+        will show you files ending in .txt
  238
+        """
  239
+        self.glob_pattern = glob_pattern
  240
+
  241
+    def tab_matches(self, event):
  242
+        """Callback for the IPython annoation tab-completion system
  243
+        """
  244
+
  245
+        matches = []
  246
+
  247
+        if event.tokens[-1] in [' ', '=', '(']:
  248
+            fmt = "'%s'"
  249
+        elif event.tokens[-1] == ',':
  250
+            fmt = " '%s'"
  251
+        else:
  252
+            fmt = '%s'
  253
+
  254
+        file_matches = [fmt % m for m in glob.glob(event.text + self.glob_pattern)]
  255
+        dir_matches = [fmt % m for m in glob.glob(event.text + '*/')]
  256
+
  257
+        return file_matches + dir_matches
  258
+
  259
+
  260
+class tab_instance(AnnotationCompleterBase):
  261
+    """Annotation for function arguments that recommends python variables in
  262
+    your namespace that an instance of supplied types"""
  263
+
  264
+    def __init__(self, *klasses):
  265
+        """Set up a tab completion callback with isinstance matching
  266
+
  267
+        Parameters
  268
+        ----------
  269
+        klasses : the classes you'd like to match on
  270
+
  271
+        Example (python 3)
  272
+        ------------------
  273
+        In[7]: x, y = 1, 2
  274
+        In[8]: def f(x : tab_instance(int)):
  275
+        ...        pass
  276
+
  277
+        In[9]: f(<TAB>
  278
+        will show you files ending in .txt
  279
+
  280
+        Limitations
  281
+        -----------
  282
+        Because of python's dynamic typing, this can't check the type of the
  283
+        return value of functions, so this won't be able to recommend something
  284
+        like max(1,2) in the previous example.
  285
+        """
  286
+        self.klasses = set(klasses)
  287
+
  288
+        # add some extras
  289
+        self.klasses.update([types.FunctionType, types.BuiltinFunctionType,
  290
+            typetype, types.ModuleType])
  291
+
  292
+    def tab_matches(self, event):
  293
+        """Callback for the IPython annoation tab-completion system
  294
+        """
  295
+
  296
+        matches = []
  297
+        for key in event.ipcompleter.python_matches(event.text):
  298
+            try:
  299
+                if any(isinstance(eval(key, event.ipcompleter.namespace), klass)
  300
+                        for klass in self.klasses):
  301
+                    matches.append(key)
  302
+            except:
  303
+                continue
  304
+        return matches
  305
+
  306
+
  307
+if __name__ == '__main__':
  308
+    # some testing code.
  309
+    text_file = tab_glob('*.txt')
  310
+    isstring = tab_instance(str)
  311
+
  312
+    @tab_completion
  313
+    def function1(arg1 : text_file, arg2):
  314
+        pass
  315
+
  316
+    @tab_completion
  317
+    def function2(arg1, arg2 : text_file):
  318
+        pass
  319
+
  320
+    # this should be the same as function1
  321
+    @tab_completion(arg1=text_file)
  322
+    def function1_1(arg1, arg2):
  323
+        pass
  324
+    
  325
+    assert function1.__tab_completions__ == function1_1.__tab_completions__
  326
+
  327
+    @tab_completion
  328
+    def function4(arg1, arg2) -> isstring:
  329
+        pass
  330
+
  331
+    @tab_completion_return(isstring)
  332
+    def function4_1(arg1, arg2):
  333
+        pass
  334
+
  335
+    @tab_completion
  336
+    def function5(arg1 : isstring):
  337
+        pass
  338
+
  339
+    assert function4.__tab_completions__ == function4_1.__tab_completions__
  340
+ 
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.