Function annotation based hooks into the tab completion system #2636

Closed
wants to merge 22 commits into
from

Projects

None yet

3 participants

@rmcgibbo

NOT READY FOR MERGING

This PR adds a way for user code to hook into the ipython tab completion system via function annotations PEP 3107. python 2.x support is provided via a decorator.

This feature was discussed on IPython-dev along with the use of function annotations for the feature on Python-ideas as it relates to interoperability between annotation schemes.

For example, using the glob tab completion, only files matching the appropriate ending (and directories) are shown:

>>> from IPython.extensions.annotations import tab_glob
>>> def f(x : tab_glob('*.txt')):
...    pass
>>> load(<TAB_CHARACTER>
'COPYING.txt'        'build/'             'ipython.egg-info/'  'setupext/'          
'IPython/'           'docs/'              'scripts/'           'tools/'   

Currently, the annotations provided are

  • tab_glob, which recommends string literals that are directories or files matching a pattern
  • tab_literal, which recommends from a set of enumerated literals
  • tab_instance, which recommends objects that exist in the current namespace that are instances of a supplied (set of) class(es).

The API for adding new annotations is pretty simple (the implementation of tab_instance is ~15 LOC), so users can easily both add these annotations to their functions or add new annotations that implement other types of matches.

I'm sure their are some bugs. One that I know about is poor handling by tab_glob of filenames with spaces, since the spaces cause the tokenizer to token-break where you don't want it to.

This feature is complicated enough to require a significant number of tests (which I haven't written yet -- I've just been testing it interactively). Testing it is a challenge in and of itself, since you need to somehow simulate the TAB key in the testing environment and parse the output of readline. If anyone has any ideas for how to test the final product automatically, that would be awesome. Obviously the methods can be tested individually, which is straightforward.

[Also, the completion machinery currently in ipython could use some serious refactoring, but I didn't try to tackle that here]


I'm not really sure what the next step forward on this is. It's a big enough change, IMO, that it deserves some serious discussion, user-facing documentation (IMHO I wrote good docstrings, but that's not enough), etc.

Also, I don't really like the names tab_glob, tab_literal, and tab_instance.

Thoughts?

@rmcgibbo

grrr. doctest doesn't like my examples that try to show what happens when you enter the tab key. I want to just write "TAB" to show what happens, but that doesn't work w.r.t doctests.

@asmeurer

Shouldn't they have IPython In[1]: instead of >>>?

@rmcgibbo

This is from the reply I sent to the Python-ideas thread that @takluyver started on function annotations, but it should probably go here too.

Here's the scheme that this code is using to check the annotations for tab completion information:

1) Check if the function in question has an annotation.

  • If not, bail out and skip this feature. No harm, no foul.
  • If so, go to (2).

2) Check if the annotation is one of our objects / implements our interface.

  • If so, fantastic! Start invoking this feature.
  • If not, go to (3).

3) Check if the annotation is iterable, and if it is iterable check if any of the items are one of our objects or implement our interface.

  • If so, fantastic! Start invoking this feature.
  • If not, bail out and skip this feature.

(An obvious optimization is skipping step (3) if the annotation is a string.)

One thing that I'm not sure about is whether the check for whether an annotations is ours is by looking for it to subclass an (abstract) baseclass extensions/annotations.py:AnnotationCompleterBase, by using hasattr to check for the method tab_matches implements the behavior one of these tab matching annotations needs to provide, or by simply trying to call obj.tab_matches inside of a try/catch block. I've read some discussion that hasattr is broken by design, but I don't know if that's relevant to this issue.

@asmeurer

I wouldn't call that broken by design. It was just originally written with the equivalent of except: instead of except AttributeError:. This was changed in Python 3. I don't think it affects us here. The worst that can happen is that it will mask an error in someone else's code, but actually, when tab completing, we might even want to mask all errors (or maybe not. It should be discussed).

@rmcgibbo

If you keep reading the thread, the discussion morphs into the fact that hasattr isn't really the static introspection that you might think it is (I did). (But I don't really care about hasattr, and I'd rather not get bogged down.)

Maybe the bigger issue is Nick Coglan's comments on python-ideas, saying that it would be better to use a decorator in addition to the annotation.

The syntax that he's recommending is something like:

>>> @tab_expansion
>>> def foo(filename : glob_expansion('*.txt')):
...    pass

Where the @tab_expansion decorator moves the glob_expansion information from __annotations__ into an ipython specific metadata location.

As Nick said:

Mixing annotations intended for different consumers is a fundamentally bad idea, as it encourages unreadable code and complex dances to avoid stepping on each other's toes. It's better to design a separate API that supports composition by passing the per-parameter details directly to a decorator factory (which then adds appropriate named attributes to the function), with annotations used just as syntactic sugar for simple cases where no composition is involved.

I am inclined to follow this advice, because he (1) makes a very good point and (2) is a python core developer.

@asmeurer

If you keep reading the thread, the discussion morphs into the fact that hasattr isn't really the static introspection that you might think it is (I did).

No, almost nothing in Python is static. But that's the way the language works.

Where the @tab_expansion decorator moves the glob_expansion information from annotations into an ipython specific metadata location.

We'll probably want to move it elsewhere at some point anyway, for caching purposes, because tab completion is one of those things that you want to be absolutely instantaneous, or else it significantly degrades the user experience.

I am inclined to follow this advice, because he (1) makes a very good point and (2) is a python core developer.

Yes, this is a good point.

Here are some other thoughts I've had:

  • We might want a way to "decorate a whole module", so to speak. That is to say, if you know that every function in a module acts the same way, you should be able to say that in one fell swoop.

  • Does your code work for classes? It will need to look at __new__ or __init__.

  • We also need to look at return types. There is annotations syntax for this as well. I think this case is easier, because we really just need to annotate the type of the object being returned. It is only needed for things like f(x).<TAB>. I guess eventually it can be made to be smarter for functions whose return type depends on the input.

  • Combining the last two points, return types for classes that define __new__.

My personal use case is SymPy. A lot of SymPy's API uses method chaining, which is annoying since it can't be tab completed unless you do it in steps (i.e., set the object to a variable, and then call the method on that variable). For my first point, almost all public facing SymPy functions and classes take in Expr objects as input, and give them as output. So it would be nice to just tell the completer to compete Exprs methods on basically everything, with as little work as possible on the part of me (the SymPy developer).

Aside from method chaining, the case where I personally find myself wishing for better tab completion is not in the types of the arguments, but in the names of the arguments themselves, i.e., keyword arguments. When the keyword arguments are explicitly named in the function definition, the tab completer should be able to get that information straightaway, using whatever functions from the inspect module. When they are given as **kwargs, they will have to be annotated.

@rmcgibbo

Thanks so much for looking over this closely. Your suggestions are top notch and really appreciated!

In no particular order:

  • It would be pretty straightforward to do a class decorator that would translate the annotations for every method. (Or a metaclass, but I prefer class decorators when the extra features of metaclasses are not required). I will add this.
  • It should be possible to run the "decorator" on every function in a module, either by putting something at the bottom of the file that references dir(sys.modules[__name__]) I've always felt like putting influential code at the bottom of a file is kind of hiding it -- it would be nicer at the top. Perhaps a sympy __init__.py could invoke the decorators before the functions are actually imported?

Thanks for mentioning SymPy and your use case, because I was thinking about the feature rather narrowly, since it was (in my head) just for my own uses.

  • f(x).<TAB> is a good idea. It would really be another feature, since the current PR only deals with lines that have an unclosed parentheses, but that's fine. Given that I've spent a few-tens-of-hours with the completer codebase, I know just how to do it (assuming that the syntax for passing in the type annotations is settled).
  • Building on the previous point, how should we handle foo(<TAB> when foo's first argument is known to take a certain type and a function bar is known to return instances of that same type. Currently only objects that are instances of that type would be recommended -- should bar and other functions known to return the type also be recommended?
  • The current code works for methods on classes, but not for class constructors/initializers. I forgot about that and will add it.

When the keyword arguments are explicitly named in the function definition, the tab completer should be able to get that information straightaway

  • I think this is already done (e.g. In[9), but I agree that it could be better. This is the current behavior.
In [7]: def foo(longname1=1, longname2=2):
   ...:     pass
   ...:    

In [8]: def bar(longname1, longname2):
   ...:     pass
   ...: 

In [9]: foo(longn<TAB>
longname1=  longname2= 

In [10]: bar(longn<TAB>
[system bell rings and no completions offered]

Perhaps we could do better by adding keyword arg complete such that In[10] would complete with longname1=?

@takluyver
IPython member
@asmeurer
@rmcgibbo

Thanks so much for looking over this closely. Your suggestions are top notch and really appreciated!

Actually, I didn't look over the code closely. I'm leaving that to the
ipython devs, who know what to look for. I am more concerned about the
public API, as that is how I would be using it.

But frankly, the implementation is easy. Getting public API "right" is the hard part.


This just goes to show that we do need to register the completion metadata globally.

I'm not really sure what you mean by this. It seems logical to me that any return value annotation / tab completion info will be stored in the same location the argument annotation is -- that is, attached to the function. Perhaps in func.__annotations__, or perhaps in func.__tab_completions__ or perhaps in some other attribute.

Conceptually, I don't see much difference between the function arguments and the return value.


keyword argument tab completion has been the behavior since at least 0.12.1

$ ipython
Python 2.7.3 (default, Jun 29 2012, 17:56:12) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: def bar(longname1=1, longname2=2):
   ...:     pass
   ...: 

In [2]: bar(longn
longname1=  longname2=  
@rmcgibbo

@takluyver yeah, you're right. I wasn't trying to say that it should be part of the IPython API. I just meant that if @asmeurer wanted to decorate all of his functions at once, he'd have options.

The decorator will be conceptually like:

def tab_complete(func):
     # move the tab completion annotations into an ipython specific location
     func.__tab_completions = func.__annotations__

So that the normal usage is

from IPython.core.annotations import tab_complete, tab_instance
# I don't really like the name "tab_instance". I should think of something better
@tab_complete
def sympy_addition(x : tab_instance(Expr), y : tab_instance(Expr)):
    pass

But if @asmeurer wants to, he can avoid manually decorating all of his functions and do a "spooky" action at a distance decorating at the end of his module like

for fname in dir(sys.modules[__name__]):
    # you wound need some more checking in practice...
    # but you get the idea
    tab_complete(eval(f))
@asmeurer

keyword argument tab completion has been the behavior since at least 0.12.1

Did not know that. I guess I should try pressing tab more often, and see what works.

@rmcgibbo

I've got a lot of good ideas from this thread and from the python-ideas emails. But I don't think this code is really ready yet. I'm going to work on it more, and then I'll resubmit the PR when it's ready.

@rmcgibbo rmcgibbo closed this Dec 4, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment