Add columns to IPython tab completions of DataFrame attributes

[takluyver]

This addresses #230.

I'm not entirely happy about how we check whether IPython is imported, but it's the best option I could see, and I don't think we should unconditionally import IPython and dependent modules just because it's installed.

[wesm]

This is very cool-- I like it. Merged into master. I don't have a strong feeling about prefacing commits with ENH/BUG/DOC/RLS, etc. But if you feel so inclined feel free to do so

[takluyver]

I tend not to use the abbreviations, but I've seen that you do - I'm happy to conform if you'd like to keep the format consistent for the project.

[wesm]

Eh, I've been doing it for a few months and I'm kind of "meh" on it, so carry on =)

[wesm]

have you noticed any problems with this? I was getting some cases where it caused IPython to blow up "Sorry IPython has crashed..." but I was under too much time pressure with what I was doing to write down how to reproduce it. And now I can't-- ha. Well I don't think I'll let it hold up the release, can always tweak it in a bugfix release

[takluyver]

I haven't, but I admit I haven't done extensive testing. Did you grab the
traceback when it crashed? It should have been saved to a file. It might
equally be a problem in IPython - I don't think many things use the custom
completions API.

[wesm]

I updated to ipython git master yesterday and haven't had any issues since so it could have been something transient. 0.11 seems to also be OK, but will wait for users to report I suppose. It is nice to have

[wesm]

boom:

In [38]: df.sort_i---------------------------------------------------------------------------
TypeError                         Python 2.7.2: /usr/lib/epd-7.1/bin/python
                                                   Thu Oct 20 11:07:50 2011
A problem occured executing Python code.  Here is the sequence of function
calls leading up to the error, with the most recent (innermost) call last.
/home/wesm/code/repos/ipython/IPython/core/completer.pyc in complete(self=<IPython.core.completer.IPCompleter object>, text='df.sort_i', line_buffer='df.sort_i', cursor_pos=9)
    810 
    811         # Start with a clean slate of completions
    812         self.matches[:] = []
    813         custom_res = self.dispatch_custom_completer(text)
    814         if custom_res is not None:
    815             # did custom completers produce something?
    816             self.matches = custom_res
    817         else:
    818             # Extend the list of completions with the results of each
    819             # matcher, so we return results to the user from all
    820             # namespaces.
    821             if self.merge_completions:
    822                 self.matches = []
    823                 for matcher in self.matchers:
    824                     try:
--> 825                         self.matches.extend(matcher(text))
    826                     except:
    827                         # Show the ugly traceback if the matcher causes an
    828                         # exception, but do NOT crash the kernel!
    829                         sys.excepthook(*sys.exc_info())
    830             else:
    831                 for matcher in self.matchers:
    832                     self.matches = matcher(text)
    833                     if self.matches:
    834                         break
    835         # FIXME: we should extend our api to return a dict with completions for
    836         # different types of objects.  The rlcomplete() method could then
    837         # simply collapse the dict into a list for readline, but we'd have
    838         # richer completion semantics in other evironments.
    839         self.matches = sorted(set(self.matches))
    840         #io.rprint('COMP TEXT, MATCHES: %r, %r' % (text, self.matches)) # dbg

/home/wesm/code/repos/ipython/IPython/core/completer.pyc in python_matches(self=<IPython.core.completer.IPCompleter object>, text='df.sort_i')
    604         if ' ' in main_text and not main_text.startswith('sudo'):
    605             return []
    606         text = os.path.expanduser(text)
    607         aliases =  self.alias_table.keys()
    608         if text == '':
    609             return aliases
    610         else:
    611             return [a for a in aliases if a.startswith(text)]
    612 
    613     def python_matches(self,text):
    614         """Match attributes or global python names"""
    615 
    616         #io.rprint('Completer->python_matches, txt=%r' % text) # dbg
    617         if "." in text:
    618             try:
--> 619                 matches = self.attr_matches(text)
    620                 if text.endswith('.') and self.omit__names:
    621                     if self.omit__names == 1:
    622                         # true if txt is _not_ a __ name, false otherwise:
    623                         no__name = (lambda txt:
    624                                     re.match(r'.*\.__.*?__',txt) is None)
    625                     else:
    626                         # true if txt is _not_ a _ name, false otherwise:
    627                         no__name = (lambda txt:
    628                                     re.match(r'.*\._.*?',txt) is None)
    629                     matches = filter(no__name, matches)
    630             except NameError:
    631                 # catches <undefined attributes>.<tab>
    632                 matches = []
    633         else:
    634             matches = self.global_matches(text)

/home/wesm/code/repos/ipython/IPython/core/completer.pyc in attr_matches(self=<IPython.core.completer.IPCompleter object>, text='df.sort_i')
    377             expr, attr = m2.group(1,2)
    378         else:
    379             return []
    380 
    381         try:
    382             obj = eval(expr, self.namespace)
    383         except:
    384             try:
    385                 obj = eval(expr, self.global_namespace)
    386             except:
    387                 return []
    388 
    389         words = dir2(obj)
    390 
    391         try:
--> 392             words = generics.complete_object(obj, words)
    393         except TryNext:
    394             pass
    395         # Build match list to return
    396         n = len(attr)
    397         res = ["%s.%s" % (expr, w) for w in words if w[:n] == attr ]
    398         return res
    399 
    400 
    401 class IPCompleter(Completer):
    402     """Extension of the completer class with IPython-specific features"""
    403 
    404     def _greedy_changed(self, name, old, new):
    405         """update the splitter and readline delims when greedy is changed"""
    406         if new:
    407             self.splitter.set_delims(GREEDY_DELIMS)

/home/wesm/code/repos/ipython/IPython/external/simplegeneric/_simplegeneric.pyc in dispatch(*args=(         0       1      2       3       4     
x...1
p  q  r  0.344   0.71   2.196  -1.939  -1.103 
, ['T', '_AXIS_ALIASES', '_AXIS_NAMES', '_AXIS_NUMBERS', '__add__', '__array__', '__array_wrap__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dict__', '__div__', '__doc__', '__eq__', '__floordiv__', '__format__', '__ge__', '__getattr__', '__getattribute__', ...]), **kw={})
     70             for o in obs:
     71                 if _by_object.setdefault(id(o), (o,f))[1] is not f:
     72                     raise TypeError(
     73                         "%r already has method for object %r" % (func, o)
     74                     )
     75             return f
     76         return decorate
     77 
     78 
     79     def dispatch(*args, **kw):
     80         f = _gbo(id(args[0]), _sentinel)
     81         if f is _sentinel:
     82             for t in type(args[0]).__mro__:
     83                 f = _gbt(t, _sentinel)
     84                 if f is not _sentinel:
---> 85                     return f(*args, **kw)
     86             else:
     87                 return func(*args, **kw)
     88         else:
     89             return f[1](*args, **kw)
     90 
     91     dispatch.__name__       = func.__name__
     92     dispatch.__dict__       = func.__dict__.copy()
     93     dispatch.__doc__        = func.__doc__
     94     dispatch.__module__     = func.__module__
     95 
     96     dispatch.when_type = when_type
     97     dispatch.when_object = when_object
     98     dispatch.default = func
     99     dispatch.has_object = lambda o: id(o) in _by_object
    100     dispatch.has_type   = lambda t: t in _by_type

/home/wesm/code/pandas/pandas/core/frame.pyc in complete_dataframe(obj=         0       1      2       3       4     
x...1
p  q  r  0.344   0.71   2.196  -1.939  -1.103 
, prev_completions=['T', '_AXIS_ALIASES', '_AXIS_NAMES', '_AXIS_NUMBERS', '__add__', '__array__', '__array_wrap__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dict__', '__div__', '__doc__', '__eq__', '__floordiv__', '__format__', '__ge__', '__getattr__', '__getattribute__', ...])
   3204         homogenized[k] = v
   3205 
   3206     return homogenized
   3207 
   3208 def _put_str(s, space):
   3209     return ('%s' % s)[:space].ljust(space)
   3210 
   3211 def install_ipython_completers():
   3212     """Register the DataFrame type with IPython's tab completion machinery, so
   3213     that it knows about accessing column names as attributes."""
   3214     from IPython.utils.generics import complete_object
   3215 
   3216     @complete_object.when_type(DataFrame)
   3217     def complete_dataframe(obj, prev_completions):
   3218         return prev_completions + [c for c in obj.columns \
-> 3219                                                 if py3compat.isidentifier(c)]
   3220 
   3221 # Importing IPython brings in about 200 modules, so we want to avoid it unless
   3222 # we're in IPython (when those modules are loaded anyway).
   3223 if "IPython" in sys.modules:
   3224     try:
   3225         install_ipython_completers()
   3226     except Exception:
   3227         pass
   3228 
   3229 
   3230 if __name__ == '__main__':
   3231     import nose
   3232     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],
   3233                    exit=False)

/home/wesm/code/pandas/pandas/util/py3compat.pyc in isidentifier(s=0, dotted=False)
      1 import sys
      2 
      3 PY3 = (sys.version_info[0] >= 3)
      4 
      5 if PY3:
      6     def isidentifier(s):
      7         return s.isidentifier()
      8 
      9 else:
     10     # Python 2
     11     import re
     12     _name_re = re.compile(r"[a-zA-Z_][a-zA-Z0-9_]*$")
     13     def isidentifier(s, dotted=False):
---> 14         return bool(_name_re.match(s))

TypeError: buffer size mismatch

**********************************************************************

Oops, ipython crashed. We do our best to make it stable, but...

A crash report was automatically generated with the following information:
  - A verbatim copy of the crash traceback.
  - A copy of your input history during this session.
  - Data on your current ipython configuration.

It was left in the file named:
    '/home/wesm/.ipython/Crash_report_ipython.txt'
If you can email this file to the developers, the information in it will help
them in understanding and correcting the problem.

You can mail it to: Fernando Perez at fperez.net@gmail.com
with the subject 'ipython Crash Report'.

If you want to do it now, the following command will work (under Unix):
mail -s 'ipython Crash Report' fperez.net@gmail.com < /home/wesm/.ipython/Crash_report_ipython.txt

To ensure accurate tracking of this issue, please file a report about it at:
http://github.com/ipython/ipython/issues


---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/home/wesm/code/pandas/<ipython-input-38-c82c6e0d9c3c> in <module>()
----> 1 df.sort_i

/home/wesm/code/pandas/pandas/core/frame.pyc in __getattr__(self, name)
    878             return self[name]
    879         raise AttributeError("'%s' object has no attribute '%s'" % \
--> 880                                                     (type(self).__name__, name))
    881 
    882     def __setitem__(self, key, value):

AttributeError: 'DataFrame' object has no attribute 'sort_i'

[takluyver]

Hmmm, that's two bugs in one go, because IPython shouldn't be susceptible to errors in custom completer functions. That will need hardening.

I haven't seen the 'buffer size mismatch' error before. I'll look into it. Do you know what the column names were when this happened?

[wesm]

They were integers. That's almost certainly the problem

[takluyver]

I can get a different TypeError ("expected string or buffer") if the argument to match() isn't a string. I guess we need an isinstance(c, basestring) in there.

[takluyver]

Also opened an IPython bug here: ipython/ipython#907