Add logic for automatic display in ipython

[ellisonbg]

Using the IPython display architecture you can register functions that are automatically called when a DataFrame is displayed. This will enable users to optionally enable qgrid to display DataFrame by default, without an extra call. Running on a plane now, but please ping the ipython devs (or me later) on how this can be done. Awesome work!

[ssanderson]

@ellisonbg sounds good. Would the basic expected syntax be something like:

import qgrid

qgrid.register_displayhook()

# Displays with qgrid automatically.
pd.DataFrame(...)

[TimShawver]

I think the registration code is explained in the same Custom Display Logic example notebook I used a bunch:
http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/IPython%20Kernel/Custom%20Display%20Logic.ipynb

Based on that example I think our registration code would look something like this:

ip = get_ipython()
html_f = ip.display_formatter.formatters['text/html']
html_f.for_type_by_name('pandas', 'DataFrame', show_grid_function)

[ellisonbg]

Yep, here is the code I am using in a different library to define an automatic display function for numpy arrays:

def enable():
    """Enable automatic visualization of NumPy arrays in the IPython Notebook."""
    try:
        from IPython.core.getipython import get_ipython
    except ImportError:
        raise ImportError('This feature requires IPython 1.0+')
    ip = get_ipython()
    f = ip.display_formatter.formatters['text/html']
    f.for_type(np.ndarray, _array_to_html)


def disable():
    """Disable automatic visualization of NumPy arrays in the IPython Notebook."""
    try:
        from IPython.core.getipython import get_ipython
    except ImportError:
        raise ImportError('This feature requires IPython 1.0+')
    ip = get_ipython()
    f = ip.display_formatter.formatters['text/html']
    f.type_printers.pop(np.ndarray, None)

The naming of these types of functions tends to be something like enable/disable or enable_notebook/disable_notebook. These function are also a good place to load the JS on the page as well.

[TimShawver]

Awesome, thanks @ellisonbg!

[ellisonbg]

Question - do you always send the entire data back to the browser? Do you
allow for the setting of an upper limit on the size of the data that can be
send back? Sending a huge dataframe to the browser would definitely kill
things...

Then there is a question of how you subsample the frame...

On Sat, Oct 11, 2014 at 3:38 PM, Tim Shawver notifications@github.com
wrote:

Awesome, thanks @ellisonbg https://github.com/ellisonbg!

—
Reply to this email directly or view it on GitHub
#8 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[ellisonbg]

Also, this should be pretty easy to turn into a widget that allows data to
be edited in the widget and saved in the DataFrame...

On Sat, Oct 11, 2014 at 3:41 PM, Brian Granger ellisonbg@gmail.com wrote:

Question - do you always send the entire data back to the browser? Do you
allow for the setting of an upper limit on the size of the data that can be
send back? Sending a huge dataframe to the browser would definitely kill
things...

Then there is a question of how you subsample the frame...

On Sat, Oct 11, 2014 at 3:38 PM, Tim Shawver notifications@github.com
wrote:

Awesome, thanks @ellisonbg https://github.com/ellisonbg!

—
Reply to this email directly or view it on GitHub
#8 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[TimShawver]

Currently I do always send all of the data back, and yes a huge DataFrame will freeze the UI (I think the limit is around 50K rows right now). I like the idea of allowing people to set an upper limit on the amount of data.

Writing back to the DataFrame would be awesome. SlickGrid does support editing cells so it's probably not a ton of work.

[ellisonbg]

OK, maybe add a set_limit() function (for the automatic display) and a
limit kwarg for explicit display and then play around with what a good
default should be?

To get the write back, we will have to figure out how to tie it into the
widget fraemwork as well. Shouldn't be too bad, we have been meaning to
figure out how to have a single object that can work as display and
widget...

Cheers,

Brian

On Sat, Oct 11, 2014 at 4:03 PM, Tim Shawver notifications@github.com
wrote:

Currently I do always send all of the data back, and yes a huge DataFrame
will freeze the UI (I think the limit is around 50K rows right now). I like
the idea of allowing people to set an upper limit on the amount of data.

Writing back to the DataFrame would be awesome. SlickGrid does support
editing cells so it's probably not a ton of work.

—
Reply to this email directly or view it on GitHub
#8 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[TimShawver]

Yep that sounds reasonable. Thanks again for the help.

[ssanderson]

One thing that's tricky here is that we're currently using both display_html and display_javascript inside the _ipython_display_ method for SlickGrid, and the automatic display logic above relies on the output only having one mimetype. My guess is the fix for this is to combine the html and javascript into a single html chunk by putting the javascript pieces in a <script> block.

[ellisonbg]

Ahh, what we really need to do is have a display function registration for
the _ipython_display_ method as well. Can you open an issue and ping
@minrk on that.

On Sun, Oct 12, 2014 at 4:35 AM, Scott Sanderson notifications@github.com
wrote:

One thing that's tricky here is that we're currently using both
display_html and display_javascript inside the ipython_display method
for SlickGrid, and the automatic display logic above relies on the output
only having one mimetype. My guess is the fix for this is to combine the
html and javascript into a single html chunk by putting the javascript
pieces in a <script> block.

—
Reply to this email directly or view it on GitHub
#8 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[ssanderson]

Opened up an issue in the main IPython repo here: ipython/ipython#6687.

[ellisonbg]

Here is the code that can do this:

https://gist.github.com/ellisonbg/5c54f995bfcd64b90dc2

[ellisonbg]

ping @aggFTW

[ssanderson]

@ellisonbg do you have thoughts on what a reasonable API for enabling this would be?

Presumably it'd be something like:

from qgrid import autouse

# Internally sets a display formatter for DataFrame and/or Series
autouse()

where the signature of autouse would be something like:

def autouse(autouse_dataframe=True, autouse_series=<what should this be?>):

[ssanderson]

(The other option would be to enable this by default as an import-time side effect, but that feels too aggressive.)

[TimShawver]

Finally got some time to do some dev work on qgrid and I included the automatic display capability in the new qgrid 1.0 beta. There are two new methods enable and disable. Here are the docs for these functions: http://qgrid.readthedocs.io/en/latest/#qgrid.enable

I don't think I'll have to add a set_limit function anymore because I ended up implementing "virtual scrolling" so that we only send 200 or so rows down to the browser at a time (which means qgrid can render very large DataFrames now).

See the installation instructions on the readme if you'd like to try out the new beta.