Inject display into builtins #10596

Carreau · 2017-05-24T17:55:39Z

display is an underutilized feature of IPython which is relatively hard
to access to. You need to explicitly import from IPython.display,
which not only is long to type, but users get confused if they should
use IPython.display or IPython.display.display also many tutorials
tell users to import from IPython.core.display which is not right.

So inject that directly in builtin module – so that user can overwrite
and delete without issues – and document display a lot more.

cc @rgbkrk, @ivanov

rgbkrk · 2017-05-24T17:59:14Z

🙇 Thank you!

rgbkrk · 2017-05-24T18:00:50Z

IPython/core/display.py

+
+    >>> del plain.type_printers[int]
+    >>> display(3+6)
+    9


This is so great to have these examples.

They are also in the Custom Display logic notebook.

Totally minor nitpick, but it'd be even more clear if the display calls both used the same expression inside...
Make both display(3+6) for example, so the top one is even more clear about how it's doing the formatting...

rgbkrk · 2017-05-24T18:00:50Z

IPython/core/interactiveshell.py

@@ -618,6 +619,7 @@ def init_builtins(self):
        # removing on exit or representing the existence of more than one
        # IPython at a time.
        builtin_mod.__dict__['__IPYTHON__'] = True
+        builtin_mod.__dict__['display'] = display


away we go!

I think that pixiedust should fallback on ipython's display

Makes sense, I'll enter a Github issue in PixieDust repo

Makes sense, I'll enter a Github issue in PixieDust repo

Thanks !

Or, if we can figure out a way to use IPython machinery for pixeldust (as we already have dispatch per object types) that would be great. Feel free to crosslink here or ping me.

rgbkrk · 2017-05-24T18:26:14Z

I was recently alerted to the fact that pixiedust exposes display as a top level for use in the notebook, likely to match up with zeppelin/databricks/etc., which is also why I said we should expose display by default.

The extra thing most of these other displayers do though is hook the display for data frames (spark and pandas) and Spark GraphFrames.

rgbkrk · 2017-05-24T18:26:25Z

/cc @DTAIEB

Carreau · 2017-05-24T18:44:00Z

I was recently alerted to the fact that pixiedust exposes display as a top level for use in the notebook, likely to match up with zeppelin/databricks/etc., which is also why I said we should expose display by default.

Where is it injected ? in user_ns or builtins ? AFAICT that will lead to user having:

Pixiedust didn't find any vizualization plugins that can render this object: <__main__.Foo object at 0x7fd58600d810>

Or whatever object is given to it. I think that pixiedust should fallback on ipython's display.
It should even to hook into IPython formatter. Right now if you display(df) you get a nice rich representation. If you just return it as the last statemtn of the cell you get a DataFrame[Colors: string, %: bigint] output, which is less nice.

Registering formatter should allow to have rich representation in both case which should make the experience more uniform for users.

The other things is that with this PR IPython's display is restored as soon as you delete any things that shadows it:

$ ipython
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 13:19:00)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: display
Out[1]: <function IPython.core.display.display>

In [2]: display = lambda x:X

In [3]: display
Out[3]: <function __main__.<lambda>>

In [4]: del display

In [5]: display
Out[5]: <function IPython.core.display.display>

ivanov

Looks sweet, Matthias! With the display_update stuff, I think display is an even more powerful feature now that should available more prominently.

ivanov · 2017-05-24T19:01:44Z

IPython/core/display.py

+
+    >>> del plain.type_printers[int]
+    >>> display(3+6)
+    9


Totally minor nitpick, but it'd be even more clear if the display calls both used the same expression inside...
Make both display(3+6) for example, so the top one is even more clear about how it's doing the formatting...

ivanov · 2017-05-24T19:01:44Z

IPython/core/display.py

+    You can refer to the documentation on IPython display formatters in order to
+    register custom formatters for already existing types.
+
+    since IPython 5.4 and 6.1 display is automatically made availlable to the


" Since IPython 5.4 and 6.1 , " (capital S, comma after 6.1)

ivanov · 2017-05-24T19:01:44Z

IPython/core/display.py

+
+    since IPython 5.4 and 6.1 display is automatically made availlable to the
+    user without import. If you are using display in a document that might be
+    used in a pure python context or with older version of IPython use the


comma after IPython

ivanov · 2017-05-24T19:01:44Z

IPython/core/display.py

    """
    raw = kwargs.pop('raw', False)
    if transient is None:
        transient = {}
    if display_id:
-        if display_id == True:
+        if display_id is True:


Nice bug fix! display_id=1 would have always generated a new ID for this.

ivanov · 2017-05-24T19:01:44Z

IPython/core/displayhook.py

@@ -16,6 +16,8 @@
 from traitlets import Instance, Float
 from warnings import warn

+from IPython.display import display


Not sure why this import is here. Could you document that if it needs to stay here?

rgbkrk · 2017-05-24T19:03:36Z

The other things is that with this PR IPython's display is restored as soon as you delete any things that shadows it:

Excellent!

minrk · 2017-05-25T01:55:21Z

Great!

@ivanov since you mention the update display stuff, do you think update_display should be included, or is that too much? I realize builtins is a pretty precious namespace.

Minor technical detail: should this go in .user_ns_hidden instead of builtins, so it doesn't show up as available in module code?

Carreau · 2017-05-25T02:08:56Z

Minor technical detail: should this go in .user_ns_hidden instead of builtins, so it doesn't show up as available in module code?

It can, I did that originally, it's more complicated if we want to re-inject it when user delete it/shadow it, but doable.

I believe update_display is more a technical under the hood detail, while everyone should know about (and use) display.

This is still a relatively heavy usability change so I'm probing what other are thinking. I'd like @fperez input on this one as well (not on the code itself, but happy to get review) but wether it's ok to have display() available by default.

rgbkrk · 2017-05-25T05:02:17Z

I agree that we shouldn't surface update_display. We can still use update = True on display anyways.

Carreau · 2017-05-25T18:35:55Z

On the builtins vs user_ns_hidden:

When in user_ns_hidden it happily accept to delete it but the object will reapear:

$ ipython
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 13:19:00)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: display
Out[1]: <function IPython.core.display.display>

In [2]: display = 1

In [3]: del display

In [4]: display
Out[4]: <function IPython.core.display.display>

In [5]: del display

In [6]: display
Out[6]: <function IPython.core.display.display>

If in builtins, il will complain that object can't be found if trying to delete it:

$ ipython
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 13:19:00)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: display
Out[1]: <function IPython.core.display.display>

In [2]: display = 1

In [3]: display
Out[3]: 1

In [4]: del display

In [5]: display
Out[5]: <function IPython.core.display.display>

In [6]: del display
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-78e18c419bea> in <module>()
----> 1 del display

NameError: name 'display' is not defined

The user_ns_hidden also have the case that the object reapears only after each cell execution so cutting cell may change behavior. For example NameError vs NoError

In [1]: display=1
   ...: print(display)
   ...: del display
   ...: print(display)
   ...:
1
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-bb2e1216e38f> in <module>()
      2 print(display)
      3 del display
----> 4 print(display)

NameError: name 'display' is not defined

In [2]: display = 1
   ...: print(display)
   ...: del display
1

In [3]: print(display)
<function display at 0x102dfbd08>

We can also not reinject it, but I believe it is the wrong solution.

Carreau · 2017-05-25T18:46:29Z

Or, we can change the exec for exec(code_obj, self.user_global_ns, ChainMap(self.user_ns, hidden_ns))

(you can't ChainMap global, exec refuses), that should avoid a lot of the hidden_ns dance we do.

display is an underutilized feature of IPython which is relatively hard to access to. You need to explicitly import from `IPython.display`, which not only is long to type, but users get confused if they should use `IPython.display` or `IPython.display.display` also many tutorials tell users to import from IPython.core.display which is not right. So inject that directly in builtin module – so that user can overwrite and delete without issues – and document display a lot more.

Carreau · 2017-05-29T02:40:19Z

Rebased, updated, tested, documented.

Carreau · 2017-05-30T00:26:16Z

@fperez seem to +1,
@jasongrout was a good devil advocate and seem to agree this is a good idea
@willingc seem to thing for users.

There is one (longstanding) issue that calling display in a pure Python terminal break the repl by instantiating an InteractiveShell (which has crazy side effects like nuking your namespace and setting sys.ps1), I'm unsure it's a blocker for this PR.

takluyver · 2017-05-30T13:59:18Z

But that will only happen if you start a plain Python shell after starting IPython in the same process, right? I don't think that comes up very often.

Carreau · 2017-05-30T18:39:24Z

But that will only happen if you start a plain Python shell after starting IPython in the same process, right? I don't think that comes up very often.

Not even, it break if you just call display in a pure python repl. We can fix that, the only question if what to do. If Interactiveshell._instance is None fallback on print ? Even if raw=True ?

$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 13:19:00)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython.display import display
>>> a = 1
>>> display(a)
1
In : a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
In :

But that's another issue.

rgbkrk · 2017-05-30T18:44:14Z

@meeseeksdev backport

meeseeksdev · 2017-05-30T18:44:25Z

Oops, something went wrong applying the patch... Please have a look at my logs.

Merge pull request ipython#10596 from Carreau/display-builtin Inject display into builtins

Backport PR #10596 on branch 5.x

takluyver · 2017-05-31T10:29:32Z

Oh, I see, you mean if code explicitly imports it. That's not affected by injecting display into builtins when IPython runs, though.

Carreau · 2017-05-31T15:46:38Z

Oh, I see, you mean if code explicitly imports it. That's not affected by injecting display into builtins when IPython runs, though.

Yes, sorry the missing datapoint is:
If we inject it into builtins, we might want nbconvert --to script to add the import at the top.

takluyver · 2017-05-31T16:16:50Z

Ah, that makes sense. Yeah, that's tricky.

Carreau requested review from ivanov and rgbkrk May 24, 2017 17:56

rgbkrk approved these changes May 24, 2017

View changes

Carreau force-pushed the display-builtin branch from c3041ad to bad9eea Compare May 24, 2017 18:12

ivanov reviewed May 24, 2017

View changes

ivanov mentioned this pull request May 24, 2017

display_id = 1 always generates a new ID #10597

Closed

minrk approved these changes May 25, 2017

View changes

Carreau force-pushed the display-builtin branch 2 times, most recently from 4e5fd23 to b3c4dfc Compare May 25, 2017 02:17

takluyver mentioned this pull request May 26, 2017

Release 6.1 and 5.4 soon ? #10603

Closed

Carreau added 3 commits May 28, 2017 18:39

Add test that display is available without imports.

10be44c

Document that display is there by default

1dc063d

Carreau force-pushed the display-builtin branch from b3c4dfc to 1dc063d Compare May 29, 2017 02:38

Carreau added this to the 5.4 milestone May 29, 2017

rgbkrk approved these changes May 29, 2017

View changes

Carreau mentioned this pull request May 30, 2017

display() instantiates a shell in pure Python #10617

Closed

rgbkrk merged commit 4f93bf1 into ipython:master May 30, 2017

Carreau pushed a commit to Carreau/ipython that referenced this pull request May 30, 2017

Backport PR ipython#10596 on branch 5.x

40c551f

Merge pull request ipython#10596 from Carreau/display-builtin Inject display into builtins

Carreau mentioned this pull request May 30, 2017

Backport PR #10596 on branch 5.x #10619

Merged

Carreau added the backported PR that have been backported by MrMeeseeks label May 30, 2017

Carreau added a commit that referenced this pull request May 30, 2017

Merge pull request #10619 from Carreau/backport-prs-display-in-builtinss

dece91b

Backport PR #10596 on branch 5.x

Carreau deleted the display-builtin branch May 31, 2017 17:54

Inject display into builtins #10596

Inject display into builtins #10596

Carreau commented May 24, 2017

rgbkrk commented May 24, 2017

rgbkrk May 24, 2017

Carreau May 24, 2017

ivanov May 24, 2017

rgbkrk May 24, 2017

DTAIEB May 24, 2017

Carreau May 25, 2017

rgbkrk commented May 24, 2017

rgbkrk commented May 24, 2017

Carreau commented May 24, 2017

ivanov left a comment

ivanov May 24, 2017

ivanov May 24, 2017

ivanov May 24, 2017

ivanov May 24, 2017

ivanov May 24, 2017

rgbkrk commented May 24, 2017

minrk commented May 25, 2017

Carreau commented May 25, 2017

rgbkrk commented May 25, 2017

Carreau commented May 25, 2017

Carreau commented May 25, 2017

Carreau commented May 29, 2017

Carreau commented May 30, 2017 •

edited

takluyver commented May 30, 2017

Carreau commented May 30, 2017

rgbkrk commented May 30, 2017

meeseeksdev bot commented May 30, 2017

takluyver commented May 31, 2017

Carreau commented May 31, 2017

takluyver commented May 31, 2017

Inject display into builtins #10596

Inject display into builtins #10596

Conversation

Carreau commented May 24, 2017

rgbkrk commented May 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgbkrk commented May 24, 2017

rgbkrk commented May 24, 2017

Carreau commented May 24, 2017

ivanov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgbkrk commented May 24, 2017

minrk commented May 25, 2017

Carreau commented May 25, 2017

rgbkrk commented May 25, 2017

Carreau commented May 25, 2017

Carreau commented May 25, 2017

Carreau commented May 29, 2017

Carreau commented May 30, 2017 • edited

takluyver commented May 30, 2017

Carreau commented May 30, 2017

rgbkrk commented May 30, 2017

meeseeksdev bot commented May 30, 2017

takluyver commented May 31, 2017

Carreau commented May 31, 2017

takluyver commented May 31, 2017

Carreau commented May 30, 2017 •

edited