Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for image/png format in the terminal #10610

Open
wants to merge 6 commits into
base: master
from

Conversation

@stephanh42
Copy link

stephanh42 commented May 28, 2017

Introduction

There are nowadays several terminal emulators which support showing images in the terminal.
On such terminal emulators the terminal IPython version can support the image/png mime type.

The following is a summary of such terminal emulators to the extend that I am aware of them.

  • ITerm2 on macOS
  • Terminology, the terminal emulator of Enlightenment
  • All terminal emulators which support the Sixel protocol

In all these cases there is a simple program which takes the name of the image file and just displays
it on the terminal. E.g. for ITerm2 there is the catimg script which used like this:
catimg someimage.png
Similar (but unfortunately different) programs exist for the other terminal emulators mentioned.

This pull request introduces an environment variable IPYTHON_IMAGE_VIEWER . When set, the terminal IPython will use the named program to view an image of type image/png .

Unresolved issues

  • I decided to use an environment variable and not a configurable. This is because IPYTHON_IMAGE_VIEWER really describes a property of the terminal and is therefore a bit analogous to TERM. However this is probably something to be discussed.
  • It should now be possible to use %matplotlib inline in the terminal but I couldn't figure out how to enable that.
  • I am unsure how to write a testcase for this.
@Carreau

This comment has been minimized.

Copy link
Member

Carreau commented May 28, 2017

@stephanh42

This comment has been minimized.

Copy link
Author

stephanh42 commented May 28, 2017

@Carreau

I looked at the iterm2-tools code but it is unclear what it is supposed to do in IPython, since that is not documented and the code does not, in fact, work, with the current IPython.

There is at least no suggestion in the source that it is trying to anything with image/png mime types, or that it even enables the generation of that mime type in terminal IPython. The example image shows some arrows being added to the IPython prompts.

image_data = format_dict.get("image/png")
if image_data is not None and self.view_image(image_data):
return

This comment has been minimized.

Copy link
@Carreau

Carreau May 28, 2017

Member

Hum that seem to be wrong, you likely do not want to modify anything in IPython.core as it will affect kernels regardless of the frontend. You likely want to get the data in the frontend and display it. It might be hard to find I'll see if I can have shot I finding where to put this.

This comment has been minimized.

Copy link
@stephanh42

stephanh42 May 29, 2017

Author

Hi @Carreau,

I moved the code from core/displayhook.py to the RichPromptDisplayHook in terminal/prompts.py . Hope this is better.

@Carreau Carreau modified the milestone: wishlist May 28, 2017
md_dict : dict (optional)
The metadata dict to be associated with the display data.
"""
image_data = format_dict.get("image/png")

This comment has been minimized.

Copy link
@Carreau

Carreau Jun 1, 2017

Member

I think we might want to dispatch on image/* but i'm ok starting with just image/png

This comment has been minimized.

Copy link
@stephanh42

stephanh42 Jun 3, 2017

Author

I wanted to avoid a situation where we also have to specify a list of valid MIME types for each viewer. The ones I checked all support png.

This comment has been minimized.

Copy link
@Carreau

Carreau Jun 5, 2017

Member

I wanted to avoid a situation where we also have to specify a list of valid MIME types for each viewer. The ones I checked all support png.

That's fair, and I don't think many backend do return anything other than PNG.

try:
# We don't want to clutter the display with errors on failure,
# so redirect stderr to DEVNULL.
subprocess.check_call(tuple(image_viewer) + (png_file,), stderr=subprocess.DEVNULL)

This comment has been minimized.

Copy link
@Carreau

Carreau Jun 1, 2017

Member

we likely need something more complex, as image viewer may take arguments to open in a non-blocking way.

This comment has been minimized.

Copy link
@stephanh42

stephanh42 Jun 3, 2017

Author

Hi @Carreau ,

Please note that I already allow options to come before the filename, by using shlex.split on the value of the environment variable. Do you think something even more complex is needed, i.e. a syntax like

viewer -some_option %s -another_option

where %s is then replaced by the actual filename?

Stephan

This comment has been minimized.

Copy link
@Carreau

Carreau Jun 3, 2017

Member

Ah I missed the shlex.split(), no I think this is fine. I"m wondering if we should use the normal IPython config system that allow different option based on profile or sub-application. I was also thinking that we might want to extend this mechnism to other mimetype (I don't see why an MP3, or GeoJson file could not have a viewer), but that a question for likely another PR.

This comment has been minimized.

Copy link
@stephanh42

stephanh42 Jun 3, 2017

Author

Hi @Carreau,

Yes, I was wondering about the configurable too. Perhaps that is better after all.

Perhaps a syntax like this:

c.TerminalIPythonApp.ExternalViewers = {
      "image/png": "imgcat",
      "video/mp4": "some-video-player"
 }

I can make this change if you wish.

Stephan

This comment has been minimized.

Copy link
@Carreau

Carreau Jun 5, 2017

Member

Let's get that (with just images) in first. We can figure out the rest after. I'm trying to have it work locally.

This comment has been minimized.

Copy link
@Carreau

Carreau Jun 5, 2017

Member

ok, so that works with blocking image viewers, not with non blocking one as the temporary directory will be torn down before the viewer actually read the the file. We'll have to (likely) attach the teardown to a sys.exit hook, or to the del method of RichPromptDisplayHook. Maybe with a cache to not let more than a couple of things open.
I'm also thinking that the viewer should likely not be a subprocess, but a python callable, so that you can hook into pure-python display system like for example inline images.

We'll have to also figure out a clean way to tell matplotlib it can use inline.

This comment has been minimized.

Copy link
@stephanh42

stephanh42 Jun 6, 2017

Author

Hi @Carreau,

The image viewers which use the terminal imaging capability are all blocking, but they execute very quickly. Basically, they just send a bunch of control sequences to stdout and then exit. I am not sure what kind of non-blocking viewers should be supported.

Viewers which show the result in an external window already work for matplotlib. But those don't rely on external programs.

@Carreau

This comment has been minimized.

Copy link
Member

Carreau commented Jun 1, 2017

I'm happier this way I think we can get something along these lines. Let's get IPython.display.Image to work in the terminal, and we can figure the rest for matplotlib which will be a bit more complicated.
We have a loong meeting this week, but I add that on my todo list.

@Carreau

This comment has been minimized.

Copy link
Member

Carreau commented Jun 6, 2017

Hi @Carreau,

The image viewers which use the terminal imaging capability are all blocking, but they execute very quickly. Basically, they just send a bunch of control sequences to stdout and then exit. I am not sure what kind of non-blocking viewers should be supported.

Viewers which show the result in an external window already work for matplotlib. But those don't rely on external programs.

Yes, but you don't need an external viewer to see int he terminal, it's relatively straitforward to write in pure python:

# for iterm
print('\033]1337;File=inline=1;width=100%%:%s\a\n' % data.decode())

Which is way easier to ensure we have available (heck if it's less than 12 lines we can bundle it and special case it), so it would be nice to have the ability to have this be a python callable than to have to subprocess.

An example of non-blocking viewer on OSX is open, which will manage to open almost ay file (not only PNG) in a separate windows. As we can't know that user will use it only for inline view, we need to at least make some guaranties that we try to handle thing correctly.

@stephanh42

This comment has been minimized.

Copy link
Author

stephanh42 commented Jun 7, 2017

Hi @Carreau,

You bring up several issues, so let me discuss them point-by-point.

  1. Callable instead of command line to execute.
    I am fine with this . It would be useful to have some
    convenience callables readily available.

    I would propose the following convenience callables:
    * viewer for blocking external program (like imgcat)
    * viewer for external program like, open, xdg-open, start
    * built-in viewer for iTerm2
    See below for rationale.

  2. Support for non-blocking image viewers like "open".
    These need to be treated differently from viewers like "imgcat".
    For "imgcat" and similar viewers we need to be sure it is done before
    anything else is printed. We could have a dedicated convenience callable for
    viewers like "open" on macOS or "xdg-open" on Linux or "start" on Windows.

  3. Direct implementation of control codes in Python vs calling external viewer.
    For iTerm2 it is simple, but sixel is its own format and the conversion from PNG
    is non-trivial. I tried to avoid to rely on the existing libsixel (C library).
    Perhaps a good compromise would be to put sixel support in a separate PIP-installable
    package. I can also check if a pure-Python implementation is feasible but I think
    it will need in any case to depend on something like PIL/Pillow to provide PNG decode
    support.

Thanks,

Stephan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.