Graphics rendering

[kovidgoyal]

Thoughts on implementing support for raster graphics in kitty (and terminals more generally). Out of curiosity, I spent some time looking into the existing imaging solutions in terminals, I found:

1. https://github.com/saitoha/libsixel (seems the most advanced, but is fundamentally limited by backwards compatibility)
2. https://www.iterm2.com/documentation-images.html (pretty basic, only supports displaying image files)
3. https://git.enlightenment.org/apps/terminology.git/tree/README (seems to be file based so useless over ssh)
4. https://github.com/withoutboats/notty (seems largely similar to (2))

My question is, why are we limiting ourselves to this image file display paradigm? Why not allow programs to render arbitrary pixel data in the terminal? The way I envision this working is:

1. An escape code that allows programs running in the terminal to query the terminal for the current character cell size in pixels (this is similar to how querying for cursor position works)

2. An escape code that allows the program running in the terminal to specify arbitrary pixel data to render at the current cursor position (in a single cell, think of it as sending a "graphical character" instead of text character). The pixel data can be binary for maximum efficiency (taking care to escape the C0 control codes for maximum robustness).

With these two primitives, programs will be able to draw arbitrary graphics (including image files) in terminals. This, to me, seems like a more general, and powerful, abstraction to build rather than just the ability to send image files in a few formats.

I am considering building this into kitty, so I thought, that before I do so, it would be good get some more opinions on the subject. Maybe get a little consensus going. Note that once this is built it is easy to support displaying image files on top of it, if needed.

A specification for this protocol is here: https://github.com/kovidgoyal/kitty/blob/gr/graphics-protocol.asciidoc

Progress on implementing the specification:

• Loading image data
• Displaying images (any loaded image can be displayed multiple times). This is a little tricky because we want support for z-index. In particular images with negative z-index must be rendered in between the cell background and the cell foreground. This allows text to be written on top of images.
• Deleteing displayed images
• Scrolling the images with the text
• Implementing checks to prevent DoS attacks by loading too many images
• Handling window resizes/font size chages
• Handling reset (I plan to just have the terminal emulator delete all images on reset)
• Handling the alternate screen (The alternate screens images will be cleared when switching to it, but the main screen must preserve the images when switching to/away from it).

[Firenox89]

I really love this idea.

So to put this into my own words:
A program asks kitty about the height and width of a character,
then it chops the image to display, into fitting pieces and sends them piece by piece to kitty,
finally kitty starts to draw this pixel-characters at the current cursor position.

Assuming I got it right, the first issue that comes to my mind would be the drawing at cursor position, some applications like ranger show some image preview like this.
So setting a position to render to could be useful.
But I have no clue about how this ncurse shell rendering works so maybe its not an issue.

Next thought:
Define a header after the escape code which is extensible something like this:
<ESC>r x=100,y=100,format=ARGB\n

My main point here is to use named arguments so its easy to add new arguments while older versions still work because they ignore unknown args.

Next:
Wouldn't it be possible to render whole images?
I guess it would not integrate into your char grid as nicely, but on the other hand simplify the creation of pixel data that could be displayed.
Also drawing pixelchar by pixelchar could lead to funny effects when the drawing or data producing is too slow, especially for videos or gifs.

[kovidgoyal]

Yes, there will be the possibility to specify formats for image data, probably something like:
f=1 or f=8 or f=24 or f=32 for (1-bit, 8-bit, 24bit RGB and 32bit RGBA) Other formats such as compressed image data can also be added later.

And yes you can specify whole images at a time, they will simply span multiple cells, something like this:

<ESC code>f=24,x=2,y=3,w=500,h=300;image data

Then the terminal will render the 500x300 pixels image starting at position (2, 3) from the top of the current cell.

[kovidgoyal]

I have created a spec for the protocol here: https://github.com/kovidgoyal/kitty/blob/master/protocol-extensions.asciidoc#graphics-rendering

[BwackNinja]

What were the issues/limitations you saw with sixel? Implementing that protocol would allow you to take advantage of projects that already support sixel graphics (including a replacement for w3mimgdisplay, allowing images to work in w3m and ranger) rather than adding support in those applications afterwards.

Would you consider adding sixel support in addition to this new graphics rendering protocol?

[kovidgoyal]

IIRC (and please correct me if I am wrong)

1. sixel supports only 256 colors
2. it has no way to optimize transfers in the case when the client program and the terminal emulator are running on the same computer and have access to a common filesystem/shared memory
3. This may just be my fault, but, I could not find a clear description of how it works when interspered with text -- so, how does the client know what size (number of pixels) of images to send and how does it control their placement? What happens in a full screen application that wants to display an image and then overwrite it with text and so on.
4. There is no facility for persistence. Think of the case of displaying an animated GIF, for example. With the protocol I describe above the client has to send each frame to the terminal emulator only once. With sixels it has to be done repeatedly.
5. A minor issue: The sixel encoding is IMO unnecessarily complex.

[kovidgoyal]

As for adding sixel support -- I have nothing against it, but, practically speaking, since I dont have any need for it either, it will not get done unless someone else contributes. However, if and when I implement some variation of this protcol in kitty, adding support for additional legacy schemes, should be fairly easy to do on top of that. Basically all that has to be done is implement the parsing for the escape codes and then map the result onto this scheme (which is a strict superset of all the legacy schemes).

[BwackNinja]

1. Sixel only supports a 256 color palette, but it can be cycled. 24-bit color is technically possible, but the point still stands though, because libsixel itself only supports up to 15-bit color when encoding, though there's an open issue for 24-bit color support.
2. Fair point, though I'd expect performance to be a lesser concern because you'd probably stop displaying in a terminal after a certain point.
3. I'd expect that it acts like normal characters do, using escape sequences to reposition for animations. Text and graphics probably cannot overlap. I'm reasonably confident that's how it works, but I haven't read a real description myself.
4. While interesting, I would expect such a feature at the toolkit level, and reuse would be something only gifs would use because the buffer would need to be very large otherwise.
5. Probably true.

As an aside, the palette format and low color support of sixels might be a good thing dealing remotely if all someone wants is a preview of a file. A 3.1MB 2560x1600 jpeg encoded as a 15-bit sixel image can be around 11 MB, while with 256 colors only it is 4.5M, and base64 encoded it is 4.1MB. After a certain point, it may be better to get the original image from the source and then display it because of how much data you'd be dealing with.

[kovidgoyal]

2. My motivation for implementing graphics support at all is to port https://github.com/kovidgoyal/iv to run in a terminal. This can display full screens worth of image thumbnails, that can be animated. Performance, especially over a network is a definite concern.

3. How does the client know:

• How many pixels to send to cover an mxn block of character cells. i.e. how many pixels is a single character cell?
• What happens if the client sends more or less pixels than the number of cells available?
• Is there a specification for how the cursor moves in case of overflow/underflow of pixels?

4. I dont see how this can be implemented in a toolkit. In the general case, when running over a network, there is no way for the client program to implement persistence server side without co-operation from the terminal emulator. Also, it is not only for GIFs. Another application is sprite maps. If an application wants to use special characters/symbols in its rendering it can generate a sprite map of them and send it to the terminal emulator once, and then simply refer to them in the future, instead of resending the individual spite data at each occurrence.

2 and 4 are designed precisely to workaround the fact that images are very large as raw data. And it is trivial to add an indexed, 8-bit format to the new protocol if desired.

[BwackNinja]

2. Fair enough.
3. ranger uses termios to get the size of a cell. With sixel, anything going past the horizontal limits is cut off. Everything past the vertical limit scrolls, just like with text.
4. Having fully read your specification, the display of images is still completely controlled by the application and not handed off to the terminal in any way, so my concern here was unfounded.

Extending that last point, if the goal is remote image display, what you've got is probably the right way to do it. Locally though, w3m on X (when you're not using the sixel support) is drawing directly onto the terminal window, not having images decoded and drawn by the terminal at all. That route certainly has better performance, but is limited to applications running locally and is not display server agnostic.

Thanks for the discussion. I had only briefly looked at the sixel protocol before and now I know more about it and why you've determined it is not an ideal solution.

[kovidgoyal]

Interesting, I did not realize one could get cell width height and not just number of cells per row/number of cells per column using termios. At least according to http://man7.org/linux/man-pages/man4/tty_ioctl.4.html the ws_xpixel and ws_ypixel values are unused in the linux console. I assume they are actually made use of in some X terminals. I should probably implement that in kitty. And once implemented remove the extra escape code to get cell size from this protocol, since it is not needed. Thanks, good to know.

One some investigation -- libvte based terminals and konsole both return 0 for those values. xterm returns the window size.

[jwhear]

Is the the spec implemented in master yet? I just tried it with no effect, trying to determine if user error.

[kovidgoyal]

No not yet, I will close this bug when it is implemented. I'm rather busy with other things at the moment.