Kitty Image Protocol Support

[wez]

This issue is tracking the status of supporting Kitty's image protocol.

Spec at: https://sw.kovidgoyal.net/kitty/graphics-protocol/

• termwiz: parse APC escapes
• termwiz: parse baseline Kitty Image protocol APC escapes
• term: basic placement implementation on top of existing iTerm2/sixel image data model
• term: add placement data model to support full range of kitty placement options
• wezterm-gui: teach renderer to allocate quads for new placement model
• termwiz: parse animation/composition kitty image protocol sequences
• term: apply animation to model
• handle animation control
• termwiz: support data transmission via shared memory on unix
• termwiz: support data transmission via shared memory on Windows
• placeholder support, which might enable support to function in tmux

Using it:

Enable the protocol by setting enable_kitty_graphics=true in your config.

Known conformance issues

• The spec models image placements independently from the terminal cells, but wezterm maps them to terminal cells at placement time. As a result, over-writing cells with text can poke holes in wezterm which may not appear in kitty.

[kovidgoyal]

I am happy to see this, please don't hesitate to ping me if you have any questions.

[wez]

I am happy to see this, please don't hesitate to ping me if you have any questions.

Great to hear! Do you have a set of tests or similar that could be made to run against wezterm to sanity check conformance?
The surface area of the spec is quite large and I'm sure I'm going to overlook something. My game plan is try running @dankamongmen's notcurses demo when there's enough of an implementation working and see if anything looks egregiously bad.

[kovidgoyal]

I do have unit tests in kitty itself, for individual bits of functionality from the protocol, but I doubt they can be easily lifted for another terminal emulator. If you wish to have a look, please see kitty_tests/graphics.py

I would actually be happy to collaborate to make those (and other) tests defined in a terminal independent fashion so anyone implementing the protocol could use them. Perhaps a simple txt or json based format that specifies the input as escape codes and the output as a set of image placement data or similar.

[dankamongmen]

I am happy to see this, please don't hesitate to ping me if you have any questions.

Great to hear! Do you have a set of tests or similar that could be made to run against wezterm to sanity check conformance?
The surface area of the spec is quite large and I'm sure I'm going to overlook something. My game plan is try running @dankamongmen's notcurses demo when there's enough of an implementation working and see if anything looks egregiously bad.

first off, i enthusiastically support this move. the kitty protocol is (at least for my purposes) a tremendous improvement upon both sixel and iterm. it simplifies things (no more requirement that one draw in terms of 6 rows, sane deletion behavior), improves performance in several areas (fast moves, fast changes to drawn images), and makes certain things possible that otherwise are not (text drawn atop bitmaps without killing entire cells). relative to iterm2, it's much more flexible and powerful. at the same time, it currently shows worse local performance in Notcurses due to (a) greater bandwidth demands than sixel and (b) time spent in zlib. i expect to be able to hide the latter in most applications via threading. it would be possible to reduce the local bandwidth via using the filesystem as a side channel to load images, but i have not yet embraced this, and might never do so (but might, who knows).

right now Notcurses selects the Kitty protocol strictly based on heuristics, as opposed to doing the recommended query. I intend to add support for the latter (and really ought have by now), but hadn't bothered since no one else had implemented it (and also because said query frustratingly doesn't let you determine the version of the kitty protocol supported--as you note, it's a large protocol). you can ensure kitty graphics are being used by running notcurses-info:

where it says "rgba pixel animation support", you want that. wezterm currently says "sixel graphics" or "iterm graphics", i forget which one. if it says "rgba ....", you're driving kitty graphics. i'll try to add this query soon.

Notcurses uses a pretty wide subset of the protocol, and indeed motivated/proposed some of it. Among the elements it exercises are:

• image loading via 32-bit RGBA without display
• display of loaded images
• movement of loaded images via position commands
• deletion of loaded images specified by id
• deletion of all images present
• scrolling of images
• inhibition of responses via q=2
• non-scrolling of images via C=1
• images in both the regular screen and alternative screen
• fast cell-sized wipes via the animation protocol + RGBA load
• fast cell-sized restores via the animation protocol + reflection (composition)
• display of images larger than the terminal width (kitty cuts off excess material on the right), but only due to an existing bug

i do not currently exercise: sideloading images via the filesystem, managed animations, scaling, or z-indices other than 0, though i expect to start using the last Really Soon Now.

notcurses-tester and ncplayer -bpixel will also effectively test portions of your implementation.

if you have any questions, don't hesitate to ask me; i reckon i know more about the kitty graphics protocol than anyone living save @kovidgoyal himself. happy hacking!

[dankamongmen]

if you have any questions, don't hesitate to ask me; i reckon i know more about the kitty graphics protocol than anyone living save @kovidgoyal himself. happy hacking!

oh and let me add that i make use of kitty's honoring of transparency values in RGBA (at at least a bimodal level), just as I do in iterm, and using unspecified pixels in sixel when P2=1 in sixel.

[kovidgoyal]

@dankamongmen regarding using heuristics for detection, because you dont have a version. You shouldnt need a version. You can create a dummy image and try to perform every operation you want on it and see if it succeeds, i.e. the terminal does not respond with an error. Thus you can know exactly what the terminal you are running in supports.

And of course as am sure you already know, if you really want a version us XTVERSION or XTGETTCAP

[dankamongmen]

And of course as am sure you already know, if you really want a version us XTVERSION or XTGETTCAP

this is exactly what i'm doing, see https://github.com/dankamongmen/notcurses/blob/master/src/lib/termdesc.c#L501-L527

@dankamongmen regarding using heuristics for detection, because you dont have a version. You shouldnt need a version. You can create a dummy image and try to perform every operation you want on it and see if it succeeds, i.e. the terminal does not respond with an error. Thus you can know exactly what the terminal you are running in supports.

yep, i could do that. what i've got now works for me pretty well, though.

and let it be known in all lands the sun touches: i wholeheartedly affirm the use of the Kitty graphics protocol over others. indeed, i sat down a few weeks ago to describe my ideal terminal graphics protocol, and ended up with something very similar to Kitty's.

i will be adding support for coarse kitty graphics support very shortly, probably tonight, in dankamongmen/notcurses#1998.