Kitty's graphic protocol support

[nscotto]

• nvim --version: 0.4.4
• Operating system/version: fedora 32
• Terminal name/version: kitty 0.18.3
• $TERM: xterm-kitty

Kitty terminal created a protocol for graphic support using escape characters, we could display images in neovim by supporting the protocol.
I would love to see this feature implemented as it would enable full notebook (R notebook, jupyter notebook, also markdown) experience in neovim as well as many other applications.
While Kitty is (as far as I know) the only terminal to support graphics at the moment, their protocol is designed to integrate well with current terminal architectures.

[theHamsta]

I would love to have protocol in Neovim that pretty much tells GUIs: you can render this region as image/html/markdown instead of the alt-text that's actually there.

[nscotto]

Yes, this would be really neat, making neovim stand up against modern IDEs on every level!
GUIs just need to support this protocol if neovim supports it.
This protocol is a neat way of supporting graphics for terminal based applications running in a GUI, like terminal emulator or a GUI wrapper around neovim for example.

[nscotto]

I am looking at the source code trying to understand where this would fit, can anyone point at me the files/function to look at?

[KillTheMule]

Guess that's a question for @bfredl or @justinmk

[bfredl]

maybe we want some kind of concept for representing images in GUI:s in general. then a kitty-protocol implementation in TUI would just follow, as TUI is just another GUI already.

[nscotto]

Not too sure about the extent of what you mean by concept.
As a first draft of what could be done:

• we could have commands for inserting images under cursor
• internally the images would be represented using kitty protocol, i.e. using the graphic escape code, GUIs would implement the protocol in order to render the images.
• maybe having an option for enabling the feature

[theHamsta]

@nscotto I guess what bfredl is trying to say is that implementing special escape sequences shouldn't live in the main TUI client and in this repo. There could be kitty-client in a separate repo that could display images. For that it must be able to ask nvim where there are actually images (and meta data like size etc.). This could implemented by nvim lua module that places extmarks at ranges where images can be displayed. The kitty client can just ask the lua module for the images and then output the escape codes. This way there would be virtually no code in the main repo and each client can implement the rendering of the meta data their way.

[clason]

Just wanted to note that there's already image support in Markdown; some LSP servers use that to get VS Code to show hover previews with images, e.g., as

![preview](data:image/png;base64, .... )

I second the suggestion of @theHamsta to start with extending the floating window protocol to include "rich" content so you can indicate that the buffer is in fact Markdown or HTML that should be interpreted rather than shown literally (which a GUI can then do, while the TUI may instead strip the markup and do its best with conceal, like the current LSP hover/signature help code). The next step would then be "extended language injection" to allow applying this to specific regions of a buffer.

[bfredl]

We don't want to hard code the kitty sequences in the versioned ui protocol (:help ui.txt) itself, but it is not completely out of the picture to have tui/tui.c support if it is not far too complex (also: neovim 0.6 will decouple "core" from "tui" more formally).

[pta2002]

I'd love to have something like this in neovim, but I am having a hard time imagining what an API for this would look like - you'd definitely want to have a way for programatically drawing images, not just a file on a disk. Maybe an API where you could specify a framebuffer to draw at a specific position on the buffer? And then different UIs would implement it differently - the TUI could implement it if it detects a compatible terminal, for example.

[pta2002]

Another alternative to kitty's protocol, which I think only kitty supports, is sixel which IIRC was recently implemented by vte-ng, which gnome-terminal and termite use.

Of course, the best option here would really be to just have a way for plugins to implement the details of the graphics protocol themselves, while neovim just forwards a buffer with raw pixel data.

[bfredl]

I tried to understand sixel from the available documentation but I think I failed

[nscotto]

It makes sense. In this discussion kitty's author explain why he wrote a new protocol, worth checking out.
So yes it is definitely good to be able to support different protocol using another layer of abstraction.

[christianparpart]

@bfredl The problem with Sixel sadly is, that the amount of grid cells involved depends on the font metrics and size being used.
Kitty's (and iTerm2's) image protocol are a little better in that regard, they both support specifying the amount of rows/cols to use for rendering. Caveats are (to my research) that it is also the by far most complex image protocol available.

Before going to "just implement" one image protocol, I think we've to admit there is some fragmentation and it would make sense to whatever image protocol is going to be supported by (Neo)?vim it should probably made in a more abstract way to support not just one image protocol (NB: there's great work from @csdvrx to display pixmaps via a unicode rasterizer, a great way to work around non-supporting terminals).

There is (or was) quite some heat discussions on a future image protocol in the terminal-wg forum however, that I think is worth keeping an eye on: 12 and 16.

p.s.: I'd love to see that happening in a somewhat terminal-independant manner at some point so that when I'm editing latex, my latex plugin can render math formulas in little popup windows :D

[joelostblom]

In addition to kitty and libsixel, there is also imgcat in iTerm2 and tycat in terminology.

Without understanding how involved these different approaches would be to implement in Neovim, from a user's perspective I think it seems reasonable to focus on supporting just one solution to start out, so that people who are really keen on viewing images could use that setup. kitty seems like a good choice based on that it is available on both Linux and MacOs. Maybe there could also be a fallback mode using half blocks similar to the image viewer viu?

I tried displaying images using kitty's icat in the floatterm plugin, but ran into issue #8259. Viu works with floatterm, but only in fallback mode.

[christianparpart]

In addition to kitty and libsixel, there is also imgcat in iTerm2 and tycat in terminology.

Without understanding how involved these different approaches would be to implement in Neovim, from a user's perspective I think it seems reasonable to focus on supporting just one solution to start out, so that people who are really keen on viewing images could use that setup. kitty seems like a good choice based on that it is available on both Linux and MacOs. Maybe there could also be a fallback mode using half blocks similar to the image viewer viu?

I tried displaying images using kitty's icat in the floatterm plugin, but ran into issue #8259. Viu works with floatterm, but only in fallback mode.

I am sorry, that would be a really hard choice - trying to be objective here... But the kitty image protocol is implemented by only one terminal emulator and was already said to be easy to complex to be adopted by the general public. It is not a bad protocol, just to have that said, but only implementing half of it would make it so.

I am not a fan of either, but (again, objectively speaking) i think Sixel is, despite all all odds of complains, the one implemented by most TEs.

[elianiva]

that do not depend on the conceal feature.

we can do that already by using virtual text which can be placed anywhere on the buffer.

[clason]

https://github.com/edluffy/hologram.nvim

[traverseda]

Konsole in kde also supports the kitty image protocol. When I try to use the same command inside of a neovim term:// it says Error: Terminal does not support reporting screen sizes in pixels, use a terminal such as kitty, WezTerm, Konsole, etc. that does.

[cathaysia]

• https://github.com/dankamongmen/notcurses
• https://github.com/romgrk/kui.nvim
• https://github.com/romgrk/kirby.nvim

[christianparpart]

https://github.com/dankamongmen/notcurses

@cathaysia Although, one forgot to mention how amazed dankamongmen is about the blazingly fast performance of the Kitty image protocol. From an adopters point of view, that must be considered as well. ;-)