Clarification about the graphics protocol

[antirez]

Hi! And thank you for creating Kitty and popularizing the Kitty Graphics Protocol. For us people living in the terminals is an incredibly powerful augmentation of our experience :)

I'm writing seeking for help. I'm writing a program (a Commodore 64 emulator) that should work both in Kitty and Ghostty. To do so, I need to update the same image (placement) on the terminal, so that every time I have a new frame buffer from the emulator, I can replace the image.

I'm able to do this easily in Ghostty referencing the same image id and using a=t in order to re-transfer the data: the image updates. I'm also able to do it in Kitty using the animation protocol. Like that:

• First frame: printf("\033_Ga=T,i=%lu,f=24,s=%d,v=%d,q=2,c=30,r=10,m=%d;" ...) followed by printf("\033_Gm=%d;", more_chunks);

• Next frames: printf("\033_Ga=f,r=1,i=%lu,f=24,x=0,y=0,s=%d,v=%d,m=%d;", ...) followed by printf("\033_Ga=f,r=1,m=%d;", more_chunks); and at the end of the chunks: printf("\033_Ga=a,c=1,i=%lu;", kitty_id);.

However apparently Ghostty does not support the animations, nor Kitty believes it is valid protocol to re-send the image data with a=t of an already displayed image, so I can't find a way to make both work. I wonder if you have any hint, and I hope that the two implementations will try to work in a more similar way in the future :) I believe that in the original specification having a=t in an already displayed image is unspecified behavior, so perhaps it could be used as well in Kitty as an update opreation not involving the animation? This way there could be a simpler way to handle animations that is compatible in both the implementations (and perhaps also the other implementations, like Konsole and so forth). Thanks!

[antirez]

Pinging @mitchellh as well that may be interested in this discussion.

[mitchellh]

I'll note that supporting animation is an accepted and open issue on Ghostty: ghostty-org/ghostty#5255 So that's an outstanding issue. But as I noted in that issue, I don't think there is any other terminal other than Kitty at the moment that supports that part of the spec.

I didn't realize that sending updates via a=t of a displayed image isn't technically valid. I'm on mobile now but I'm curious if that's specified or just unspecified behavior. It seems like an easy win for all terminals attempting to implement the graphics protocol would be to extend the spec (and Kitty) to allow this, since many more terminals implement basic image transfer and display.

But, overall, this is Kitty's spec. I'll defer to Kovid what he wants to do and I'm happy to follow suit.

[kovidgoyal]

Re-transmission of image data with the same id is supposed to delete the
existing image placements and replace its data with the new data,
although I cant recall if I specified this explicitly in
the actual spec or not. To understand why this is necessary, think of the use
case of two unrelated programs drawing images to the same terminal
instance one after the other. If a=t as update was allowed, then you
would get different effects depending on whether they happened to re-use
the same image id or not. In general I try to avoid such persistent
global state wherever possible in my protocol designs.

However, I don't quite understand your use case. If what you
want to do is resend the full image data anyway, why not just follow the
protocol's design and use a=T? It means you do have to re-specify the
position of the image, but is that really an issue for you? This should
work on all implementations since its pretty basic. If its atomic
updates you are worried about you can use two image ids transmit your
second frame using a=t and then synchronously delete the first image and
place the second by using the synchronized update escape code.

The animation protocol isn't really necessary for this. It really shines
if you want to do partial updates of the image thereby avoiding
re-transmitting every frame. Or, of course, if you want to do animations
simply offloading the actual animation logic to the terminal.

[kovidgoyal]

Oh and note that while I have closed the issue you are welcome to
continue discussing. It may be worth adding some explicit language to
the spec clarifying the behavior of a=t re-transmission.

[kovidgoyal]

Added a note to the spec clarifying the behavior for re-transmission.
a023a0d

[antirez]

Thank you both for your reply. I understand the motivation for a=t not updating, but then perhaps we need a a=u?
The reason why it is not viable to use a=T again, is that it both in Ghostty and Kitty would recreate a new placement, unless I would use C=2 (if I remember the option correctly) in order to don't move the cursor, but then, what's the point of having a terminal that can display image and text at the same time, if I can't also write to the terminal if needed. This is for instance what my emulator will do after loading a program successfully in the Commodore 64 memory:

It will tell you that the program was successfully loaded, and you can then hit "RUN" to execute it. Or I may output warnings, and so forth. But I want to still update the same placement. Right now I can do it only using the animation protocol.

Btw, in general, design wise, it looks like that the only way a=T allows you to update an image without resorting to the animation protocol is by controlling the cursor position, which is somewhat mixing two different concepts at a logical level: where the image is placed, and how the image is identified (its id). It looks odd that only the latter is not enough to update the image, regardless of its placement. Moreover, an old C programmer like mine, by recreating images with a=T not moving the cursor, will always have the feeling that new images objects are created and not freed, since semantically it is to "stick upon". So my feeling is that something is needed.

There is also another problem here: since Ghostty is getting very widespread adoption in a short timeframe, I have the feeling that more programmers may do my same mistake of using a=t to update an existing image, and this will create hard to fix incompatibilities in the future.

For now the only way I have to fix my program is, I believe, to have an expliciy --ghostty or --kitty switch to switch which protocol to use. Thank you for the useful discussion!

[kovidgoyal]

I dont follow, do the following:

1. Use a=t with a second id to transmit data for second frame
2. save cursor position (by this I mean use the escape code to have the terminal save the cursor position)
3. Move cursor to where you need the placement
4. Create new placement
5. delete previous image
6. restore cursor position
7. continue to output whatever text you want to output

It does not disrupt your ability to output text in any way.

[antirez]

This works but it is, in my opinion, too complicated (and requires bi-directional communication), to the point that it is simpler to use the animation protocol like I'm doing right now: I can handle it only writing to the terminal, and since my terminal is in raw mode, to capture the keystrokes to send the emulator, I don't want to mix querying the terminal. I believe there is no middle ground between just showing a static image and the more complicated animation protocol. My feeling is that, by mistake, Ghostty provided this middle ground option (but, indeed, I understand your design for a=t, so a different action would be needed). This plus the fact that Ghostty and I believe all the other Kitty graphics protocol implementation also handle large images without requiring 4k chunking changes significantly the work needed in order to reach a given effect.

In this use case, with Ghostty I need to:

1. Send the first frame with a=T.
2. Send the next frames with a=t using the same exact code.
3. Send the bulk image base64-encoded with a single write.

Let's compare this with Kitty:

1. Save the cursor position, I need to implement the right escapes, most people don't know what VT100 is at all.
2. Send the first frame with a=T.
3. Chunk the base64 encoded output to 4k. I need to understand what to specify in the next chunks, handle the more flag, ...
4. If I produce output, restore the cursor position. And I'm not sure this works in case of scrolling.

So I believe that there are rooms for simplification. Anyway, thanks for the discussion, while I believe the above, I believe more that it is important to avoid stressing too much open source maintainers! So I'll implement the switch, or maybe I'll even try to implement auto detection of the terminal I'm using. Thanks.

[kovidgoyal]

Eh, I dont see how your comparison works. There is no bi directional communication required.
Chunking is completely orthogonal to this discussion and has no bearing when comparing the two approaches, it needs to be done regardless of which frame you are transmitting. You use the exact same code for sending the first and all subsequent frames in both cases. Indeed if you dont care about synchronous updates, then the function is utterly trivial.

1. Save cursor position (DECSC write the bytes ESC and 7, thats all)
2. move cursor and transmit image with a=T and fixed id and C=1
3. restore cursor position (DECRC write the bytes ESC and 8, thats all)

this function can be used identically for every frame. It is likely to work on all implementations of the kitty protocol (there is more than ghostty to consider FYI) and has exactly the semantics you want. Exact same code used for every frame, does not require bi-directional communication, updates the image in place, does not interfere in any way with outputting text. And yes it works robustly there is no scrolling involved.

The only "complication" is you need to know the escape codes to save and restore cursor position. If you are low level enough to be writing escape codes for the graphics protocol then I dont see how that's an undue burden. If you find it an undue burden, then I recommend using one of the many libraries that wrap the protocol in higher level APIs. And let me emphasize here, it is not you saving the cursor position, you are asking the terminal to save and restore it for you, hence no bi-directional communication.

You have just been led up the wrong tree by a bug in ghostty (or more charitably a lack of clarity in the spec, now fixed).

And on the subject of chunking, it can be implemented in about 5 lines of code, as demonstrated by the minimal examples on the graphics protocol spec page.

[antirez]

Fair points @kovidgoyal thank you so much for your work on Kitty! And have a great day. I'll try to find the best compromise :)

[kovidgoyal]

You are welcome! All the best on your emulator.