Alternative approach to image placement that works with tmux (proof of concept) #4021
Replies: 21 comments 29 replies
-
|
this might work in vim, but I dont see how it would work in tmux. The image placement would need to be modified by the multiplexer when it is presenting more than one window on the screen. And for vim there is already https://github.com/edluffy/hologram.nvim |
Beta Was this translation helpful? Give feedback.
-
|
Never mind I misread the proposal, you want to still use the normal graphics protocol but specify its position using private use characters. |
Beta Was this translation helpful? Give feedback.
-
|
I think it can be made a lot simpler and avoid need for colors by simply extending the graphics protocol a bit. Create a new "char based placement mode". In this, you transmit the image as normal without placing it. Then instead of sending a placement command, you instead move the cursor to the top left cell of the image, send a "start char placement" command that causes the terminal to record the current cursor position. Then you move the cursor to the bottom right cell of the image and send an "end char placement" command. The terminal will then create a placement based on those two cursor positions No need for PUA chars. |
Beta Was this translation helpful? Give feedback.
-
|
And actually thinking some more about it I'm not sure why you need to do anything at all to support tmux other than wrapping the image escape codes in tmux passthrough. The image placement position is already based on the current cursor position. So move the cursor to where you want the image, and transmit the image escape codes wrapped to pass through tmux. All you need to do is size the image to fit which since nowadays tmux supports TIOCGWSZ properly is easy to do. This will get you basic image display, though of course anything like changing the window layout in tmux or resizing the terminal or scrolling will gum things up royally. |
Beta Was this translation helpful? Give feedback.
-
Yes, the whole point of the character-based approach is to make it work with scrolling, tab switching and layout change. |
Beta Was this translation helpful? Give feedback.
-
|
On Mon, Sep 13, 2021 at 06:27:43PM -0700, Sergei Grechanik wrote:
> This will get you basic image display, though of course anything like changing the window layout in tmux or resizing the terminal or scrolling will gum things up royally.
Yes, the whole point of the character-based approach is to make it work with scrolling, tab switching and layout change.
Ah, you want the terminal to reposition the image whenever it sees
the characters. That will impose a perf penalty since the terminal
will need another branch when processing chars.
I think the scheme can still be significantly simplified by using
just three characters: start, middle and end. The image id and placement
id can be encoded in the three color fields (foreground, background and
underline). Thats a total of 72bits to encode 64bits worth of ids.
The three characters can be three fixed PUA chars.
The top, left corner of the image will be the position of the start
char, the bottom the position of the end char and the right the widest
position of the middle char. This way there is no need for ranges of PUA
characters to be used. There might be issues with using so many colors
in vim, though. And there are likely to be implementation difficulties
when a image moves partly into the scrollback.
|
Beta Was this translation helpful? Give feedback.
-
|
I guess to handle the partial scroll case one needs row numbers. So maybe limit ourselves to 16 bits for placement ids and then we will have 16bits for row numbers, which should be enough. And of course we would need to combine the start and middle PUA into just one char. Since we cannot rely on the start char being present. |
Beta Was this translation helpful? Give feedback.
-
|
And actually, one could even use diacritics from say https://unicode-table.com/en/blocks/combining-diacritical-marks/ to encode more information. There are 112 in that block but a total of 2300 in the entire unicode standard (see is_combining_char() in the kitty source code). So row number and column number can be specified for any char using two diacritical characters limiting us to images of max size 2300x2300 cells which should be enough for practical purposes. So the scheme would be: image id and placement id encoded as colors, row and column number encoded as combining chars. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I think using diacritics is a nice idea, but it needs some experimentation to see how well it works in practice. Regarding colors, I would avoid using background color for IDs because it is useful for transparency. Underline color may be useful for 256-color mode, since 8 bits of foreground color for image id is not great, but 16 bits should be ok for many applications. In 24-bit mode, personally, I think 24 bits should be enough for image ids. |
Beta Was this translation helpful? Give feedback.
-
|
Yes I agree background color could be useful, so I suggest of the available 48bits use 32 for image id and 16 for placement id. That way clients can use randomized 32 bit ids avoiding the need for a roundtrip. |
Beta Was this translation helpful? Give feedback.
-
|
It would be great with some flexibility in what you can use here, to support different applications that has different limitations in what they support. I would love to be able to use this to have images in WeeChat, but it only supports 256 colors, and it only supports foreground and background colors, not underline colors (it supports bold, reverse video, italic and underline, so you could get 4 bits from that). |
Beta Was this translation helpful? Give feedback.
-
|
On Mon, Sep 20, 2021 at 04:03:57AM -0700, Trygve Aaberge wrote:
It would be great with some flexibility in what you can use here, to support different applications that has different limitations in what they support. I would love to be able to use this to have images in WeeChat, but it only supports 256 colors, and it only supports foreground and background colors, not underline colors (it supports bold, reverse video, italic and underline, so you could get 4 bits from that).
It would just be restricted to using 255 ids in that case, and no
placement ids (aka one placement per image). Which runs
the risk of colliding if used inside a multiplexer but otherwise should mostly work.
Bold and italic can be used but reverse and underline cannot as they
would cause rendering artifacts for images with transparent regions or
those that dont cover all cells fully. Adding 2bits extra doesnt really
seem worth it.
|
Beta Was this translation helpful? Give feedback.
-
|
That's great, though a bit limiting. I'm not sure I understood the part about background color vs. transparency. Why can't the background color be used? Why would reverse and underline cause rendering artifacts? When it's a character that's used for image placement, wouldn't kitty just not render the reverse and underline? |
Beta Was this translation helpful? Give feedback.
-
|
On Mon, Sep 20, 2021 at 06:14:19AM -0700, Trygve Aaberge wrote:
I'm not sure I understood the part about background color vs. transparency. Why can't the background color be used?
If your image has transparent portions then the background color of the
cell will show through.
Why would reverse and underline cause rendering artifacts? When it's a character that's used for image placement, wouldn't kitty just not render the reverse and underline?
Reverse causes the background to be rendered with the foreground. And
underlines are rendered regardless of the character present (you can
underline spaces as well).
|
Beta Was this translation helpful? Give feedback.
-
|
Ah, okay. I was thinking kitty would avoid rendering the character and attributes if it's one of these special characters. If it is going to render it, and just place the image on top, won't the character itself be visible if the font has the character and the image is transparent at that position? |
Beta Was this translation helpful? Give feedback.
-
|
On Mon, Sep 20, 2021 at 07:44:05AM -0700, Trygve Aaberge wrote:
Ah, okay. I was thinking kitty would avoid rendering the character and attributes if it's one of these special characters. If it is going to render it, and just place the image on top, won't the character itself be visible if the font has the character and the image is transparent at that position?
the character will be special cased to render blank as that happens on
the CPU. Underline rendering happens on the GPU which has no knowledge
of the character in the cell it is rendering.
|
Beta Was this translation helpful? Give feedback.
-
|
I see. Thanks for the explanations :) |
Beta Was this translation helpful? Give feedback.
-
|
I experimented with specifying rows and columns with diacritics, and it worked fine. So what do you think about adding this functionality to kitty? |
Beta Was this translation helpful? Give feedback.
-
|
On Fri, Oct 01, 2021 at 09:13:48PM -0700, Sergei Grechanik wrote:
I experimented with specifying rows and columns with diacritics, and it worked fine. So what do you think about adding this functionality to kitty?
In principle, yes.
|
Beta Was this translation helpful? Give feedback.
-
|
On second thought, using combining character might not be such a great idea, because unicode is crazy. Like, today I learned that in the combining diacritical marks block there is a character that is not a combining character: https://en.wikipedia.org/wiki/Combining_Grapheme_Joiner. Since it's zero-width, it can still be used for row/column numbers, but I'm not sure if it should. Also, there are reserved characters that may become combining characters in the future, shifting the character->row/column mapping. These problems can be solved by specifying the exact character->row/column mapping in the protocol and maintaining backward compatibility, but still I'm starting to doubt. |
Beta Was this translation helpful? Give feedback.
-
|
On Sun, Oct 10, 2021 at 10:00:14PM -0700, Sergei Grechanik wrote:
On second thought, using combining character might not be such a great idea, because unicode is crazy. Like, today I learned that in the combining diacritical marks block there is a character that is not a combining character: https://en.wikipedia.org/wiki/Combining_Grapheme_Joiner. Since it's zero-width, it can still be used for row/column numbers, but I'm not sure if it should. Also, there are reserved characters that may become combining characters in the future, shifting the character->row/column mapping. These problems can be solved by specifying the exact character->row/column mapping in the protocol and maintaining backward compatibility, but still I'm starting to doubt.
The way it would work is you specify the alphabet in the spec and fix it
for all time. Pick a set of zero width/combining characters that work
with all applications of interest and use that as the alphabet. It
definitely shouldn't be "all combining characters in the spec" as that
is very poorly specified. I would actually be fine with the following
scheme:
1) An alphabet of 256 carefully selected combining chars that are tested
to be zero width in vim and tmux and weechat and whatever else
2) The row and column numbers will be made of one or more of these
alphabets. So basically it will be a base 256 encoding.
3) One extra combining char such as the zero non-joiner or whatever that
acts as a separator between the row and column numbers.
So one just needs to find 257 safe combining chars. Or if that's too
hard, 129 or even 65.
|
Beta Was this translation helpful? Give feedback.
-
|
Ok, that should work. I would suggest using combining characters of the same combining class, like 230 (the class of marks that go above the character, there are about 500 of them). Combining characters of the same class shouldn't be reordered during normalization, removing the need for a row/column separator. |
Beta Was this translation helpful? Give feedback.
-
|
Yes that is a good idea. |
Beta Was this translation helpful? Give feedback.
-
|
Although normalization is annoying and we should stick to combining marks that dont get merged into their base character during NFKC processing. |
Beta Was this translation helpful? Give feedback.
-
|
So, I reimplemented this thing using diacritics for row/column numbers and foreground color for image ids: kitty with unicode placeholders. Image uploading is performed as usual. To enable unicode placeholders for an image, a virtual placement should be created by specifying the flag U=1: Currently only one placement is supported. And I also implemented a utility to upload and display images using this method: tupimage And I also forked st and implemented a subset of the kitty graphics protocol (only unicode placeholders can be used for placement), mostly for fun (and testing): st with graphics. |
Beta Was this translation helpful? Give feedback.
-
|
Nice, why only one placement? And for tmux do the graphics escape codes have to be wrapped in tmux passthrough? |
Beta Was this translation helpful? Give feedback.
-
|
It's not essential for basic usage, so I was lazy to implement it, but encoding placement id in the underline color should be very straightforward. |
Beta Was this translation helpful? Give feedback.
-
|
On Sat, Jul 16, 2022 at 09:24:29PM -0700, Sergei Grechanik wrote:
It's not essential for basic usage, so I was lazy to implement it, but encoding placement id in the underline color should be very straightforward.
Ah OK I just wanted to be sure there is no fundamental limitation.
Yes, all uploading and placement creation still needs to be wrapped in tmux passthrough sequences. It's not a big deal in my opinion (although in latest tmux versions passthrough sequences have to be explicitly enabled in config, which is slightly annoying).
You can just have the image display program run
tmux set -p allow-passthrough on
See the ssh kitten for an example.
The real problem with tmux is when the uploading pane is inactive, and tmux doesn't allow pass-through from inactive panes (and even if it did, the response from the terminal wouldn't be returned to the correct pane). I've implemented some workarounds, but they are quite awkward.
One could use quiet mode where there are no responses from the terminal,
however, if tmux doesnt even allow passthrough, then I guess its
hopeless.
|
Beta Was this translation helpful? Give feedback.
-
|
Correction: actually tmux allows pass-through from inactive but visible panes, so using quiet mode will help in this case. Invisible panes are still problematic (I have a patch enabling pass-through for invisible panes here, just in case). |
Beta Was this translation helpful? Give feedback.
-
|
Update: now tmux supports passthrough from invisible panes when |
Beta Was this translation helpful? Give feedback.
-
|
On Wed, Aug 03, 2022 at 12:03:41PM -0700, Sergei Grechanik wrote:
Update: now tmux supports passthrough from invisible panes when `allow-passthrough` is set to `all`, see [this commit](tmux/tmux@42ba6c1)
Nice!
|
Beta Was this translation helpful? Give feedback.
-
|
This is so cool! I tested an image in weechat inside tmux and it's working great. It would be really useful for me to have this. I had a problem with using tupimage inside tmux though. It seems it's not able to get responses from the terminal, it just waits a while after printing "Awaiting terminal response" and then it says:
The way I got it to work was to run tupimage before I started tmux, by switching to the alternate buffer and redirecting the output from tupimage to a file. Then I started tmux and printed the file. @sergei-grechanik: Do you have any idea why tupimage isn't working inside tmux for me? It seems that it does for you? I have allowed passthrough sequences, and sending kitty notifications from within tmux (which also requires passthrough sequences) is working. |
Beta Was this translation helpful? Give feedback.
-
|
Could be tmux dropping passthrough sequences or a bug in tmux detection. Could you run tupimage with -V and copy the logs here? Also, what's your value of $TERM inside tmux? (Maybe we should actually create a thread for this bug in tupimage issues). |
Beta Was this translation helpful? Give feedback.
-
Sure, I'll do that and post the info there. Thanks! Edit: Here: sergei-grechanik/tupimage#1 |
Beta Was this translation helpful? Give feedback.
-
|
@kovidgoyal So what do you think about including this in the kitty graphics protocol officially? I can refactor the implementation, add some docs, tests, a flag to icat, etc, and then create a pull request. Before that we need to agree on some details though:
|
Beta Was this translation helpful? Give feedback.
-
|
On Sat, Oct 22, 2022 at 07:16:24PM -0700, Sergei Grechanik wrote:
@kovidgoyal So what do you think about including this in the kitty graphics protocol officially? I can refactor the implementation, add some docs, tests, a flag to icat, etc, and then create a pull request. Before that we need to agree on some details though:
In principle, I am OK with it.
- The placeholder character. Currently it's `U+EEEE`, it's from PUA and easy to remember, doesn't seem to be used by [nerdfonts](https://www.nerdfonts.com/cheat-sheet), but otherwise I don't know how to choose unicode characters for things like that.
I would suggest using a character from the later planes, as the first
plane is used by lots of other things. Probably the range U+100000..U+10FFFD
if you like EEEE then use U+10EEEE. The downside is of course using 4
bytes instead of 3 to encode it in utf-8 during transmission. I dont
think that's a very large cost.
- The set of diacritics to use for rows/columns. Currently I use diacritics from the class 230, from which I excluded several ones that may be fused during normalization (although it's not that important since they will not be fused with PUA characters anyway). Here is the list: https://github.com/sergei-grechanik/kitty/blob/unicode-placeholders/rowcolumn-diacritics.txt
This seems OK to me. Note that currently kitty supports three diacritics
per cell at the time of our original discussion it was two.
- What attributes are used for IDs. Currently the foreground color encodes the image ID, and the underline color is planned to encode the placement ID.
foreground color seems fine, since we can special case the rendering of
the chosen char to render blank it wont cause any issues. underline
color also seems fine, since with no underline style specified they are
also not rendered. Otherwise you could potentially have rendering
artifacts.
- How to indicate that the placement is to be used by unicode placeholders in the placement creation command. Currently it's indicated by `U=1`.
U=1 seems OK.
There will likely be more details to take care of, such as avoiding
overhead when the scheme is not used. Possibly having icat use it
automatically if it detects tmux, etc. But those can be discussed
when reviewing the PR.
|
Beta Was this translation helpful? Give feedback.
-
|
I've recently learned a trick which makes it work in terminals running within tmux (like screen or terminal mode in vim and neovim). Apparently, it's possible to get the pty corresponding to the current tmux pane using the following command: Then we can send graphics command there directly, so there is no need to have pass-through support in nested terminals. Responses should still be read from /dev/tty. This seems to work for file-based uploading quite well, but unfortunately direct uploading commands get corrupted. |
Beta Was this translation helpful? Give feedback.
-
|
Is this a new tmux feature? Also if it doesnt work for direct transmission its a non-starter for icat since it needs to be robust. Is it because long escape codes are truncated? |
Beta Was this translation helpful? Give feedback.
-
|
I don't think it's new, I found it in some blog post from several years ago, and in the tmux repo I see pane_tty mentioned in some commit from 2013. |
Beta Was this translation helpful? Give feedback.
-
|
But wont this race with tmux? Presumably tmux can write to this device at any time, for example if the user has enabled tmux status bar it will update the status bar periodically. We then have interleaved writs on the device leading to undefined behavior. |
Beta Was this translation helpful? Give feedback.
-
|
No, the pane pty is for programs running in that pane. Tmux shouldn't write to it if I understand correctly. However, a race condition with these programs (i.e. the inner multiplexer) could be real. I think it can be mostly avoided by writing each command in a single write system call and making sure it's under POSIX_BUF bytes (4K on linux), in which case it's guaranteed to be atomic (to be precise, this is guaranteed for pipes, but I'm not 100% sure about pseudoterminals). It means that graphics commands themselves will be intact, but there is still a risk that they will be issued in the middle of an escape sequence produced by the inner multiplexer if it's not careful enough to write them in a single syscall. However, I would assume that multiplexers are usually written carefully enough. |
Beta Was this translation helpful? Give feedback.
-
|
On Sun, Oct 06, 2024 at 09:40:50PM -0700, Sergei Grechanik wrote:
No, the pane pty is for programs running in that pane. Tmux shouldn't write to it if I understand correctly. However, a race condition with these programs (i.e. the inner multiplexer) could be real. I think it can be mostly avoided by writing each command in a single write system call and making sure it's under POSIX_BUF bytes (4K on linux), in which case it's guaranteed to be atomic (to be precise, this is guaranteed for pipes, but I'm not 100% sure about pseudoterminals). It means that graphics commands themselves will be intact, but there is still a risk that they will be issued in the middle of an escape sequence produced by the inner multiplexer if it's not careful enough to write them in a single syscall. However, I would assume that multiplexers are usually written carefully enough.
Writes are only atomic on pipes if the pipe buffer does not
already contain data, if it does the write can only be atomic if its
size is under the available free space in the buffer. And that's for
pipes as far as I know there are no such guarantees for terminal devices.
And then you can have networks in the middle that chunk up
data arbitrarily. This kind of thing leads to very hard to debug
heisenbugs.
Also is there some documentation for this extra pty device? From what
you say it seems like this is a second pty device allocated per pane by
tmux, that tmux reads from and then internally multiplexes the data with
data read from /dev/tty of the pane? If that's the case why is it a pty
at all and not a unidirectonal pipe?
|
Beta Was this translation helpful? Give feedback.
-
It should block in this case.
But yes, I cannot find anything saying explicitly whether it works for pseudoterminals.
Not between tmux and the client program though, because we write to the pane pty directly. There might be ssh or whatever outside of tmux, but tmux will interleave graphics commands with everything else correctly (we still have to wrap them in passthrough).
Just "Pseudo terminal of pane", that's all documentation I see in the tmux repo. Empirically, it's the same one returned by |
Beta Was this translation helpful? Give feedback.
-
|
On Mon, Oct 07, 2024 at 08:24:06AM -0700, Sergei Grechanik wrote:
$ tmux display-message -t $TMUX_PANE -p "#{pane_tty}"
/dev/pts/36
$ tty
/dev/pts/36
```
Then it's the same as /dev/tty.
```sh
kitty
$ tty
/dev/pts/10
tmux
tty
/dev/pts/11
ll /proc/$$/fd/? master
lrwx------ 1 kovid kovid 64 Oct 7 21:07 /proc/308656/fd/0 -> /dev/pts/11
lrwx------ 1 kovid kovid 64 Oct 7 21:07 /proc/308656/fd/1 -> /dev/pts/11
lrwx------ 1 kovid kovid 64 Oct 7 21:07 /proc/308656/fd/2 -> /dev/pts/11
```
We see that there is exactly one new pty device created by tmux, number
11. This is the pane tty and it is the same device as
stdout/stderr/stdin. All of these are connected to /dev/tty
So writing to the pane_tty is the same as writing to
stdout/stderr and /dev/tty
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, unless there is a nested multiplexer. From screen running within a tmux pane: |
Beta Was this translation helpful? Give feedback.
-
|
Ah ok, now I get it. But the concern about the undefined behavior due to
races remains.
|
Beta Was this translation helpful? Give feedback.
-
|
Another trick to make it work with non-tmux multiplexers, even without tmux, is to use a named pipe:
I implemented this in tupimage: description, the code that redirects commands. This is an experimental proof-of-concept, and also direct transmission is 10x slower this way, so I'm not going to recommend it to anyone. |
Beta Was this translation helpful? Give feedback.
-
I've been thinking how to modify the kitty graphics protocol to make it work with tmux and vim without them knowing anything about the graphics protocol, and I got an idea that we can allocate a symbol (e.g. from the unicode private use area) to represent an image and then use the foreground color to distinguish between lines of this image. For example, if the symbol
♥is assigned to an image, it can be displayed by outputting something like this:I have a proof-of-concept implementation of this approach here. I would be glad to hear any useful feedback.
Screenshot:
Beta Was this translation helpful? Give feedback.
All reactions