Implement OSC 8 hyperlink support

[egmontkob]

GNOME Terminal (VTE) and iTerm2 have just implemented a brand new and hopefully really cool feature: hyperlinks (that is, HTML-like anchors).

This is different from the autodetection of URLs that appear onscreen. This time the text that appears does not have to look like a URL, it can be anything, and the target URL is specified explicitly by the utility along with the displayed text, using the OSC 8 escape sequence.

We'd like to get some wider support for this feature, and it would be great if kitty joined the game!

For further information on this feature, see

• A dedicated page with all the details: https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda

• GNOME Terminal bugreport (way too long and probably not that interesting at this point, although might contain some useful tips): https://bugzilla.gnome.org/show_bug.cgi?id=779734

• iTerm2 bugreport: https://gitlab.com/gnachman/iterm2/issues/5158

[kovidgoyal]

Sure, sounds useful, I'll look into it when I have some time. It's a fairly large change as there are currently no spare bytes in kitty's cell structure.

[kovidgoyal]

Shouldn't there be some way to query the terminal emulator for support of this feature? I'd suggest using the existing DECRQM escape sequence with a value of 2018 (2016 and 2017 are used by the kitty protocol extensions) https://github.com/kovidgoyal/kitty/blob/master/protocol-extensions.asciidoc

[egmontkob]

Indeed it's not one of those easy-to-implement features. :) It might as well make sense for you to wait and see if apps start using it.

As for querying: Ideally there should be a way, but I really despise asynchronous solutions. DECRQM/DECRPM might be nice for feature completeness against ancient specs, but I don't see much practical use.

What would a utility or application do? Emit this and then wait for the response... for how long? Some terminal emulators don't send a response at all. So there must be a timeout. Why make the app slower by this timeout in certain terminal emulators? (Even think about shell scripts launching such utilities in a loop.) What if the timeout is hit because of the delay over ssh? Would the app then just not emit hyperlinks, totally confusing the user why it's randomly sometimes buggy?

Fullscreen apps could just paint their screens without hyperlinks and then repaint once the positive response arrives. Silly user experience if the delay is visible, plus makes it quite complicated to implement in certain apps (they'd have to recognize this escape sequence in all parts of their code wherever they read and parse input, e.g. all widgets, dialog boxes etc.). Simple utilities that "just" print stuff can't go for this approach.

Pretty much all terminal emulators ignore unknown OSC sequences, and I believe that's more than good enough to get starting with wide deployment of this feature. Especially since I believe apps and utilities should not blindly emit this sequence, but make it conditional to some app-specific user preference or cmdline switch.

If some detection is implemented, it should be readily (synchronously) available for apps, e.g. as part of termcap/terminfo.

VTE, according to my best knowledge, does not implement DECRQM/DECRPM and maybe we have a low-traffic bugreport about it but I'm not even sure, maybe we don't have any. It's not something that users really miss.

[kovidgoyal]

termcap/terminfo is an awful solution because it means you have to have the terminfo files on remote servers as well when using SSH. Not to mention that the maintainers of terminfo have gone permanently to sleep.

As for asynchronicity, the problem domain is fundamentally asynchronous, because terminals have to work over networked connections (SSH). There is no way around that. Well technically, the way around that is to have the ssh program forward terminal info to the remote side as part of connection setup, but given that this is a network problem, making it happen is going to require lots of time and effort to get buy in from the various involved parties.

In this particular case, a well behaved application should either:

1. Wait for a response with a timeout.
2. Start outputting without OSC codes and switch to using OSC codes when a response is received.
3. Start outputting with OSC codes and stop when a response is not received.

[kovidgoyal]

Oh and assuming the terminfo maintainers can be budged, the solution for the timeout problem is simply to add a capability to terminfo for DECRQM. So if terminfo indicates the terminal supports it, client applications should wait, otherwise not.

[kovidgoyal]

Also, in case it is not clear, the reason I recommed adding detection for this feature is that client applications can change their output based onit. If the feature is present they can output clickable links as blue underlined text, for instance. That makes the feature much more discoverable.

[egmontkob]

termcap/terminfo is an awful solution

For terminfo I agree with you. For termcap, as far as I know, its value (in the $TERMCAP environment variable) could be forwarded via ssh, and that would probably be a proper design.

(Or even better, forward the entire terminal description, but instead of placing it in a giant environment variable, rather place it in a file and create an env var pointing to that file, similarly to $XAUTHORITY. Terminal emulators could initally just set this env var pointing to their local copy of the file. ssh would forward its contents, place in a temporary file on the remote host, set the env var accordingly, and remove the temporary file upon closing the connection... Hmmm, I'm wondering if they could do it with terminfo and $TERM as well...)

Anyway, I guess designing the proper solution is way out of scope of this particular feature. It would be nice to come up with a solution accepted by all major OSes, libraries etc., quite a few people have brought this up on various forums recently, but unfortunately I don't see a reasonable chance for this to happen.

maintainers of terminfo have gone permanently to sleep

ncurses's terminfo database is actively maintained, and despite a few folks requesting it, its maintainer has repeatedly refused to introduce an entry to denote 24-bit true color support. So I don't see a chance he being in favor of adding a flag for the hyperlink feature.

As for asynchronicity, the problem domain is fundamentally asynchronous

The full duplex channel is asynchronous by its nature, but normally it doesn't limit the overall speed. The problem is when you have to wait for a response because then roundtrip time becomes the bottleneck. With a small utility using DECRQM executed a thousand times in a shell script in a loop, and a roundtrip time of 0.1s, this becomes an additional 100 second delay. That's a huge problem! It's okay for ssh to take a few roundtrips as it initializes the connection, but subsequent operation should avoid that.

3. Start outputting with OSC codes and stop when a response is not received.

This sounds the worst idea to me so far. Like, the app starts to use a feature, and at one point it figures out it's not available and then it goes like "oops, sorry, lemme redo without it"? What do we expect for when it emits hyperlinks? The screen to appear correctly? Then why stop doing it? Or the screen to get garbled? Why would we allow to have a temporarily garbled screen?

Even with 1. and 2., the main problems are that detecting availability is slow and is extremely complicated. As such, I'm pretty sure hardly any (if any at all) applications will bother. I am a bit familiar with Midnight Commander, so let's take that as example. Each widget (e.g. the copy dialog if you press F5) is responsible for handling the input on its own. So each widget would need to look for this escape sequence possibly arriving, and instruct the global app to repaint. Probably requires huge and ugly refactoring. Keep in mind that you can typeahead, press F5 before mc starts up, that is, the DECRPM response would only arrive at mc after the F5 keycode. Plus, the user-visible behavior would still be suboptimal to say the least. We've been to similar situation, the urxvt mouse extension protocol pretty much failed because it was hard (not impossible at all, just hard) for applications to implement it. If we need a way of autodetecting at all, we need a damn simple way.

Also, think about simple utilities that "just" print stuff. They can have their standard file descriptors redirected, e.g. input read from a file, output piped to less -R. Where to send DECRQM and where to expect DECRPM to arrive from, then? Maybe the controlling terminal? What if it doesn't exist (e.g. ssh hostname command)?

No, this DECRQM approach is just not viable at all. Terminfo has its serious downsides, we agree in that, but I firmly believe DECRQM is even worse from a practical point of view.

So if terminfo indicates the terminal supports it, client applications should wait, otherwise not.

This might look good on paper, but not in real world practice. Terminfo is not something that works perfectly, it's something that "mostly works". Some emulators, e.g. VTE and konsole set TERM=xterm-256color, although their behavior is not exactly what's defined there. They used to set TERM=vte or TERM=gnome or TERM=konsole, but that approach just caused even more problems. Also take a look at stackoverflow how many people still to this day have problems with incorrect TERM variables inside tmux, etc. You simply cannot count on TERM being set properly, risking a complete freeze of apps if it isn't.

client applications can change their output based onit. If the feature is present they can output clickable links as blue underlined text, for instance. That makes the feature much more discoverable

Just because a terminal emulator supports hyperlinks, I don't think it means the application should suddenly change their behavior. There is a chance that the user doesn't like this behavior or just prefers the layout he got used be. Even if an app only wants to linkify some pieces of text, I think it could ask users whether they want it. If an app wants to change its layout (change its displayed text) because of hyperlinks, it should definitely be a user config, not some magic autodetection.

On discoveability, blue underline: VTE dashed-underlines hyperlinks for discoverability. (This might change in the future.) I believe it's better if the terminal emulator gives a unique look of all such strings, rather than if each app is responsible for choosing and explicitly setting the hyperlink style. Although I'm not sure it's good mixing the semantics and the presentation the way we did. We'll hopefully gather user feedback during the next months or few years, and revise if necessary...

[kovidgoyal]

On Fri, Apr 28, 2017 at 02:21:03PM -0700, egmontkob wrote: "termcap/terminfo is an awful solution" – For terminfo I agree with you. For termcap, as far as I know, its value (in the $TERMCAP environment variable) could be forwarded via ssh, and that would probably be a proper design. (Or even better, forward the entire terminal description, but instead of placing it in a giant environment variable, rather place it in a file and create an env var pointing to that file, similarly to $XAUTHORITY. Terminal emulators could initally just set this env var pointing to their local copy of the file. `ssh` would forward its contents, place in a temporary file on the remote host, set the env var accordingly, and remove the temporary file upon closing the connection... Hmmm, I'm wondering if they could do it with terminfo and $TERM as well...)

This can be done already via the TERMINFO env var. This is incidentally how kitty manages to work without needing to be installed using root access. But, yeah, I dont have the time/patience to convince the maintainers of ssh to use this, and in any case this would only be useful if the maintainers of terminfo could be convinced to actually add modern capabilities to their database.

Anyway, I guess designing the proper solution is way out of scope of this particular feature. It would be nice to come up with a solution accepted by all major OSes, libraries etc., quite a few people have brought this up on various forums recently, but unfortunately I don't see a reasonable chance for this to happen.

I agree.

"maintainers of terminfo have gone permanently to sleep" – ncurses's terminfo database is actively maintained, and despite a few folks requesting it, its maintainer has repeatedly refused to introduce an entry to denote 24-bit true color support. So I don't see a chance he being in favor of adding a flag for the hyperlink feature.

Really, that must have changed recently. But anyway, in my experience they are asleep, in yours they are un-cooperative, in either case, no dice.

"As for asynchronicity, the problem domain is fundamentally asynchronous" – the full duplex channel is asynchronous by its nature, but normally it doesn't limit the overall speed. The problem is when you have to wait for a response because then roundtrip time becomes the bottleneck. With a small utility using DECRQM executed a thousand times in a shell script in a loop, and a roundtrip time of 0.1s, this becomes an additional 100 second delay. That's a huge problem! It's okay for ssh to take a few roundtrips as it initializes the connection, but subsequent operation should avoid that.

See isatty(), which is synchronous and any well behaved app should only wait if its output stream is actually connected to a tty. And applications that are designed for rapid multi-execution can very well choose not to wait. The essential point you seem to be missing here is that feature detection is not compulsory. If it exists, applications for which it makes sense can use it, applications for which it does not, will not.

"3. Start outputting with OSC codes and stop when a response is not received." – This sounds the worst idea to me so far. Like, the app starts to use a feature, and at one point it figures out it's not available and then it goes like "oops, sorry, lemme redo without it"? What do we expect for when it emits hyperlinks? The screen to appear correctly? Then why stop doing it? Or the screen to get garbled? Why would we allow to have a temporarily garbled screen?

Because it is better than a permanently garbled screen.

Even with 1. and 2., the main problems are that detecting availability is slow and is extremely complicated. As such, I'm pretty sure hardly any (if any at all) applications will bother. I am a bit familiar with Midnight Commander, so let's take that as example. Each widget (e.g. the copy dialog if you press F5) is responsible for handling the input on its own. So each widget would need to look for this escape sequence possibly arriving, and instruct the global app to repaint. Probably requires huge and ugly refactoring. Keep in mind that you can typeahead, press F5 before mc starts up, that is, the DECRPM response would only arrive at mc _after_ the F5 keycode. Plus, the user-visible behavior would still be suboptimal to say the least. We've been to similar situation, the urxvt mouse extension protocol pretty much failed because it was hard (not impossible at all, just hard) for applications to implement it. If we need a way of autodetecting at all, we need a damn simple way.

Huh? Have the global application query support for all the extensions it needs once at startup, and set a global variable indicating whether it exists or not. After that all the application components/widget/what have you can simply read from the global variable.

Also, think about simple utilities that "just" print stuff. They can have their standard file descriptors redirected, e.g. input read from a file, output piped to `less -R`. Where to send DECRQM and where to expect DECRPM to arrive from, then? Maybe the controlling terminal? What if it doesn't exist (e.g. `ssh hostname command`)?

Once again, isatty().

No, this DECRQM approach is just not viable at all. Terminfo has its serious downsides, we agree in that, but I firmly believe DECRQM is even worse from a practical point of view.

The ability to optionally detect if a feature exists is never worse than no ability to detect it exists at all.

"So if terminfo indicates the terminal supports it, client applications should wait, otherwise not." – This might look good on paper, but not in real world practice. Terminfo is not something that works perfectly, it's something that "mostly works". Some emulators, e.g. VTE and konsole set TERM=xterm-256color, although their behavior is not exactly what's defined there. They used to set TERM=vte or TERM=gnome or TERM=konsole, but that approach just caused even more problems. Also take a look at stackoverflow how many people still to this day have problems with incorrect TERM variables inside tmux, etc. You simply cannot count on TERM being set properly, risking a complete freeze of apps if it isn't.

I said applications should wait if terminfo indicates support for DECQRM, not that they should wait permanently. The only downside to mis-configured systems is then a short delay at startup. But I totally agree that terminfo is awful. The proper solution for terminal querying capabilities is to have the kernel add support for it. Either via a new termios system call or via a new device file like /dev/pts-cap/<num>

"client applications can change their output based onit. If the feature is present they can output clickable links as blue underlined text, for instance. That makes the feature much more discoverable" – Just because a terminal emulator supports hyperlinks, I don't think it means the application should suddenly change their behavior. There is a chance that the user doesn't like this behavior or just prefers the layout he got used be. Even if an app only wants to linkify some pieces of text, I think it could ask users whether they want it. If an app wants to change its layout (change its displayed text) because of hyperlinks, it should definitely be a user config, not some magic autodetection.

Huh? The choices here are either: 1) No feature detection which means client applications always change their output regardless of terminal emulator used 2) Feature detection: which means client applications have at least the possibility of doing the right thing, depending ont he terminal emulator being used application configurability is completely orthogonal to this.

On discoveability, blue underline: VTE dashed-underlines hyperlinks for discoverability. (This might change in the future.) I believe it's better if the terminal emulator gives a unique look of all such strings, rather than if each app is responsible for choosing and explicitly setting the hyperlink style. Although I'm not sure it's good mixing the semantics and the presentation the way we did. We'll hopefully gather user feedback during the next months or few years, and revise if necessary...

Yes, and in order for a user to know some random piece of text is clickable they have to first hover their mouse over it. There is a reason links on the web are highlighted.

[kovidgoyal]

Actually thinking about it, one does not even need to involve the kernel at all. All one needs is a simple program that can store and retrieve terminal capabilities keyed by terminal device file. So the terminal emulator calls

echo hyperlinks,colored-underlines,... | terminal-capabilities set /dev/pts/[fd-no]

Client programs call

terminal-capabilities get

to get the above list. It will get the capabilities for the current controlling tty

SSH will simply call terminal-capabilities get on the local computer and then call terminal-capabilities set on the remote computer with the result.

For security terminal-capabilities set should only allow the capabilities to be set once per (file, creation-time, inode) key. That way, only the program creating the pty can change the capabilities.

There you have, robust, synchronous detection of terminal capabilities.

The only requirement is that the terminal-capabilities program be installed on the computers.

[kovidgoyal]

Obviously, the format and semantics of the capabilities needs to be defined more carefully, but the essential mechanism of setting/getting them will work as above, regardless of the details.

[kovidgoyal]

I edited the above proposal slightly, to use file paths instead of fd numbers.

[egmontkob]

The essential point you seem to be missing here is that feature detection is not compulsory. If it exists, applications for which it makes sense can use it, applications for which it does not, will not.
[...]
The ability to optionally detect if a feature exists is never worse than no ability to detect it exists at all.

I would love to have a simple way to autodetect this feature. One that's simple enough for pretty much all utilities to use without going into the inevitable troubles of asynchronity. But not one which is so complicated as this proposal and has so many problems.

Introducing such a troublesome way of autodetection is, in my opinion, much more of an additional burden and source of problems to carry than a solution for apps that are interested.

VTE has been doing fine for maybe like 15 years without DECRQM. It has tons of features that cannot be autodetected. Just to stay at the OSC world, there's no way to detect whether OSC 0, 1, 2 (title, icon title), 6, 7 (current file and directory) etc. work, and at this point we're no longer specific to VTE. This hasn't been a problem so far. E.g. just as with hyperlinks, an app might want to know if whatever it puts in the title will actually be visible for the user or not. No way to do that, and noone has been missing this feature yet.

If you add an asynchronous escape sequence, which I'm still strongly against, I believe you should figure out a generic way for OSC sequences, rather than maintaining an arbitrary mapping between OSC and DECSET numbers, and reserving a number in the DECSET range which doesn't have a corresponding DECSET/DECRESET behavior. E.g. similarly to OSC 4, 5, 10... reporting the given color, it could be OSC 8 with a ? parameter. Althouogh, for OSC 4, 5, 10... these actually report a real value rather than the existence of the support.

Also, I believe this should at the very least be discussed involving all people who participated in the hyperlink feature.

Due to the problems I've mentioned above (and clarified below as well), I don't have any motivation in implementing or advocating any asynchronous behavior. Of course I can't and don't speak in the name of VTE's main developer, or iTerm2's developer.

If there's anything I'd be happy to see and happy to contribute some work towards, although I don't have time to drive it, would be to get OSes and distributions to patch their terminfo downstream, or to have a fork of terminfo, which would contain flags for features like truecolor, hyperlinks and possibly other new ones. See e.g. the comments from https://gist.github.com/XVilka/8346728 dated this Feb-Mar.

Huh? Have the global application query support for all the extensions it needs once at startup

What you are missing here: There might be (and usually there are) multiple places in an app's code where it reads input. And all of them, without any exception, have to be prepared for receiving the given sequence due to its asynchronous nature and the possibility of typing input keys before the application starts up. Or, the application refactored so that the input goes through a common preprocessing phase (which handles and removes this response) before it reaches those individual places where it is read.

Or just think about a simple shell script using the read builtin. How would you prevent the value you read from stdin into a shell variable from getting corrupted with the async response??

Once again, isatty().

Such responses address a small subset of the problem area, but not the entire area. E.g. what if I pipe the output to less -R (with OSC 8 support) and hence definitely do want hyperlinks to be present? (Sure there's another one-liner answer for that which will raise further questions, but I just don't see the whole picture getting put together.)

2. Feature detection: which means client applications have at least the possibility of doing the right thing, depending ont he terminal emulator being used

Assuming that the feature detection is designed well, works robustly, and is supported across terminal emulators. Which your proposal is not.

---

To summarize, and to make myself more explicit and more pragmatic, because I feel like this discussion is no longer leading us anywhere:

• I strongly believe that any asynchronous approach is fundamentally broken, and even having it as an option is worse than not having it (due to the problems the partial terminal emulator support and partial application support would introduce and we'd need to carry in the long run).

• I see absolutely no way you or anyone else could change my opinion here.

• I am not the guy making the decisions at VTE, and not an iTerm2 developer either. I am a regular contributor (developer if you wish) of VTE.

• I am not willing to spend any time and effort on a feature that I find broken. Unless i) the main VTE developer approves it and ii) someone pays a truly decent amount of money to me, which I find pretty unlikely to happen. I am speaking in my name only, not in the name of VTE's main developer or other contributors or the VTE project.

• I am happy to put some limited work (way less than the entire amount of work necessary) into getting certain terminfo patches accepted across various OSes and distros that introduce flags for the truecolor and hyperlink feature.

• I am happy to put some limited work (magnitudes less than the entire amount of work necessary) into coming up with a brand new design and implementation that addresses all the fundamental problems with all the technologies we have right now.

• All that being said, Kitty is your project and of course you do with it whatever you want. :-)

[kovidgoyal]

If I do implement this feature in kitty, in the future, it will be with feature detection. The idea that not having any feature detection is better than having feature detection is just plain wrong. Since you dont seem to want to have a discussion, I wont bother elaborating on that.

[egmontkob]

It's not that I don't want to have a discussion. We had a decent one, and I believe we got to the point where our opinions won't get any closer to each other. Hence I see no point in repeating ourselves over and over again.
Cheers!