Implement SGR dim?

[maximbaz]

Hello,

I'm migrating my alacritty configuration to kitty, and while configuring colors I noticed that alacritty has a whole separate section for "dim" colors besides "normal" and "bright", while kitty does not allow defining "dim" colors.

How is it working today, is kitty approximating "normal" colors to get "dim" versions of those? If yes, since I have exact colors from my theme, I'd prefer to use those instead of relying on the approximation.

This is the PR that introduced dim colors for alacritty: alacritty/alacritty#633

[kovidgoyal]

I'm not sure what you mean. In kitty you can configure all 256 colors that can be accessed via ANSI escape codes. If you mean support for the DIM SGR code then that is a legacy code intended for cathode ray tube based hardware terminals, it doesn't really make sense any more, and kitty does not support it. And as far as I know most terminals either dont support it or are intending to drop it

[maximbaz]

I'm not sure in terminology too, in alacritty's PR it was called "VTE 'dim' flag". Judging by the discussion in this PR (alacritty/alacritty#492), this is supported in alacritty, Konsole, urxvt and iTerm2.

I found one way how to use dim colors in alacritty - via tput dim (will show below). In the PR discussion people used colorama/demo01.py to showcase the behavior of "dim" colors.

In alacritty, I can configure "normal", "bright" and "dim" colors, and then:

$ tput setaf 3; echo 'aa'              # this renders "yellow - normal"
$ tput setaf 11; echo 'aa'             # this renders "yellow - bright"
$ tput dim; tput setaf 11; echo 'aa'   # this renders "yellow - normal"
$ tput dim; tput setaf 3; echo 'aa'    # this renders "yellow - dim"

In kitty, I can configure "color3" and "color11", and then:

$ tput setaf 3; echo 'aa'              # this renders "yellow - normal"
$ tput setaf 11; echo 'aa'             # this renders "yellow - bright"
$ tput dim; tput setaf 11; echo 'aa'   # this renders "yellow - bright"
$ tput dim; tput setaf 3; echo 'aa'    # this renders "yellow - normal"

Basically, kitty ignores this dim flag. Does this not make sense to support? If I use apps that were made with colorama, I will not see as many colors in kitty as I can see in the rest of the terminals.

[kovidgoyal]

Yeah that's the SGR dim and no kitty is not going to support it. As I said, it is a holdover from the days of hardware terminals. I'm pretty sure it is slated for removal from VTE as well, (https://bugzilla.gnome.org/show_bug.cgi?id=791596) and #135 . There is no point to it in a software terminal that supports full true color and fully theme-able 8-bit colors. If you want dim yellow you can get it by using a direct 24bit color or by using a 8-bit color. Like this:

printf '\033[38:5:178m this is dim yellow' (using 8-bit colors)
or
printf '\033[38:2:215:175:0m this is dim yellow' (using 24bit colors)

Oh and there is no need to use tput to do any of this. All terminals support essentially the same set of SGR codes for colors. I dont know why people still use tput/terminfo for colors. See https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters Ironically the one place where terminal implementations differ, syntax for 24bit colors, terminfo is useless, because the maintainer of ncurses thinks no one needs more than 256 colors. For that refer to https://gist.github.com/XVilka/8346728

[maximbaz]

All right, basically my primary concern was the existence of tools that use this old SGR dim instead of normal colors. But so far I haven't seen any actual example of this being an issue, and I'm all up for reducing the amount of unnecessary legacy in the world 🙂

Regarding tput, that was just the first result in google on how to use a dim color 😉

Thanks for the examples and the explanation.

[jordwalke]

The wildly popular terminal test runner Jest makes very good use of Dim and I have found many use cases that 256/trucolor cannot easily replace. For example, when you want to write a terminal program for showing errors and dim some portion of the output to draw the user's attention to a specific part of the output. I haven't been able to find any other way besides dim, which is actually very well supported in iTerm. The reason it is challenging is because for command line output, you want to use the current terminal theme, and not invent a new color scheme. You don't know which hue of red will look decent for the user's current background (when they have configured their 8/16 colors). So by using dim, you're saying that the terminal/user can decide what it means to have a subdued version of a color. Imagine if I chose "solarized" as the color scheme for my error output, but the user's terminal was configured with a totally different set of colors. It wouldn't look natural. But with dim, I can create very beautiful output that looks great regardless of what color scheme the user has configured their terminal with.

That is the output from a command line program that dims the code around the red error highlighting, and that program works great with any terminal 16 color theme. "dim" is the only way I can imagine implementing a command line app like that.

Note that without dim, the code surrounding the red error highlight would be very bright blue/orange and distract from the primary error. Here's how it looks on Kitty:

[kovidgoyal]

Sure, you can implement such a program without using dim. Use the escape codes to query the current value of the colors in the terminal and then choose your dim shade accordingly. It's a little more work but works for far more use cases than just dimming something. The only downside is that the program cannot output dim colors until it receives a response from the terminal emulator, which should not matter for a test runner, since time to run tests is going to be orders of magnitude higher that RTT to the terminal.

For reference, you can look up the query codes here: http://invisible-island.net/xterm/ctlseqs/ctlseqs.html

[kovidgoyal]

Also, I should add, that the correct way to have more than 16-themeable colors would be to theme all 256 colors in the 8-bit color table (which kitty supports). Then in your example, to access dim red, the terminal program could just use <ESC>[38:5:52m to change to dull red, which would be themed to match whatever color red is.

See https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit

While this means more work for theme creators (you now have to theme 256 colors instead of 16 colros) it is a one time cost and it allows terminal programs to have 256 themeable colors, instead of just 16 (or 24 with dim).

This has the virtue of being more elegant code wise, more general, and is supported on more terminals out of the box than dim.

[kovidgoyal]

kitty supports changing all 256 colors via its conf file and dynaically at runtime. See https://github.com/kovidgoyal/kitty/blob/master/README.asciidoc#how-do-i-change-the-colors-in-a-running-kitty-instance for the latter.

[jordwalke]

I've never heard of being able to query the terminal for their concrete 16 colors. It's cool to hear that this is supported, but how well supported is it by various terminals? Part of the goal is to make it so command line app builders can implement something that tends to work in most places.

[jordwalke]

What was the specific color querying command you were referring to? I had a hard time finding it in that large document.

[kovidgoyal]

As far as I know it is supported by many terminals, probably more than support dim. See the section Operating System Commands in that document. Those codes allow you to both set and query various colors.

[kovidgoyal]

Thinking about it a little, there is one way in which implementing dim makes sense. That is, rather than having 8 configurable colors, dim should cause the foreground color to be rendered with increased alpha, so that it blends more into the background. Advantages:

1. This would work with both light-on-dark color schemes and dark-on-light color schemes, without special configuration by the user
2. it would affect all colors, not just the first eight and it would be closer in spirit to the original use of dim, which, I believe was to reduce the intensity of the electron beam in a CRT display, and not to manipulate colors at all.

This would actually work with @jordwalke use case as well, since the dimmed regions of text would fade comapred to the non-dimmed regions.

I think it should be fairly easy to implement in kitty, since kitty already has the machinery to alpha blend foreground colors (it is used for the inactive_text_alpha option).

That said, this is just a proposal, I am not particularly keen to implement it. Comments are welcome.

@egmontkob thoughts?

[egmontkob]

Probably the approach that makes the most sense to me is what you're doing right now: not to support dim at all :-D

This suggested approach also has some minor inconsistencies, e.g. the final look (color) of the U+2588 FULL BLOCK character depends on the background.

That being said, I think it's a good idea, definitely a reasonable one.

[jordwalke]

If it's entirely possible for people to implement Dim in "user space" by manually blending the colors in 256 terminals, then that is also a nice approach. No one seems to do anything like that as far as I can tell. I believe that iTerm takes the approach you described for Dim (lowering the alpha) and it works really well. This kind of Dim has one other valuable purpose which is really hard (impossible?) to implement otherwise (happy to be corrected if wrong): You can output colors with dims, then after the colors are output, the user can change their terminal theme and all the colors transfer over into the new output, along with their dimmings. In this screenshot, I output the Dim codes in iTerm, and only after the log had been output, did I change the theme to be light. Notice how the original terminal output could be "rendered" accurately in another color scheme since I didn't output hard-coded 256 colors. I think that without Dim, that would be very hard to accomplish in a way that works in nearly all terminals. This use case is important because the whole reason you might switch to another color scheme is that it is too bright outside to read some log output that is on your screen right now, and so you need a switch to a light colorscheme. Often, rerunning a build to regenerate the output in the new color scheme wouldn't work. With Dim, you can toggle themes. The eight color pallet is actually kind of brilliant in this respect - it's just that it's too limited with only 8 colors. Dim kind of comes to the rescue.

[kovidgoyal]

That works automatically with my second approach of themeing 256 colors and using the dimmer variants from them in the application. When you change the theme, the dim colors will be automatically remapped, in all existing output.

There is not really any use case for DIM that cannot be implemented in other nicer, more portable, but slightly harder ways.

The question is really whether to support DIM for compatibility with software that does not do the right thing. There are plenty of warts with DIM.

1. It works differently in different terminals (applying to only eight colors, applying to all colors, being user configurable, or not, etc.). It does not work at all in many terminals.

2. It has no SGR code to turn it off (instead the code to turn off bold turns off dim as well, which means you cannot combine DIM with BOLD).

Given those constraints I'm really surprised application developers choose to use it. I presume people aren't aware of the alternative approaches.

[jordwalke]

"That works automatically with my second approach of themeing 256 colors and using the dimmer variants from them in the application"

I think Dim still has an advantage because it allows:

1. An application author to write a program that works on most popular terminals.
2. Dimming effect.
3. Without the user having to configure special color schemes.
4. Where the user can toggle color schemes after the output has been logged.

I believe the 256 pallet solution requires custom configuration on the user's part which they're unlikely to do. I could be wrong, but I'm really glad to see you implemented DIM anyways and I look forward to trying it out. Thanks for working on Kitty! It's excellent.

(To solve the problem where SGR code for turning off bold also turns off DIM, I think that terminal drawing libraries will need to implement some custom handling. Unfortunate, but I think the users will be happy.)