set_color updates for bright colors, fixes for low/no-color terminals #3260

Merged
merged 6 commits into from Sep 11, 2016

Projects

None yet

6 participants

@floam
Member
floam commented Jul 23, 2016 edited

Description

In #3176 I noticed that the reason the addition of bright colors was also causing issues with TERM=xterm (on modern terminals), was that fish would generate invalid ANSI:
We assumed that it we could use tparm to emit color escapes as long as the color was under 16 or max_colors from terminfo was 256:

if (idx < 16 || term256_support_is_native()) {                                                                                                                          
    // Use tparm to emit color escape
    writembs(tparm(todo, idx);      

But if a terminal has max_colors (terminfo) of, say, 8, here is what will occur as it is asked for a number higher than that (except tparm did the same thing in output.cpp):

> env TERM=xterm tput setaf 7 | xxd
00000000: 1b5b 3337 6d                             .[37m 
> env TERM=xterm tput setaf 9 | xxd
00000000: 1b5b 3338 6d                             .[39m

The first escape is good, that second escape is not valid. Bright colors should start at \e[90m:

> env TERM=xterm-16color tput setaf 9 | xxd
00000000: 1b5b 3931 6d                             .[91m

This is what caused "white" not to work in #3176 in Terminal.app. (On real 8-color terminals the bigger issue was that white should cause color 7, not color 15, addressed below)

So we replace the < 16 || term256_support_is_native(), with the latter function just checking if max_colors is 256 or not, with a function that takes an int and checks terminfo for the. colors supported by the terminal (really, we care that tparm can be expected not to output garbage.)

/// Returns true if we think tparm can handle outputting a color index                                                                                                  
static bool term_supports_color_natively(unsigned int c) { return max_colors >= c; }                                                                                    
...                                                                                                                                                                         
if (term_supports_color_natively(idx) {
    // Use tparm to emit color escape.
    writembs(tparm(todo, idx));

If terminfo can't do it, the "forced" escapes no longer use the fancy format when handling colors under 16, as this is not going to be compatible with low color terminals. The code before used:

else {
    char buff[16] = "";
    snprintf(buff, sizeof buff, "\x1b[%d;5;%dm", is_fg ? 38 : 48, idx);

I added an intermediate format for colors 0-15:

else {                                                                                                                                                                  
     // We are attempting to bypass the term here. Generate the ANSI escape sequence ourself.                                                                            
     char buff[16] = "";                                                                                                                                                 
     if (idx < 16) {                                                                                                                                                     
         snprintf(buff, sizeof buff, "\x1b[%dm", ((idx > 7) ? 82 : 30) + idx + !is_fg * 10);                                                                             
     } else {                                                                                                                                                            
         snprintf(buff, sizeof buff, "\x1b[%d;5;%dm", is_fg ? 38 : 48, idx);                                                                                             
     }

The PR also fixes the definition of color white, fixes color values for all colors. The non-brights were defined with the bright values and brights had something invented by fish. e.g. FF0000 for red (instead of 800000) and FF5555 for brred (instead of FF0000).

I got rid of the hack in config.fish. For #2951, one thing we should do is decide if we really want to send ANSI at terminals with max_colors <= 8, if no, solve it in C++?

Fixes #3176, #2951

TODOs:

  • Changes to fish usage are reflected in user documenation/manpages.
  • Tests have been added for regressions fixed
@faho faho and 1 other commented on an outdated diff Jul 23, 2016
share/tools/web_config/js/colorutils.js
- str = str.toLowerCase()
- if (str == 'black') return '000000'
- if (str == 'red') return 'FF0000'
- if (str == 'green') return '00FF00'
- if (str == 'brown') return '725000'
- if (str == 'yellow') return 'FFFF00'
- if (str == 'blue') return '0000FF'
- if (str == 'magenta') return 'FF00FF'
- if (str == 'purple') return 'FF00FF'
- if (str == 'cyan') return '00FFFF'
- if (str == 'white') return 'FFFFFF'
- if (str == 'normal') return ''
- return str
+ str = str.toLowerCase();
+ if (str == 'black') return '000000';
+ if (str == 'red') return '800000';
@faho
faho Jul 23, 2016 Member

Wouldn't it be nice here to be consistent with other programs, particularly terminals?

From wikipedia, it seems X will use F00 for red, but 008000 for green and 00F for blue, like we currently do. Xterm is a bit weirder.

@floam
floam Jul 23, 2016 edited Member

There are good arguments for matching another terminal or some tangible standard. However if we might ever do something like let set_color red apply a 24bit color code instead of one of these 16 colors, I think there are better arguments for very carefully picking out a great set of preferred colors that work well on both dark and light terminals to make the color contrasts a bit easier on the eyes.

In the case above though, I was just going by this table, except for the colors which I left in for the sake of compatibility at their prior values that do not actually exist as named colors in ANSI (brown, purple, grey). I'm happy with using any values we can agree on.

http://www.calmar.ws/vim/256-xterm-24bit-rgb-color-chart.html

@floam
floam Jul 23, 2016 edited Member

Huh, those few colors in the bright column for X are surprising (in a good way.) They're actually quite near what I end up using in my terminal configurations instead of full blast green and blue.

@faho
Member
faho commented Jul 23, 2016

The documentation changes are a definite ACK, and I think I'm okay with white->grey. I'd prefer it if we still accepted "grey" as an alternative name - set_color erroring out isn't a great experience.

I'll have to test this a bit, particularly in a VT.

@faho faho added this to the next-2.x milestone Jul 23, 2016
@floam
Member
floam commented Jul 23, 2016

ACK?

@floam
Member
floam commented Jul 23, 2016 edited

prefer it if we still accepted "grey" as an alternative name - set_color erroring out isn't a great experience.

We do. What I decided to do was remove them from the documentation and hide them from the color list but make sure set_color does what the scripts using those values intend.

@faho
Member
faho commented Jul 23, 2016

ACK?

http://www.catb.org/jargon/html/A/ACK.html, meaning 4. You really need to grow your beard 😃.

Okay, in a VT this seems to "just work" - set_color maps the argument to the 8+8 normal/bright colors, and everytime it hits a bright one the VT just ignores it.

In a monochrome TERM=vt100, this hits #2951 in full force.

We do. What I decided to do was remove them from the documentation and color list but make sure set_color does what the scripts using those values intend.

Sorry, I should have tried before. I'm not sure if we shouldn't stick a note in the docs - "For compatibility, 'grey' is accepted as a synonym for white". Not having it in the list is probably right, though.

@faho faho and 1 other commented on an outdated diff Jul 23, 2016
src/output.cpp
@@ -37,15 +37,20 @@ static int (*out)(char c) = writeb_internal;
/// Whether term256 and term24bit are supported.
static color_support_t color_support = 0;
+
+/// Set the function used for writing in move_cursor, writespace and set_color and all other output
@faho
faho Jul 23, 2016 Member

What's up with the three "/" here? I see they've been here before, but why?

@floam
floam Jul 23, 2016 edited Member

These are HeaderDoc comments.

@floam
Member
floam commented Jul 23, 2016 edited

What result does the monochrome terminal give for tput colors? I think if we document grey we should also document the other two aliases - brown and purple.

@faho
Member
faho commented Jul 23, 2016

What result does the monochrome terminal give for tput colors?

"-1".

In what way? The wrapper script is nuked from config.fish, so I know not the error quoted on that issue, what exactly is the terminal receiving for escapes and exhibiting?

It prints nothing, which is annoying in combination with the cartesian product - echo (set_color red)rose would not print anything. Maybe set_color should at least print a newline, or am I missing something?

(Also I've noticed that, when I give set_color an RGB value, $fish_term24bit (set via config.fish because I have $KONSOLE_PROFILE_NAME set) takes precedence and it'll print the RGB sequence)

@floam
Member
floam commented Jul 23, 2016 edited

Hm, set_color probably shouldn't print a newline, any character that isn't whitespace? What would be a good one that would effectively be a NOP and not take a column or cause a linebreak on a vt100?

@faho
Member
faho commented Jul 23, 2016

Hm, set_color probably shouldn't print a newline.

Why not?

But I think any character that would effectively be a NOP and not take a column or cause a linebreak on a vt100 should suffice.

Which do you have in mind? I'm leaning towards a '\n' because that's the most-used character at the end of a programs output, so it seems to be the least likely to cause issues. Also it'd work (by being eliminated) in a command substitution, which is how set_color is used the most.

Compare:

echo (printf '\n')banana | od -t x1z
echo (printf '\r')banana | od -t x1z

And non-ASCII characters are right out here (haven't seen a vt100 with emoji support yet).

@floam
Member
floam commented Jul 23, 2016

Isn't that going to... add a spurious linebreak scripts were not expecting when there isn't the cartesian product thing happening?

@floam
Member
floam commented Jul 23, 2016

I was thinking something like an escape sequence that does nothing.

@faho
Member
faho commented Jul 23, 2016

Isn't that going to... add a spurious linebreak scripts were not expecting when there isn't the cartesian product thing happening?

You're right. I was under the impression we'd reset the terminal mode between commands, but we only do that for the prompt (which is why running set_color red interactively has no effect).

I was thinking something like an escape sequence that does nothing.

Maybe the "reset" sequence (\e[0m)? I mean in a command substitution that'd be captured, but that's okay since you were expecting something to be captured anyway (e.g. checking for string-equivalency on a variable that includes a color sequence is usually a dumb idea).

@floam
Member
floam commented Jul 23, 2016 edited

I think that fixed it.

@ridiculousfish ridiculousfish and 1 other commented on an outdated diff Jul 23, 2016
share/functions/__fish_config_interactive.fish
@@ -299,26 +299,6 @@ function __fish_config_interactive -d "Initializations that should be performed
set -g fish_pager_color_description yellow
set -g fish_pager_color_progress cyan
@ridiculousfish
ridiculousfish Jul 23, 2016 edited Member

Do I understand correctly that what this deleted function used to do (per commit f71e877 ) is now handled by the builtin set_color? That seems good if so.

@floam
floam Jul 23, 2016 edited Member

Not exactly. It does not mask off certain arguments for TERM=linux (or TERM=vt100), but the goal is it will work on a VT by not throwing escapes at it that are problematic. It silently might just do something other than try to set the color they asked for.

@floam
Member
floam commented Jul 23, 2016 edited

OK, I think 6d12fbe should stop causing the empty output with echo (set_color red)foo - I hope there aren't side effects.

@floam
Member
floam commented Jul 23, 2016

(Also I've noticed that, when I give set_color an RGB value, $fish_term24bit (set via config.fish because I have $KONSOLE_PROFILE_NAME set) takes precedence and it'll print the RGB sequence)

Is that OK? (Because that's only set when you're actually in Konsole?)

@floam floam changed the title from Bright colors: fix naming and bugs on terminals with 8-16 colors. to set_color updates for bright colors, fixes for low/no-color terminals Jul 23, 2016
@floam floam added a commit that referenced this pull request Jul 24, 2016
@floam floam clang-format colorutils.js
.... should make the #3260 diff shorter.
dfe363c
@faho faho commented on the diff Jul 24, 2016
src/builtin_set_color.cpp
@@ -179,7 +179,9 @@ int builtin_set_color(parser_t &parser, io_streams_t &streams, wchar_t **argv) {
write_color(rgb_color_t::black(), true /* is_fg */);
writembs(tparm(exit_attribute_mode));
} else {
- write_color(fg, true /* is_fg */);
+ if (!write_color(fg, true /* is_fg */)) {
@faho
faho Jul 24, 2016 Member

This appears non-obvious and in need of a comment:

"If we can't write any color, write a reset sequence so cartesian products don't nix something like echo (set_color foo)bar. This should work in monochrome terminals."

@faho
Member
faho commented Jul 24, 2016

(Also I've noticed that, when I give set_color an RGB value, $fish_term24bit (set via config.fish because I have $KONSOLE_PROFILE_NAME set) takes precedence and it'll print the RGB sequence)

Is that OK? (Because that's only set when you're actually in Konsole?)

Yes, it should be okay. Just a bit surprising.


I've now retested this and it seems to work now with TERM=vt100 (just never setting any color) in what is actually Konsole.

@ridiculousfish
Member
ridiculousfish commented Jul 24, 2016 edited

These net changes look good. Can you please squash some commits and/or clean up the commit messages? In particular c2bd798 and 9bb2de9.

I haven't caught on up what the behavior change here is, I need to see what it's like to use it.

@floam
Member
floam commented Jul 24, 2016 edited

@ridiculousfish Let me know if that is any more straightforward. I simply squashed most of the code changes together and redid the commit log to explain the behavior change.

@ridiculousfish
Member
ridiculousfish commented Jul 24, 2016 edited

Squashing looks good. I have some concerns about the commit messages:

8d48ba :
This is a great fix, but this makes me nervous: "I think this'll stop swallowing the cart. product". Well, does it or doesn't it? Also please elaborate on the issue in the commit message. We'll want to relnote it too.

817b19 :
Please briefly mention the sort of update this is.

891896 :
Wow! This message is actually too long and detailed. For example it's not necessary to duplicate the code itself in the commit messages - we can find that through git.

@ridiculousfish ridiculousfish and 1 other commented on an outdated diff Jul 24, 2016
doc_src/set_color.txt
@@ -2,36 +2,35 @@
\subsection set_color-synopsis Synopsis
\fish{synopsis}
-set_color [OPTIONS] [COLOR]
+`set_color` [OPTIONS] *VALUE*
@ridiculousfish
ridiculousfish Jul 24, 2016 Member

These backticks and asterisks are showing up in the man page and the docs. Were you trying to get some particular styling here?

@floam
floam Jul 25, 2016 edited Member

Whoops. Yes. In the rest of the manpage backticks caused <tt> in HTML or boldness in man, and asterisks caused italics in HTML and underlines in the manpage.

These are all nice behaviors and match other manpages on my system for non-fish projects where command names are bolded and keywords are underlined.

@floam floam Improve compatibility with 0-16 color terminals.
Fish assumed that it could use tparm to emit escapes to set colors
as long as the color was under 16 or max_colors from terminfo was 256::

 if (idx < 16 || term256_support_is_native()) {
    // Use tparm to emit color escape
    writembs(tparm(todo, idx);

If a terminal has max_colors = 8, here is what happenened, except
inside fish:

 > env TERM=xterm tput setaf 7 | xxd
   00000000: 1b5b 3337 6d                             .[37m
 > env TERM=xterm tput setaf 9 | xxd
   00000000: 1b5b 3338 6d                             .[39m

The first escape is good, that second escape is not valid.
Bright colors should start at \e[90m:

 > env TERM=xterm-16color tput setaf 9 | xxd
   00000000: 1b5b 3931 6d                             .[91m

This is what caused "white" not to work in #3176 in Terminal.app, and
obviously isn't good for real low-color terminals either.

So we replace the term256_support_is_native(), which just checked if
max_colors is 256 or not, with a function that takes an argument and
checks terminfo for that to see if tparm can handle it. We only use this
test, because otherwise, tparm should be expected to output garbage:

 /// Returns true if we think tparm can handle outputting a color index
 static bool term_supports_color_natively(unsigned int c) { return max_colors >= c; }
...

 if (term_supports_color_natively(idx) {

And if terminfo can't do it, the "forced" escapes no longer use the fancy
format when handling colors under 16, as this is not going to be compatible with
low color terminals. The code before used:

 else {
     char buff[16] = "";
     snprintf(buff, sizeof buff, "\x1b[%d;5;%dm", is_fg ? 38 : 48, idx);

I added an intermediate format for colors 0-15:

 else {
     // We are attempting to bypass the term here. Generate the ANSI escape sequence ourself.
     char buff[16] = "";
     if (idx < 16) {
         snprintf(buff, sizeof buff, "\x1b[%dm", ((idx > 7) ? 82 : 30) + idx + !is_fg * 10);
     } else {
         snprintf(buff, sizeof buff, "\x1b[%d;5;%dm", is_fg ? 38 : 48, idx);
     }

Restores harmony to white, brwhite, brblack, black color names.
We don't want "white" to refer to color color #16, but to the
standard color #8. #16 is "brwhite".

Move comments from output.h to output.cpp

Nuke the config.fish set_color hack for linux VTs.

Sync up our various incomplete color lists and fix all color values.
Colors 0-8 are assumed to be brights - e.g. red was FF0000. Perplexing!

Using this table:
 <http://www.calmar.ws/vim/256-xterm-24bit-rgb-color-chart.html>

Fixes #3176
3669805
@floam
Member
floam commented Jul 25, 2016 edited

Wow! This message is actually too long and detailed. For example it's not necessary to duplicate the code itself in the commit messages - we can find that through git.

This was mostly meant for you - so I've moved that to the top here. I'm doing a shorter one for the commit.

Thanks for the help!

@floam
Member
floam commented Jul 25, 2016 edited

This is a great fix, but this makes me nervous: "I think this'll stop swallowing the cart. product".

I wasn't too confident when I committed it that there wouldn't be some kind of side effect of resetting all the time and didn't try it on a real monochrome terminal. But since you say it's a great fix, I feel better. 😃

floam added some commits Jul 23, 2016
@floam floam Update set_color documentation
Update docs for "brblack", "brwhite"  existing.

We no longer mention colors like grey, brown and purple, which are aliases
for yellow, magenta, white/black. The color names still work but there
isn't a good argument for there being two ways to do that: especially in
the age of 24-bit terminals where one might expect yellow and brown or
magenta and purple to actually be different colors.

Copyedit rest of document for inaccuracies, strange advice, brevity (a lot
of "you" pronouns, for example.)

Document the color fallback feature (set_color 313554 blue) that's been
present quite a while.
644ea82
@floam floam Stop swallowing the cartesian product
This should work:
 > env TERM=vt100 ./fish -c 'echo (set_color red)"hi"'

We do a ::reset() if setting the color doesn't happen.

Fixes #2951
5afd939
@floam
Member
floam commented Jul 27, 2016 edited

What we want to do for max_colors == 8 terminals like linux is if 8 < idx < 16, enter bold mode at a higher level than where most of the work was so far, such that was_bold is correct (else we'll lose track of being in bold mode), and then do a color escape for idx - 8.

When tputs colors returns 8, we're still usually capable of 8 more, higher intensity colors. It just cannot be set independently from bold (which sucks so we want to use the tparm for the real idx or roll those aixterm escapes whenever possible).

Once I've got it doing that fish color behavior should be pretty ideal.

@faho
Member
faho commented Aug 27, 2016

@floam: What's missing here? In my (not too thorough) tests I haven't noticed anything off.

@floam
Member
floam commented Sep 9, 2016

I'm just going to merge what's here and commit a further improvement in the works later. This needs more testing prior to the upcoming release and shouldn't be delayed further.

@floam floam commented on an outdated diff Sep 11, 2016
share/functions/__fish_config_interactive.fish
@@ -24,14 +24,28 @@ function __fish_config_interactive -d "Initializations that should be performed
#
# If we are starting up for the first time, set various defaults
- #
- if not set -q __fish_init_1_50_0
+ #
+ if not set -q __fish_init_2_39_8 # bump this to 2_40_0 when rolling release if anything changes after 9/10/2016
@floam
floam Sep 11, 2016 edited Member

I want folks to pound on it with normal 0-8 colors, bright colors, bold normal colors, and #RRBBGG colors. So I did this to blast away color settings and put in new defaults. It doesn't need to be kept in for the real release.

@floam floam modified the milestone: fish 2.4.0, next-2.x Sep 11, 2016
@floam floam self-assigned this Sep 11, 2016
@floam
Member
floam commented Sep 11, 2016

Ok, merging this. Hopefully we can shake out any bugs prior to release. The final TODO I'm hammering on to be committed later, brights for "8" color terminals.

What we want to do for max_colors == 8 terminals like linux is if 8 < idx < 16, we enter bold mode at > a higher level than where most of the work was so far, such that was_bold is correct (else we'll lose > track of being in bold mode), and then do a color escape for idx - 8.

@floam floam Update pager colors, tweak pager.cpp
Adds a color reset thing, to ensure fish tries to use hard colors during
testing.

Also, work on a discrepancy (not introduced by my changes, afaik) when
with some combinations of color settings, and usage of --bold, caused super
flakey color paninting in the pager. Downwards movements that trigger
scrolling vs. upwards movement in the pager would only apply bold to
selections when moving upwards. The bold state of the command completions in
the pager was flipping flops on and off, depending on if there is a description
on the preceding line.

Implement a lame fix by reseting the color to normal and applying a
different style on the rightmost ')' which seems to be what was influencing it.

Makes fish use terminfo for coloring the newline glich char.
90e535f
@floam floam merged commit f7c6426 into fish-shell:master Sep 11, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@EMayej

command substitution does not happen. Maybe you don't want surrounding quotes.

echo "$HOME/fish_previous_colors-(date).txt"
/Users/EMayej/fish_previous_colors-(date).txt

echo $HOME/fish_previous_colors-(date).txt
/Users/EMayej/fish_previous_colors-Mon Sep 12 13:14:07 CST 2016.txt

echo $HOME/fish_previous_colors-(date '+%Y%m%d%H%M%S').txt /Users/EMayej/fish_previous_colors-20160912131557.txt

Member
faho replied Sep 12, 2016

You're correct of course, though this doesn't really matter - this is just to set colors again so we can exercise the recent changes a bit (#3260).

It should be removed before a release.

Member

doh.

Member
floam replied Sep 13, 2016 edited

And yeah, we won't be doing cat /dev/zero > $fish_color* on release. Still worth doing as intended until then.

I do want to force any settings using deprecated color-names to reset on release, however. I'm not sure it's necessary to be noisy/worry about backups. If we do pick some new defaults anywhere we really want for people to receive (perhaps for the pager, if we can fix the bolding bug), we can just quietly force those specific colors to reset.)

@TimWolla TimWolla commented on the diff Sep 12, 2016
share/functions/__fish_config_interactive.fish
#
# If we are starting up for the first time, set various defaults
- #
- if not set -q __fish_init_1_50_0
- if not set -q fish_greeting
- set -l line1 (printf (_ 'Welcome to fish, the friendly interactive shell') )
- set -l line2 (printf (_ 'Type %shelp%s for instructions on how to use fish') (set_color green) (set_color normal))
- set -U fish_greeting $line1\n$line2
+ #
+ if not set -q __fish_init_2_39_8 # bump this to 2_4_0 when rolling release if anything changes after 9/10/2016
+ set -g colors_backup "$HOME/fish_previous_colors-(date).txt"
@TimWolla
TimWolla Sep 12, 2016

This does not properly insert the date. The file is literally named fish_previous_colors-\(date\).txt. I don't think this was indented, was it?

@krader1961
krader1961 Sep 13, 2016 Member

Not only that but even if we supported command substitution inside quoted strings it is generally a bad idea to use the user locale for formatting the date in this type of context. It's almost always a better to use a fixed format such as ISO 8601. I'll take care of it.

@floam
floam Sep 13, 2016 edited Member

Why? it's just intended to be manually noticed by a developer with ls and not clobber old backups (hmm, I forgot noclobber and it bit me!) - not automatically restored or anything. Don't invest too much time in it.

@floam
floam Sep 13, 2016 edited Member

Could also just put (random) in there. The file birth date will keep the needed information.

@floam
floam Sep 13, 2016 Member

Thanks for fixing it.

@floam
Member
floam commented Sep 13, 2016 edited

I do want to force any settings using deprecated color-names to reset on release, however.

What we should probably just do is keep an init routine that will run on upgrade to 2.4.0, but what it does instead of "normalizing" colors (sorry, was a joke) by deleting them is replace previous names it encounters with the correct names.

grey -> white
brgrey -> brblack
brown -> yellow
white -> brwhite
purple -> magenta

That should prevent us from "teaching" deprecated color names through our defaults, and fix it so that colors display like how they did before this patch.

@floam
Member
floam commented Sep 13, 2016 edited

That should prevent us from "teaching" deprecated color names through our defaults, and fix it so that colors display like how they did before this patch.

Actually, I should check: did we adjust the defaults after doing our "white now refers to bright white/FF FF FF, normal white is called 'grey' now" change which this PR walked back? If no, then we could just leave those as is and they'll return to the behavior as designed finally.

edit: Looks like no

@sagebind sagebind referenced this pull request in oh-my-fish/oh-my-fish Nov 10, 2016
Closed

Anyone knows why my theme is displayed in a reversed color? #440

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment