Adding a new colour scheme to parse $LS_COLORS

[Nelyah]

Following the discussion from ranger/ranger#169 (comment)
Plugin and colour scheme to parse the $LS_COLORS
The colour scheme still needs some work. It works for the extensions (such things as *.txt), but doesn't work for special files (denoted di, ln, ...)

[rpdelaney]

os.stat.st_nlink can be used to determine if a file is a hard link. I don't know if we get access to the file path in the color scheme though. Probably other special file types can be polled with the same method.

[vifon]

The colors seem off for me. I'm attaching the screenshot.

[toonn]

@Nelyah, pinging you about this. It's something many people've wanted so it'd be great if you could fix this up/or rationalize why it can't be fixed up so we can merge this : )

[Nelyah]

Thanks for the reminder, I had indeed left it all a bit behind. I'm working on it right now, and I will probably give a more detailed update later (hopefully tonight). :)

[Nelyah]

I pushed another commit, which should properly parse the LS_COLORS. Let me know if it works for you.

I added as well device and socket colouring, however it seems to be more than just doing a os.stat on the file. From what I have seen, I should look into the container/* directory for directions about how to handle other specific kind of UNIX files.

As of now, the colour scheme recognises:

• executable
• fifo
• link
• Block Device and Character Device
• Socket
• Directory
• all specified extensions

I specifically added a rule to prevent the directory from being coloured since they (mostly) are exectuable.

Also, one issue I've been running into (but which seems irrelevant to this issue), is ranger not having the LS_COLORS variable properly set. If you intend to use ranger as a default file manager, the LS_COLORS value should be available session wide, and not only from the shell.

[toonn]

Briefly testing this seemed to work. About the plugin, what's your reasoning for having that code in a plugin? (I'm just thinking about simplicity of distribution it'd be easier if there was only one file. Otherwise we have to completely rethink the colorschemes repo ; )

[Nelyah]

I originally followed the documentation which was suggesting to define new context tags from the plugin folder.

However, I put the code from the plugin file to the colorscheme one and it seems to work. I pushed the last version with the colorscheme file at the top level of the git repo so you wouldn't have to reorganise it :)

edit: Also added a rule to no colour links as executables (same as with ls I believe).

[pseyfert]

when I unset LS_COLORS this throws an exception that's uncaught. (getenv('LS_COLORS') is None, so getenv('LS_COLORS').split(':') raises an AttributeError.)

I think you want to check if getenv('LS_COLORS') is None instead of ls_colors. Or catch the exception.

[toonn]

Can you provide a default to getenv? Otherwise you can use os.environ.get('LS_COLORS', default value). Imo that looks like the proper way to solve this problem.

[pseyfert]

Had to look it up myself, getenv('LS_COLORS', default='') looks good. (NB: in two places in the file)

What I didn't realise earlier. It seems the uncaught exception makes ranger fall back to the default colors. Once I add the default value or exception handling in class base(ColorScheme): and when LS_COLORS is not set, then ranger becomes black&white.

I don't think this is what we want here (ls --color falls back to default colors when LS_COLORS is not set).

[Nelyah]

So, if we want the default LS_COLORS, we should either let ranger automatically fall back to its default scheme, or we should manually read the default dircolors file or run the dircolors command without arguments to get the default colouring. Also, are we sure that is what ranger really does? (falling back to default LS_COLORS) This could simply be a matter of catching the exception.

I've had trouble however finding the system wide config file that dircolors automatically reads. The man page (man dir_colors) points to either ~/.dircolors (user-defined) or /etc/DIR_COLORS (system-wide). I, however, do not have the system-wide one and yet dircolors is still able to read the colours from somewhere.

Have you had any luck finding whence it gets the colours?

[pseyfert]

Not sure what's the best to do. I guess given the name 'ls_colors' it should be as close as reasonable to "what ls does".

ranger gets its default colors from https://github.com/ranger/ranger/blob/master/ranger/colorschemes/default.py which happens to be the same as (my?) ls defaults when LS_COLORS is unset.

I also don't see where my dircolors gets its defaults from.

Given we don't seem to see where dircolors gets its config from (and one would need to check user file, system file, fallback) I think calling dircolors if LS_COLORS is unset is a good handling.

[Nelyah]

I added a commit to handle the case where LS_COLORS isn't set.
Scenario cases:

1. LS_COLORS is defined and parsed
2. LS_COLORS isn't defined, dircolors is called and we use its output to define the LS_COLORS for the colorscheme
3. LS_COLORS isn't defined and the dircolors command can't be found. We use a black & white colorscheme.

the 3) is probably not what the user want, but it might be a good idea to let them know something is wrong if neither a LS_COLORS env variable or a dircolors command can be found while using the ls_colors.py scheme.

[toonn]

Ranger probably "falls back", actually doesn't change to ls_colors.py, because that throws an error.
The proper behavior would be to do what ls does. The default.py colorscheme doesn't completely match what ls --color does without LS_COLORS set.

I meant to use whatever actually is the default value for LS_COLORS as the default but calling dircolors every time is probably fine too.

Do you really need to duplicate the try/except block for the getenv?

[rpdelaney]

@toonn

Do you really need to duplicate the try/except block for the getenv?

Can you elaborate? I don't see duplication here. Or should this thread be resolved?

[pseyfert]

This refers to an old version. the old file I still have from September has two calls getenv('LS_COLORS'). The current version in this PR does not have that duplication.

[Nelyah]

I removed the try/except block inside the class.
That also made me think about the necessity of putting some code outside the class itself. Do you think it would be a good idea to put the "global" code in an __init__ function inside the class (which I should probably rename ls_colors)?

This would probably allow for better interaction with potential plugins.

[toonn]

Maybe keeping it in the class is better yeah. That way themes based on this one can override the behavior. And I agree about the name. It's not necessary but it's nicer when people start extending your colorscheme.

[Nelyah]

I haven't had much time these past few days, just to let you know I have not forgotten it. I'll try to get on to it this week-end!

[Nelyah]

I (finally) added the code correctly, and called the super for the Colorscheme class (compatible python2).

As mentionned in the commit, I also fixed an issue that would make files highlighted if they ended with the extension (but without the .). For instance, considering the extension .bar, the file foobar shouldn't get coloured while foo.bar should.

[rpdelaney]

Awesome. I'll be taking a look at getting this installed locally soon. Also eagerly awaiting this being added to master.

[Nelyah]

I just want to let you know that I'll go for an extended (~1 year) trip around the world and since I like to travel light I won't have any computer (except a smartphone), which means I won't be able to patch should any issue arise.

I'm leaving on the 28th of October, and on the case where we haven't resolved this PR by then someone will have to take it up from there (I could add that someone as a contributor to my repo).

[toonn]

@Nelyah, no problem. We already have access to your repo because of the PR. I'm only waiting for @vifon to verify the earlier issues were resolved. This should be merged before you leave.

[vifon]

Currently it works great. Just one nitpick: the tags are no longer in a distinct color. Is that intentional?

[vifon]

Suggested change

print(self.ls_colors_keys)

I presume this line was here for debug purposes only?

[vifon]

I have found 3 more problems:

• the selection doesn't seem to get highlighted
• if I name a directory for example something.png, it gets colored as a directory but it's bold since the PNG files are bold in my config
• if a file has a colored background, it behaves strangely in ranger

[rpdelaney]

I guess @Nelyah has left for their trip. I'll try to see if I can find time to pick up the slack. I very much want to see this landed.

[atticus-sullivan]

Actually I don't remember exactly in which directory the second issue was, but I tried several folders in /dev (it was somewhere there) and it looked great.
I couldn't reproduce the weird colouring in the tree columns as well. For me this looks fine.

[rpdelaney]

@toonn how are we doing? :)

[toonn]

.> <.< Who? me??

This got a bit neglected, I'll review and merge soon™.

[rpdelaney]

@toonn Anything we can do to help? :)

[toonn]

Someone in the irc channel actually said they wanted to do an exa colorscheme, which would be a superset of this.
However, the only thing blocking this is it not being clear which bits are still lacking. So everyone should start bother @vifon to find out what issues they found. : )

[SmartFinn]

38;5;xx;x notation for 256 colors doesn't parse well, attr 5 should be ignored but it isn't. I faced with non-stop blinking in gnome-terminal.

[dnastase]

Hi, just bumping this as well. This would be a very nice plugin to have :)

[jmcantrell]

I tried this color scheme on my system and got strange blinking colors.

I'm using https://github.com/trapd00r/LS_COLORS.

My terminal is xterm.

Output of uname -a:

Linux garnet 5.12.6-arch1-1 #1 SMP PREEMPT Sun, 23 May 2021 00:45:50 +0000 x86_64 GNU/Linux

Output of ranger --version:

ranger version: ranger 1.9.3
Python version: 3.9.5 (default, May 12 2021, 17:14:51) [GCC 10.2.0]
Locale: en_US.UTF-8

Output of dircolors:

LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:';
export LS_COLORS

[toonn]

Oof that's terrible. Could you minimize your LS_COLORS? Pretty hard to figure out what might be causing this since it's rather long.

[rpdelaney]

Could it be related to this comment?

[jmcantrell]

Here are some specific ones.

Python files:

LS_COLORS='*.py=38;5;41' ranger --cmd="set colorscheme ls_colors"

Executable files:

LS_COLORS='ex=38;5;208;1' ranger --cmd="set colorscheme ls_colors"

[jmcantrell]

Given that every i is added to to_delete unconditionally, this is essentially clearing the entire list. Is this intended?

[toonn]

That does seem incorrect, yes. Though it is copying the entire list if I'm not mistaken.

[jmcantrell]

Yes, you're correct. It is copying. I saw to_delete and read that last condition as not in. I'm guessing that the intention was to filter out any of the style attributes that were matched in the loop above it.

[jmcantrell]

Yeah, that's not right either. Another method is making decisions based on those attributes.

[jmcantrell]

I just realized that attribute 5 is blink, which makes me wonder why my terminal isn't blinking outside of ranger. Maybe I should be using different color definitions?

[jmcantrell]

After some digging, I found that these color definitions are xterm control sequences, which are supported by dircolors. If your terminal is capable of 256 colors, the following can apply.

From https://www.xfree86.org/current/ctlseqs.html:

If 88- or 256-color support is compiled, the following apply.
P s = 3 8 ; 5 ; P s → Set foreground color to the second P s
P s = 4 8 ; 5 ; P s → Set background color to the second P s

According to that, the blink attribute is being used as a delimiter between the foreground/background code and the color value. This is obvious from:

colorschemes/ls_colors.py

Lines 144 to 149 in de43255

	if current_attr == 48 and attribute_list[i + 1] == 5:
	bg_colour = attribute_list[i + 2]
	looking_at_256_ttl = 2
	elif current_attr == 38 and attribute_list[i + 1] == 5:
	fg_colour = attribute_list[i + 2]
	looking_at_256_ttl = 2

Although, it appears that even when 256 colors are not in use, the blink attribute is interpreted as another bold:

P s = 5 → Blink (appears as Bold)

Now, in this color scheme, when parsing the attributes, blink is interpreted as-is, but should probably be ignored.

colorschemes/ls_colors.py

Lines 103 to 104 in de43255

	elif attr == 5:
	return_attr |= style.blink

@Nelyah It's clear from looking at the code that you're already aware of everything I've said above about xterm control sequences. Are there any other implications to simply ignoring the blink attribute on lines 103-104? Removing them gets rid of the blinking, but it's also causing crashes that I'm unable to see.

Does anyone know how to use ranger's logging within a color scheme? Using a standard logger overwrites the interface with messages, and it's difficult to use.

[Nelyah]

Sorry for the reply that was long in coming @jmcantrell. I will try to give this a closer look in the coming days.

As for logging, you can always debug the code and output errors to a file. On a separate terminal window you could use tail -f <filename> to get the log.

[benoit-pierre]

I ended up rewriting pretty much completely a previous version of this PR: ls_colors.py.

That version fix a number of issues:

• work with trapd00r/LS_COLORS: support target for links, support patterns like *LS_COLORS=48;5;89;38;5;197;1;3;4;7, correctly parse 256 colors attributes like *.1p=38;5;7, support italic
• ignore case (image.jpg and image.JPG should be handled the same)
• only use LS colors when applicable, falling back on a base colorscheme (default, configurable) for highlighting the rest: unfocused pane browser (when using set viewmode multipane), VCS info, line numbers, tags, pane indicator, ...

To change the base colorscheme, rename the file to ls_colors_BASE_SCHEME_NAME.py (or create a symlink): e.g. ls_colors_solarized.py to use the solarized color-scheme as base.

Also included is some testing code, available when running the file by itself (python colorschemes/ls_colors.py): this will create a temporary directory with a bunch of entries matching the LS colors spec (taking advantage of the fact that most entries match themselves) and open ranger on it. Prefix the command with fakeroot and some fake devices will be created too. This can be used to test any theme (ranger's config is not changed).

Screenshots:

• with default base:
• solarized terminal theme and base:

[rpdelaney]

@toonn What do you think of the above?

@benoit-pierre Do you think it would make sense to open a separate PR with your implementation?

[toonn]

A separate PR seems suitable. Don't like having the filename of the theme being part of configuration though.

[rpdelaney]

@toonn Any suggestions? Personally, I wouldn't be attached to any particular naming scheme.

[toonn]

Either have it be in rc.conf or allow passing an argument so you could use ls_color_them(fallback=some_other_theme) as a theme.

[jpggithub]

Just a ping as this PR could be useful for several people and the discussion seems to be near the end (if I understand correctly ;-)

[jfmoulin]

March 2023... still eagerly waiting...
Thanks for the nice job anyways!

[toonn]

@jfmoulin, have you tried out the colorscheme in this PR? If you have been using it and haven't noticed issues for a while then we could go ahead and merge this.

[jfmoulin]

Hi @toonn ,
I tried it out and do not see any effect :0(
I must have some conflicting options somewhere...
I copied ls_colors.py to my .config/ranger/colorschemes and renamed it by appending various basenames for locally available schemes. No joy.
I tried different versions in my rc files (with and without setting a colorscheme).
I'll have to dig deeper.

[toonn]

@jfmoulin, I don't think renaming the file does anything. The class name inside the file is the important bit. But I'm not sure what you think renaming anything will do? You're supposed to set your Ranger colorscheme to ls_colors and then configure it through the LS_COLORS environment variable.

[jfmoulin]

@toonn
well, regarding the renaming of the ls_colors.py, that is what I gathered from the docstring in the file...
I guessed one could (mis?)use some import mechanism to select different configurations.
Quoting the docstring:
Ranger color-scheme using $LS_COLORS/dircolors`.

Originally based on: https://github.com/ranger/colorschemes/raw/a250fe866200940eb06d877a274333a2a54c34f3/ls_colors.py

Usage: copy this file to ~/.config/ranger/colorschemes'. The base color-scheme used for non file system entries / the unfocused pane is default. To change it, rename this file to ls_colors_BASE_SCHEME_NAME.py(e.g. change it tols_colors_snow.pyto use thesnowcolor-scheme as base).'''``

Anyways, setting colorscheme=ls_colors is one of the several things I tried, without effect.
My LS_COLORS does not seem to be considered.

[jfmoulin]

I've tried one more thing: I ran the test in ls_colors.py, the just started ranger uses the proper coloring to display the faked files. So far, so good. However, when I jump to a directory which is not the test dir, the entries are not colored (within the same instance of ranger)!

[jfmoulin]

@toonn, things seem to be working after removing some scripts which were manipulating the LS_COLORS on the fly.
On a fresh install of ranger, it works like a charm.
Thanks!

[rpdelaney]

Still praying for this to land 🙏 Thanks for the thread bumps :D

[eggbean]

It seems to support code 5 for slow flashing text (I've not seen support for this for years - I think PuTTY?), but when 38 or 48 is followed by 5 it's supposed to allow 8-bit 256 colours as the third number, for the foreground and background respectively.

https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit

I've got a dir_colors file that uses a lot of the extended 256 colours so I see a lot of flashing files when using this.

I'd like more terminals to support 5 for broken symlinks and orphan files, but it shouldn't include codes that are 38;5;xxx and 48;5;xxx.

I don't think the older version of this ls_colors.py does this.