256/16 million color support

[Qix-]

Yes, it's time.

I spent the greater part of last night trying to figure out the best way to tackle compatibility with ANSI 256/16m (truecolor) support to where colors would degrade nicely. I started writing my own engine but alas I figured this may be better suited as a PR for an existing solution (i.e. Node's biggest package, Chalk). Now that supports-color will soon support more colors, this is a good time to look at it.

There are a few issues issues I foresee with an upgrade such as this:

• What happens to 256/16m colors on terminals that only support 16 colors ("basic" support)? 16m colors on terminals that only support 256 colors?
• How do we accurately and easily wrap color operations into an already expressive API, with full backwards compatibility?

Color Operations

The way my engine does it (and the way that seems to work fine) is to use the color-js package to perform color operations on full color values. This includes modifying lightness, saturation, and per-channel values.

These color values can then be converted to 8bpc RGB color values, which can then be used to degrade into the lower forms.

Tests

First of all, let's define our domain scaling function. This function takes a value, along with the upper/lower bound of that value's domain, and then applies it to a lower/upper bound of another domain.

applyDomain = (val, lb, ub, tlb, tub)->
  Math.round ((val - lb) / (ub - lb)) * (tub - tlb) + tlb

From there, degrading colors becomes somewhat of a simple task.

For each of the three levels, I'll be decoding an image using png-js and running the pixel data through each of the functions.

The original image:

16m (8bpc)

This one is simple. Provided a 0..255 domain of RGB values, we can then build up a pretty lossless result in the terminal.

color-js provides RGB values in floating point numbers in the domain 0.0..1.0. As you might expect, this is easily converted to the 0..255 domain using applyDomain n, 0, 1, 0, 255.

16m (8bpc) to ANSI 256

Thanks to this formula (thanks @jbnicolai) we know the per-channel domain of ANSI-256 colors is 0..5 with a translation of +16.

toAnsi256 = (r8, g8, b8)->
  r = applyDomain r8, 0, 255, 0, 5 # red, from 0..255 to 0..5
  g = applyDomain g8, 0, 255, 0, 5 # green, from 0..255 to 0..5
  b = applyDomain b8, 0, 255, 0, 5 # blue, from 0.255 to 0..5
  return (36 * r) + (6 * g) + b + 16

As you can see, it's quite accurate.

16m (8bpc) to 16 colors

This one was a little trickier, but really achieves the same effect.

toAnsi16 = (r8, g8, b8)->
  r = applyDomain r8, 0, 255, 0, 1 # red, from 0..255 to 0..1 (rounded)
  g = applyDomain g8, 0, 255, 0, 1 # green, from 0..255 to 0..1 (rounded)
  b = applyDomain b8, 0, 255, 0, 1 # blue, from 0..255 to 0..1 (rounded)
  return (b << 2) | (g << 1) | r

this yields predictable results.

coffee> toAnsi16 = require './ansi16'
[Function]
coffee> toAnsi16 255, 255, 255
7
coffee> toAnsi16 255, 0, 0
1
coffee> toAnsi16 255, 0, 126
1
coffee> toAnsi16 255, 0, 127
1
coffee> toAnsi16 255, 0, 128
5
coffee> toAnsi16 255, 127, 128
5
coffee> toAnsi16 30, 185, 128
6

As you can see, it's also pretty accurate (minus orange, which is a tricky color to get right with the standard 16 color palette).

Though by using color-js and the lightness value bound to the domain 0..1 (rounded), then we can select the brightness scheme using the extended 100m-107m bright background codes to get a better result.

Much better - and with only the basic functionality (and yes, even Windows supports those).

Internal structure

With the domain functions being pretty accurate, we can then start making assertions about color values. This means we can store color-js Color objects as the main source for color, perform operations on them, and then degrade them down to corresponding command line colors.

For instance, here is my prototype library doing this with a nice shade of Purple. There isn't a ton of logic about the 16 color palette, but I'm sure something could be thrown in there for lightness governing the bold (1m) code.

---

As you can see in the above prototype, we now have a lot of control over the kinds of colors we can output to the console. We can do advanced things like shift by hue, lighten/darken by lightness/value, and even start theming console colors with CSS (if we so choose) - all while being completely degradable by non-supporting terminals.

Existing programs can upgrade their color suites to be more colorful, and if we so choose supports-color could even include or modify an environment variable to allow users to set this in their RC files to enable system-wide 256 or 16 million color terminal applications.

Thoughts? I'm fully prepared to make a complete PR for this.

[jbnicolai]

Just a quick note (I'm traveling atm) that I did something similair a while back. You can probably find the discussion under chalk's issues and see my implementation here: https://github.com/jbnicolai/ansi-256-colors

[Qix-]

@jbnicolai I stole the RGB -> ANSI 256 formula from there 😉 This proposal does something a little more than just that, though. If you wanted to, we could beef up the ansi-256-colors repository to do the color-to-ANSI-code conversions and then let this library use that? Though ansi-256-colors is a bit of a misnomer since we'd also be supporting truecolor as well.

[jbnicolai]

Nah, I'm perfectly happy retiring that repository and beefing up the chalk/ansi-styles/supports-color stack, although I'm starting to think we should maybe move this set of repositories to their own organisation.

How would you feel about that, @sindresorhus?

[Qix-]

http://github.com/chalk seems inactive. Github is pretty good about releasing inactive accounts. http://github.com/chalkjs is also unclaimed.

EDIT: @jbnicolai don't forget ansi-regex and strip-ansi.

[jbnicolai]

@Qix- yeah, completely agree those should probably come with as well. It's @sindresorhus' call though :)

[Qix-]

👍

[Qix-]

I'm looking at the existing code for this repo. It's pretty basic. It'd be pretty much a complete re-write.

[Qix-]

I've started a bit of a re-write as it is. Any thoughts on this @sindresorhus?

[sindresorhus]

I used up all my time today moving everything over to the chalk org. Will get back to this in a couple of days when I have more time. Tweeted about getting feedback on this, so we can hopefully get some more eyes on it.

[Qix-]

Sounds good 👍

[sunesimonsen]

My package https://github.com/sunesimonsen/magicpen supports 256 with backwards compatibility with 16 color terminals. Maybe that could be a source of inspiration, the ansi serializer is here: https://github.com/sunesimonsen/magicpen/blob/master/lib/AnsiSerializer.js

[Qix-]

@sunesimonsen and what of 16m colors? @jbnicolai already had a module that did something very similar. This proposal does something completely different.

[sunesimonsen]

@Qix- we don't support that yet, but I'm curious to see what you come up with.

[Qix-]

@sunesimonsen as in the original post here, we'd be using/working with full RGB values and degrading into their lower forms (truecolor -> 256 -> 16 colors).

[Qix-]

x-post: chalk/chalk#73

[Qix-]

Also, another question is: where should everything go?

We have a few moving parts here:

• Level detection
• Color manipulation functionality (hue shifting, lightness adjustments, RGB <-> HSV/HSL conversions, etc.)
• Degradation from truecolor to 256/basic values
• The fact we want to be able to map these to basic colors (i.e. rgb(255, 0, 0) should map to 31 instead of 38;2;255;0;0)
• Listing above combinations (and applying labels to them)
• Styles (bold, italic, underline, inverse, etc.)
• The API for stringing them together

Some of the above are obvious (i.e. stringing them together would occur in chalk proper), but the others?

As well, this has the potential to introduce a major performance hit if not done correctly (color modifications aren't nearly as quick as lookups). Memoization will be important here.

I propose the following:

• Level Detection: supports-color
• Color Manipulation: ansi-styles / new module (see note below)
• Degradation: ansi-styles / new module (see note below)
• Color Mapping: ansi-styles
• Map Labels: ansi-styles
• Styles: ansi-styles
• API: chalk

---

Color manipulation (lightness, hue, etc.) was originally going to happen with color-js. However, upon actually looking at the code, it doesn't look very performant / memory conscious. The code is under the BSD clause 2, which means we can use the algorithms with attribution and build our own structure around it.

But that's all up to @sindresorhus.

[Qix-]

Oh I'm not done.

Another consideration: if we go from high->low resolution on colors, the 16 color palette is going to be tricky.

There are a few ways we can interpret RGB values for the 16 color palette.

By the math, regular colors should be 0..127 and bright colors should be 128..255, though that's not necessarily true on all VGA systems. Do we enforce this idea? People usually have them re-mapped anyway.

Surprisingly, Windows XP gets it the closest.

[sunesimonsen]

@Qix- I also use full RGB values and degrading them to the 256 and 16 color palette. I wanted 256 colors to work in terminals that supports 256 colors but can't be detected as such, so we wrap the 256 color escape sequence in a 16 color escape sequence. Old terminals will ignore the 256 color escape sequence and fallback to the 16 color escape sequence. It is a bit hacky, but it works.

[Qix-]

@sunesimonsen Ehh I feel like that's a workaround that will backfire on some terminals. Plus that craps up the terminal with more data than it needs - TTY operations are expensive as it is. Cool hack, though. Never really thought about that.

[sunesimonsen]

@Qix- I'm not saying you should copy that, I'm just telling you what I have done.

[jbnicolai]

@sunesimonsen that's pretty awesome actually 😄 - although I don't suggest we copy that here

[Qix-]

The fact is the color operations will be easy, it's the API that's troublesome.

[Qix-]

I've made a PR for color-convert for ansi16/256 color degradation algorithms.

Qix-/color-convert#18

[MoOx]

PR as been merged and released as color-convert@0.6.0.

[Qix-]

Thank you @MoOx :)