support 256 color #345

Closed
melspectrum opened this Issue May 11, 2016 · 57 comments

Projects

None yet
@melspectrum

please have 256 color support. thanks!

@zadjii-msft
Member

Thanks for the suggestion! This is actually pretty high on my personal list of requests for the future, however, we weren't able to fit it into the Anniversary Update, though I'd keep my eye out in the future for it.

A coming insider update should have support to at least take the 256-color ANSI sequences and the full RGB ANSI sequences and find the nearest color that's in our 16-color table, and use that to render, which should at least enable applications which use those sequences to not be totally broken.

I'm gonna close this as a dupe of #76, which has more discussion on the subject.

@musm
musm commented Jul 15, 2016 edited

Could we get this reopened as #76 was closed since it was decided that the scope there was not 256 color support but just properly mapping 8-bit colors to the 256-bit range.

@benhillis
Member

Good suggestion!

@benhillis benhillis reopened this Jul 15, 2016
@be5invis

It is hard, almost impossible.
Conhost uses a fixed 32-bit-per-cell layout (16 bits for properties and 16 bits for character code), which is a derivative of VGA text mode (yes, the DOS-era video memory layout, 8 bits for properties and 8 bits for color). Modifying it will definitely break compatibility, unless MS want to implement a completely different console API set (and provide a compatibility layer for existing console apps).

@musm
musm commented Jul 15, 2016 edited

Thanks @be5invis. It's 2016, breaking compatibility with DOS I don't think will in anyway affect anyone using windows 10, maybe a dozen people. It seems more logical to introduce a new fresh and modern console API that can be used for the next decade and retain the old conhost for legacy purposes.

Certainly, this wouldn't be the first time compatibility has been broken and trying to maintain backwards compatibility for something that old is impressive. I like what they did with IE; tons of stuff depended on IE, but they figured a way around it and introduced Edge, by keeping IE installed but for legacy support.

Considering the large number of devs that use the terminal on a daily basis (esp. linux devs to are married to the terminal), i think this would benefit more people than it would affect. Bash on windows is wonderful, but it's missing some polish and I think this would go a long way.

@be5invis
be5invis commented Jul 15, 2016 edited

@musm However, I am talking about NT console API, which is the basis of almost all Windows console applications and many of them are important basic industrial infrastructure, controlling important things like nuclear power plants...
The only possible solution is to create a new console API interface and a bug-to-bug compatibility layer.

@musm
musm commented Jul 30, 2016

I'm going to ref #417 as @Maximus5 suggested the fix
"@zadjii-msft This issue is not actually forces msft to implement truecolor in conhost. In fact, and instead, topic starter asked you to bypass ANSI to ConEmu, which already supports true-color in console.
Current implementation of console output in WSL prohibits ConEmu from processing any ANSI written by linux applications. That tight integration with kernel(?) is rather bad.
I believe, the issue have to be reopened, ConEmu doesn't limit user to 16-colors palette. And proposed approximation would look awful."

@be5invis

@musm @benhillis @zadjii-msft
What if users do not have installed ConEmu?
The true problem is that CONHOST already used all bits of the console buffer. And programs can directly modify it via C/C++ code.
A solution may be to provide another completely different console API (stream-based) and provide a compatibility layer to the existing one (blitting the buffer at 60fps?). Maybe you MS guys should have a serious talk with OSG.

@musm
musm commented Jul 30, 2016 edited

The suggestions is a hacky one I agree, but something hopefully sufficient for the group using conemu in the interim while MS cooks up (hopefully) a longer term fix for the future.
+1 for your idea of a new console API with a compatibility layer is far preferable of course

@be5invis

@musm
Hacky is necessary, because breaking existing console behavior may break important things like nuclear power plants. You definitely do not want that thing to happen.
If you want to connect your terminal to WSL, why do not just use SSH?

@musm
musm commented Jul 30, 2016

Could you clarify on how I would "If you want to connect your terminal to WSL, why do not just use SSH?"

@be5invis

SSH connect to localhost

@musm
musm commented Jul 30, 2016

Which problem would that solve? I mean SSHing from cmd/powershell from conemu to WSL, how does that fix things? Sorry I missed your point.

@be5invis

SSH with other tools like putty. That one should support full 256 colors

@DanielGGordon

I'm a bit confused here - I'm using Conemu - so is there a way for me to get 256 colors? Is there something I need to change? It looks like I still only have 16

@musm
musm commented Aug 4, 2016 edited

No, not possible

@DJviolin
DJviolin commented Aug 7, 2016 edited

@be5invis Then why not fork cmd and start a separate console project for Windows 10, for example cmd256? Wouldn't this be the best solution? Having full ANSI colors in Windows, but not touching the original cmd and finally vim will look as good as in xterm.

Or call it cmd-ansi or anything.

@be5invis
be5invis commented Aug 7, 2016

@DJviolin
As told by a MS employee from OSG, there was such a plan (simulates a graphics card -- yeah, really), but they choose to use the native console, perhaps for the future lxss-win32 interop.

@DJviolin
DJviolin commented Aug 7, 2016

MS people! Why not keep the lxss integration with cmd and having an alternate option to start bash.exe from cmd-ansi.exe (or any other windows commands)? Than you will have full color palette and lxss not tied to this console exclusively.

@be5invis
be5invis commented Aug 7, 2016

@DJviolin Manually start LXSS session and SSH to localhost?

@cheshrkat
cheshrkat commented Aug 9, 2016 edited

Reading through this thread - philosophically, if lxss gets caught up in worries about older windows shells, it's hard to understand the point of the project. Using the analogy from earlier... LXSS needs to be Edge vs. the IE8 of cmd/powershell.

Anyway to say what I actually came to say... +1 to fixing colours, it's not a trivial thing when it prevents things like vim colour schemes (my use case was setting up https://github.com/crusoexia/vim-monokai); and my usual PS1 didn't work because colours just disappeared. Way too easy to think of something like this as a nice-to-have, but in reality if LXSS doesn't replicate the basics of using bash on other platforms it's going to struggle to gain traction (and I really want it to gain traction :)).

@DJviolin
DJviolin commented Aug 9, 2016

We talking about a thing that already works in ConEmu with injection. None of us wants to change the cmd's default behaviour, we would be happy with an alternative console as well. Honestly, defining custom fonts, set font antialiasing, having 256 colors can be a deal breaker for many developers. We want VIM in it's full glory, inside the console, not in a heavy GUI application. Plus scrolling through a source file without any popping artifacts fast (like what VIM suffered in ConEmu).

@cowwoc
cowwoc commented Aug 13, 2016

Consider removing the duplicate tag, since this issue is no longer duplicate to #76.

@jquick
jquick commented Aug 20, 2016 edited

+1, ive had to resort to sshing into localhost with putty/mobaxterm to get support.

@zadjii-msft
Member

Woah woah woah, boy did this discussion get out of hand.
First off, a bit of a clarification, because this happens in every console issue thread and unfortunately it's not clearly explained anywhere as far as I know:
cmd.exe is different from conhost.exe.
cmd is a shell, much like bash, zsh, fish, or powershell.
conhost is the console window itself, like xterm or Terminal. All the shells above run within the conhost window, which displays the stdout and handles the stdin for your shell.

Now, conhost, is the place where the 16 color cap exists - the way the text is stored, it's only possible to keep 16 colors on any character. Fixing that is a fairly sizable amount of work, and one that we're definitely trying to work on. If it comes, it's going to be via the escape sequences mechanism already in place in conhost, not via extending/changing the console API.

I am going to de-dupe this, because #76 was solved by supporting the true RGB color sequences by matching to the nearest color in Win10 AU, but true RGB color support isn't quite there yet.

@zadjii-msft zadjii-msft removed the duplicate label Aug 24, 2016
@zadjii-msft zadjii-msft self-assigned this Aug 24, 2016
@zadjii-msft zadjii-msft added this to the Backlog milestone Aug 24, 2016
@be5invis

However, you still need to provide a compatibility layer for existing applications, and a switching mechanism for new applications to switch to new host.

???? iPhone

? 2016?8?25??13:34?SRGOM <notifications@github.commailto:notifications@github.com> ???

@zadjiihttps://github.com/zadjii is there a chance you guys will ever build a different conhost? There's no way (that I know of) to programmatically change font size and face in this. The storage (registry or shortcut file) isn't clear. Even lines aren't wrapped properly. It has come far from what it was before 10 but it still feels very primitive compared to Linux terminal emulators which are first class. I'd use Conemu were it not so slow to start. I did an experiment before anniversary updates and I think comemu has more throughput too.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/BashOnWindows/issues/345#issuecomment-242285136, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAOp2wfprR1pscz8xAMntfL6BCFXpuQ_ks5qjSlbgaJpZM4IcbDU.

@be5invis

@zadjii-msft Any plan with non-BMP, and advanced typography?
Will you release the APIs into win32, or make it WSL-only?

@zadjii-msft
Member

@be5invis I'm not really sure how much of the plan I'm allowed to talk about exactly - we wouldn't want a NMS style overpromise/under-deliver situation on our hands.
I can say that as far as any VT sequences we end up supporting are concerned, our goal has always been to avoid the expansion of the API surface. A few flags were added to the I/O modes (GetConsoleMode) in order to enable/disable the VT support, but otherwise any new features are hidden strictly behind VT sequences. For example, using an alternate buffer is supported by conhost, but only via VT sequences.
@SRGOM again, I can't talk too much about the plan, but I agree that it's frustrating that the properties and settings are VERY confusingly stashed away in shortcuts and registry entries. It's a pretty awful situation without a clear remedy. Though, I'm not sure what you mean by "the lines aren't wrapped correctly" - it's a little awkward occasionally in bash with bash redrawing it's prompt and input while we simultaneously try to reflow the buffer, but otherwise it's usually fine....

@be5invis

@zadjii-msft I can understand your consideration.
Line wrapping may relate to DBCS characters. Currently wide characters (like ๅฎ‰) cannot be typeset correctly under cp65001. IME is unavailable either.
And, yes, we all want an official PTY.

@zadjii-msft
Member

So, here's an update we know you've all been dying for:

https://blogs.msdn.microsoft.com/commandline/2016/09/22/24-bit-color-in-the-windows-console/

Full RGB color is supported as of yesterday's insider build.

@musm
musm commented Sep 22, 2016

yay! thank you !!!

@Maximus5

Is there a way to read 24bit color atributes? If not - implemented feature would be useless in ConEmu/cmder/ConsoleZ and other terminals.

@zadjii-msft zadjii-msft removed this from the Backlog milestone Sep 22, 2016
@be5invis

@maximus5 That is why we need an official API of pty, in Windows, win32 I mean.

@be5invis

@zadjii-msft How to use the 24-bit colors in Win32?

@zadjii-msft
Member

We're not planning on expanding the Win32 API surface for this feature. Every function we add to the Console API just creates another point of incompatibility with linux terminal applications.

If you want to use the new color support in your application, you'll have to use VT sequences. Support can be enabled with SetConsoleMode (see https://msdn.microsoft.com/en-us/library/windows/desktop/mt638032(v=vs.85).aspx#Output_Sequences).

The sequences to emit actual colors are as follows

Pm = 3 8 ; 2 ; Pr; Pg; Pb -> Set foreground color to the closest match in xterm's palette for the given RGB Pr/Pg/Pb.
Pm = 3 8 ; 5 ; Ps -> Set foreground color to Ps.
Pm = 4 8 ; 2 ; Pr; Pg; Pb -> Set background color to the closest match in xterm's palette for the given RGB Pr/Pg/Pb.
Pm = 4 8 ; 5 ; Ps -> Set background color to Ps.

(from http://invisible-island.net/xterm/ctlseqs/ctlseqs.html)

We're not going to be adding any API for reading the colors back out with 24-bit color, and as far as I know there are no VT sequences for reading the buffer or it's attributes.

@oblitum
oblitum commented Sep 23, 2016

This is nice. What's the expected time frame to get this in release form? I'm not relying on insider builds.

@JackofSpades707

Thank you!!! :D

@StoneCypher

@zadjii-msft - awesome, been waiting almost ten years for this. Like oblitum, I'm curious when people not on insider builds might get this.

@zadjii-msft
Member
zadjii-msft commented Oct 5, 2016 edited

@StoneCypher @oblitum I honestly can't say. I don't really know what the Windows timetable looks like for the next release. There have been two major updates in the year since Windows 10 shipped, so I'd say the next will also be sometime in the next ~6 months.

@StoneCypher

@zadjii-msft - oh. This isn't coming to non-insider builds until next windows major?

@oblitum
oblitum commented Oct 5, 2016

@zadjii-msft ok, thanks for the information.

@oblitum
oblitum commented Oct 5, 2016 edited

I think if I come back to windows I'm instead just gonna try some of these new distros for WSL that started to pop out + X, there's an Arch variant already, rolling releases FTW :)

@zadjii-msft
Member

@StoneCypher That's correct. There's no out-of-band release process for conhost unfortunately. The only way to get an update is with an OS update.

@DJviolin
DJviolin commented Oct 5, 2016

@oblitum Do you have a source for this? If it's possible Debian would be really great to use in WSL.

@oblitum
oblitum commented Oct 5, 2016

@DJviolin updated my comment.

@oblitum
oblitum commented Oct 5, 2016 edited

@DJviolin I've seen a lot of stuff these days about slackware/arch etc, with X apps etcs, some weird stuff, maybe there's something about Debian on the web, I don't recall whether I've seen something about that or not.

@bitcrazed
Collaborator

@oblitum Re. alt distro's: Note that they're unsupported at this time, but do feel free to give them a try: We don't do anything to prevent them from working; we're just focusing all our effort on delivering as solid an experience as possible on Ubuntu at this time.

@antitoxic

@zadjii-msft

We're not going to be adding any API for reading the colors back out with 24-bit color, and as far as I know there are no VT sequences for reading the buffer or it's attributes.

That means only powershell and cmd will support this, right? Which means I can't use any external manager for things like tabs, color schemes, tiling. Can you suggest a way to overcome this problem?

@be5invis

@zadjii-msft Yeah, could you please provide an example in C, calls WINAPI directly, and able to produce 24-bit color?

@zadjii-msft
Member

@antitoxic It means only conhost.exe console windows (the default for cmd.exe, powershell.exe, bash.exe, etc) will be able to get this behavior, correct. Changing the API surface is pretty much out of the question for this release.

And to be completely honest, 3rd party consoles aren't really a supported feature. They work because you are able to query the buffer and get it's contents (in lieu of proper pty-like behavior), but it's not something we go out of our way to support.

If a third party console REALLY wanted 24bit color, I'd imagine they might be able to disable VT processing mode of the console window, then parse the VT sequences themself. It's certainly not a great solution, but it might work (I honestly have no idea if it would). It would mean that the source console window and the 3rd party one would have vastly different buffer contents - which I'd imagine would cause a lot of difficulty.

@be5invis In terms of producing the colors:

bool EnableVTMode()
{
    // Set output mode to handle virtual terminal sequences
    HANDLE hOut = GetStdHandle(STD_OUTPUT_HANDLE);
    if (hOut == INVALID_HANDLE_VALUE)
    {
        return false;
    }

    DWORD dwMode = 0;
    if (!GetConsoleMode(hOut, &dwMode))
    {
        return false;
    }

    dwMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
    if (!SetConsoleMode(hOut, dwMode))
    {
        return false;
    }
    return true;
}
int __cdecl wmain(int argc, WCHAR* argv[])
{   
    argc; // unused
    argv; // unused
    //First, enable VT mode
    bool fSuccess = EnableVTMode();
    if (!fSuccess)
    {
        printf("Unable to enter VT processing mode. Quitting.\n");
        return -1;
    }
    int red = 0; // set these to whatever
    int green = 0;
    int blue = 0;
    printf("\x1b[38;2;%d;%d;%dm", red, green, blue); // produces RGB foreground
    printf("\x1b[48;2;%d;%d;%dm", red, green, blue); // produces RGB background

    int index = 0;
    printf("\x1b[38;5;%dm", index); // produces xterm color table index foreground
    printf("\x1b[48;5;%dm", index); // produces xterm color table index background
}

Most of that is copy-pasta from MSDN

@antitoxic

@zadjii-msft I'm just a dev that wants a better developer experience. I am the user. I don't want to get involved with command-line specifics to the point I know more about command line terminals, window managers and so on, than what I need to do in the first place.

I really just want tabs, color schemes, shortcuts. In order to be quick, to visually distinguish stuff, to do my job and move on.

Do you have any plans to develop this in further stage? If Microsoft will not, it seems only logical/fair to help others develop it. Colors, fonts, shortcuts, custom clipboard management - these are all fundamentals of good UX. I understand it's out of scope for cmd or powershell but please work together with the only people that do workable terminal GUI in windows. May be a separate group? A collaborative project?

@zadjii-msft
Member

@antitoxic I agree with a lot of that. We're developers too, and we want tabs too. But there are like, 4 developers working on the commandline. And as it turns out, the more you dig into the console story on windows, the crazier it gets (see miniksa's picture of the wall for reference).

We're getting there. We have plans. Patience, and we'll get you something you all like.

@StoneCypher

it's frustrating how every time someone says "i want x," other people come and decorate it with every other feature they've ever heard of

open new issues, guys. you know how issue trackers work.

@antitoxic
antitoxic commented Oct 19, 2016 edited

@zadjii-msft fantastic. I would love if there was more info on what's going on.

@StoneCypher Come on. I'm bringing up a problem that if this feature is dismissed all others I've listed won't be possible. So it makes sense to state this here, in this issue. Unless the plan is for Microsoft to release a super-beast tabbable configurable shortcutable terminal emulator app. None of us know that though. And as people talking here we must make decisions to whether contribute to this or focus elsewhere.

@zadjii-msft
Member

@antitoxic I'm gonna stop before I say too much.

@StoneCypher

@antitoxic - please stop creating alerts for me that have nothing to do with the issue I'm subscribed to, thank you

@nta nta added a commit to nta/clink that referenced this issue Nov 6, 2016
@nta nta let's be silly and turn clink into klink. 9efc5e1
@lilred
lilred commented Nov 21, 2016

@zadjii-msft FYI - most programs won't recognize 256 color support until $TERM is changed from xterm-color to xterm-256color.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment