Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ANSI colors no longer render in cmd.exe #1173

Closed
cowwoc opened this issue Oct 7, 2016 · 39 comments
Closed

ANSI colors no longer render in cmd.exe #1173

cowwoc opened this issue Oct 7, 2016 · 39 comments

Comments

@cowwoc
Copy link

cowwoc commented Oct 7, 2016

Please use the following bug reporting template to help produce actionable and reproducible issues. Please try to ensure that the reproduction is minimal so that the team can go through more bugs!

symfony/symfony#19520 confirms that other people are seeing the same problem.

  • Expected results
    ANSI colors should render in cmd.exe when applications output to stdout/stderr.
  • Actual results (with terminal output if applicable)
    Raw ANSI codes are seen. No colors are rendered.
  • Your Windows build number
    10.0.14393
  • Steps / All commands required to reproduce the error from a brand new installation
  • Run this Java testcase:
public class Test {

    public static void main(String[] args) {
        System.out.println("\033[37;1mwhite\033[0m");
    }
}
  1. You will get this output: �[37;1mwhite�[0m
  2. Redirect the application stdout to foo.log and run type foo.log
  3. Notice that the output now reads white using white text

This is reproducible 100% of the time.

@fpqc
Copy link

fpqc commented Oct 7, 2016

10.0.14393 is not the most current release. This should work totally fine in 14942 (latest). I think the processing used to be done in bash.exe. It has now been moved directly into conhost.exe as a console mode (enable rendering of terminal escape sequences)

@cowwoc
Copy link
Author

cowwoc commented Oct 7, 2016

@fpqc https://support.microsoft.com/en-ca/help/12387/windows-10-update-history disagrees with your assertion. I tried Windows Update locally (Windows 10 Pro) and it says I have the latest version.

I assume you're talking about an insider release?

Also, did you actually try running my testcase in 14942?

@fpqc
Copy link

fpqc commented Oct 7, 2016

Yes it's talking about an insider release, and yes I've attempted it.
2016-10-07 2

I'm on 14936. 14942 just came out like an hour ago and I haven't had an opportunity to install it yet.

Anyway, I suspect that the problem with Symfony is that the executable is not set to output to the command prompt in terminal-enabled mode. This is already documented in the posts about 24-bit color, linking to an msdn article iirc.

Link the Symfony people to this article: https://msdn.microsoft.com/en-us/library/windows/desktop/mt638032(v=vs.85).aspx

You need to enable terminal processing mode in the setconsolemode flags.

I know that this documentation applies to the insider builds, not sure if it works in the official ones.

@Manouchehri
Copy link
Contributor

@cowwoc For reporting bugs, you really need to be on the fast ring insider builds. The development here is done way faster than the Windows stable builds can keep up.

@bitcrazed
Copy link
Contributor

bitcrazed commented Oct 10, 2016

Closing:

24-bit color color support was first introduced in Win10 Insiders Update 14931, and isn't available in the Win10 Anniversary Update.

To exercise this feature, you'll need to join the Windows Insiders program and update your PC to install Fast-ring releases. Alternatively, you can opt for slow-ring updates and wait a few weeks for this feature to arrive.

@cowwoc
Copy link
Author

cowwoc commented Oct 14, 2016

Please reopen this issue. I am seeing the same bug in update 14946.

image

@fpqc
Copy link

fpqc commented Oct 15, 2016

@cowwoc you have to set the console mode for printf.exe to enable virtual terminal processing, probably by recompiling it. GNUWin32 needs to be recompiled with this console setting enabled for each utility.

It looks like the project is unmaintained. If it matters to you, fork the project, enable this mode as in the documentation (includes two examples), and you should be good to go.

Is there any reason you need gnuwin32? Can't you use the cygwin versions?

Another more recent project is this:

https://github.com/bmatzelle/gow

Last updated 8months ago vs 3 years ago.

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@fpqc I don't really care about printf. I was just using it because you were doing the same in your testcase. The only thing I care about is getting this to work with Java.

  1. I tried the printf version you mentioned and got the same output (no colors).
  2. Can you please run my testcase on your end?
  3. Why isn't ANSI support enabled by default (without having to enable ENABLE_VIRTUAL_TERMINAL_INPUT?
  4. Why were ANSI colors working for me (and others) in past releases and now they no longer do? This seems like a clear regression.
  5. How do I find out what console mode a process is running with? I want to reverse-engineer what Java is using.

@fpqc
Copy link

fpqc commented Oct 16, 2016

@cowwoc No idea, can you set console modes for java.exe? Request that Sun enables it in jvm?

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@fpqc There is no way to set console modes in Java without using JNI (which is not a realistic option for my use-case). It is also unlikely Sun/Oracle would enable this in the near future.

I think it would help if someone from Microsoft address point 4: why were ANSI colors working in 10.0.10586 but got broken in version 10.0.14393? Something must have changed and I'm not convinced yet this was intentional.

@fpqc
Copy link

fpqc commented Oct 16, 2016

@cowwoc I dunno. I can't help you, and sorry, but the devs on this Issues page can't help you either. I know Rich Turner (Senior PM on the console enhancement project) has a special work-only Twitter account and maybe you can ask him why the default got changed (vis-a-vis escape sequence processing) and ask if there is a way to turn it on for a console java application.

And if not, report it as a "bug" to Oracle and hope they fix it?

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@fpqc Perhaps @zadjii-msft could help? He is the one who marked #1173 as fixed.

@fpqc
Copy link

fpqc commented Oct 16, 2016

@cowwoc I dunno, Mr. Turner is kind of his boss, I think. I think you can turn it on maybe with an environment variable, can't you set those with Java?

Edit: The solution suggested to setconsolemode on Windows for Java is to make a launcher wrapper that just does setconsolemode and then uses java.exe.

@fpqc
Copy link

fpqc commented Oct 16, 2016

@cowwoc Check out the top answer here:

http://stackoverflow.com/questions/12404258/truly-interactive-command-line

There is a project called Java Native Access (JNA) that lets you call SetConsoleMode from Java.
https://github.com/java-native-access/jna

It is currently maintained and will let you call SetConsoleMode in the Windows API.

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@fpqc JNA uses JNI under the hood, so this isn't an option. The only way to interact with native code is through JNI.

@fpqc
Copy link

fpqc commented Oct 16, 2016

@cowwoc I guess then you've gotta ask the Java people to set it or write a small wrapper program. I assume MS turned it off by default for a reason.

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@fpqc They simply mean that you don't need any extra JNI code. Quoting https://github.com/java-native-access/jna:

JNA uses a small JNI library stub to dynamically invoke native code

@fpqc
Copy link

fpqc commented Oct 16, 2016

@cowwoc Yeah that's why I deleted my post. I continued reading it.

What is this super-restrictive use case you have with no JNI at all? Maybe you can be a hero and write a generic mini-loader for Java, then throw it up on github?

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@bitcrazed Seeing as you closed this issue, can you please clarify why ANSI colors were working in 10.0.10586 but no longer work in 10.0.14393?

@fpqc It's a long story, but the jist of it is that JNI is a deployment nightmare. You have to deploy different JAR files for different platforms. On platforms that use native code, you have to unpack the libraries before loading them. Maven (the prominent build system for Java) handles building and packaging native libraries very poorly. And so on...

The minute I introduce native code to my library I am pushing this complexity on users of my library (and their users, transitively). Considering what my library does (https://bitbucket.org/cowwoc/requirements/) this would be a major deterrent to its use.

I seriously hope there is a way to use this functionality without invoking SetConsoleMode(). I guess we'll have to wait for Monday to find out.

@fpqc
Copy link

fpqc commented Oct 16, 2016

@cowwoc Isn't that the point of JNA, no need to extract all those native libraries? You just have the single dll, and that lets you make calls to the Windows API, just a single quick branch on initialization?

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@fpqc I appreciate you trying to help, but whether it's one DLL or many, you still end up with the deployment problems I mentioned.

@Manouchehri
Copy link
Contributor

Manouchehri commented Oct 16, 2016 via email

@fpqc
Copy link

fpqc commented Oct 16, 2016

@Manouchehri That's not the problem, he wants his java application to run on unpatched Windows conhost in stock configuration and default settings with color-escape sequence processing enabled.

Apparently this was changed from the default in Anniversary update, and he lost compatibility.

@cowwoc
Copy link
Author

cowwoc commented Oct 16, 2016

@Manouchehri As fpqc mentioned, this works in conemu, but I want to get my Java application to output ANSI colors on an unpatched Windows conhost using the default configuration.

@Manouchehri
Copy link
Contributor

It works fine in cmd.exe on my end. Cannot reproduce.

image

Microsoft Windows [Version 10.0.14946]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\david>bash
dave@universal:/mnt/c/Users/david$ printf "\033[37;1mwhite\033[0m\n"
white

@fpqc
Copy link

fpqc commented Oct 16, 2016

@Manouchehri It depends on the console application. Bash has that mode set, but Java doesnt.

@zadjii-msft
Copy link
Member

Oh my, you leave the office for one weekend, and this happens :P

There's a lot going on in here, so let me try to clarify: The windows console only supports VT sequences of ANY kind with the ConsoleMode set to enable Virtual Terminal support. See This documentation on SetConsoleMode. Specifically, you need to turn on the output mode flag ENABLE_VIRTUAL_TERMINAL_PROCESSING.

By default, cmd.exe enables VT mode for itself. Now here's the confusion:

  • In 10.0.10586 (or 1511 or TH2), it left this mode on when calling child applications. This meant that anything you launched from cmd would get the new behavior for free. Of course, we found that unsurprisingly broke compatibility, so we had to change that. So...
  • In 10.0.14393 (Anniversary update, or greater), cmd uses VT mode, but then reverts it for launching child processes. So if you wrote a console app that uses VT sequences after October 2015, then tried running it on an AU system, you'll find that the sequences don't work anymore. You'll have to manually turn on VT support with SetConsoleMode yourself. IMPORTANT cmd also wipes out the changes a child makes to the mode. So you can't call enable_vt.exe & myapp.exe and have the VT mode from enable_vt.exe pollute the mode of myapp.exe

To further confuse, bash.exe turns on VT mode itself, because obviously, it needs that.

I can't really speak about the Java issue at hand, it's been far too long since I've used Java, and I don't really have any experience calling Win32 API's from java. I did end up writing a snippet of code that enables VT support on Windows in python, so it can be done.

In conclusion: I'm going to leave this issue closed, as this is the proper, expected behavior. Confusing, but correct.

Also, in terms of hierarchy, @bitcrazed is the PM for the console, while I'm one of the devs

@WASasquatch
Copy link

WASasquatch commented May 11, 2017

Having this issue on the latest Windows 10 Creators. Was working fine (for all my Minecraft Servers) now, the MC servers will error out with after my PC installed the Creators update.

Question: Is microsoft simply trying to interfere with people that script out their windows to run commercial systems and services? By doing something as simple as not including the flag/ability in new releases they effectively break all sorts of services people may have deployed at every update, no doubt, because it's not their Server editions.

    2017-05-11 10:50:34,157 WARN Unable to instantiate org.fusesource.jansi.WindowsAnsiOutputStream
    [ERROR] Terminal initialization failed; falling back to unsupported
    [STDERR]: java.lang.NoClassDefFoundError: Could not initialize class org.fusesource.jansi.internal.Kernel32
    [STDERR]:       at org.fusesource.jansi.internal.WindowsSupport.getConsoleMode(WindowsSupport.java:50)
    [STDERR]:       at org.bukkit.craftbukkit.libs.jline.WindowsTerminal.getConsoleMode(WindowsTerminal.java:204)
    [STDERR]:       at org.bukkit.craftbukkit.libs.jline.WindowsTerminal.init(WindowsTerminal.java:82)
    [STDERR]:       at org.bukkit.craftbukkit.libs.jline.TerminalFactory.create(TerminalFactory.java:101)
    [STDERR]:       at org.bukkit.craftbukkit.libs.jline.TerminalFactory.get(TerminalFactory.java:158)
    [STDERR]:       at org.bukkit.craftbukkit.libs.jline.console.ConsoleReader.<init>(ConsoleReader.java:229)
    [STDERR]:       at org.bukkit.craftbukkit.libs.jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    [STDERR]:       at org.bukkit.craftbukkit.libs.jline.console.ConsoleReader.<init>(ConsoleReader.java:213)
    [STDERR]:       at net.minecraft.server.v1_11_R1.MinecraftServer.<init>(MinecraftServer.java:149)
    [STDERR]:       at net.minecraft.server.v1_11_R1.DedicatedServer.<init>(DedicatedServer.java:48)
    [STDERR]:       at net.minecraft.server.v1_11_R1.MinecraftServer.main(MinecraftServer.java:1006)
    [STDERR]:       at org.bukkit.craftbukkit.Main.main(Main.java:226)
    [STDERR]:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    [STDERR]:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    [STDERR]:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    [STDERR]:       at java.lang.reflect.Method.invoke(Unknown Source)
    [STDERR]:       at com.destroystokyo.paperclip.Paperclip.main(Paperclip.java:239)

    [10:50:38 INFO]: [STDERR]: java.lang.NoClassDefFoundError: Could not initialize class org.fusesource.jansi.internal.Kernel32
    [10:50:38 INFO]: [STDERR]:      at org.fusesource.jansi.internal.WindowsSupport.getConsoleMode(WindowsSupport.java:50)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.libs.jline.WindowsTerminal.getConsoleMode(WindowsTerminal.java:204)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.libs.jline.WindowsTerminal.init(WindowsTerminal.java:82)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.libs.jline.TerminalFactory.create(TerminalFactory.java:101)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.libs.jline.TerminalFactory.get(TerminalFactory.java:158)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.libs.jline.console.ConsoleReader.<init>(ConsoleReader.java:229)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.libs.jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.libs.jline.console.ConsoleReader.<init>(ConsoleReader.java:213)
    [10:50:38 INFO]: [STDERR]:      at net.minecraft.server.v1_11_R1.MinecraftServer.<init>(MinecraftServer.java:149)
    [10:50:38 INFO]: [STDERR]:      at net.minecraft.server.v1_11_R1.DedicatedServer.<init>(DedicatedServer.java:48)
    [10:50:38 INFO]: [STDERR]:      at net.minecraft.server.v1_11_R1.MinecraftServer.main(MinecraftServer.java:1006)
    [10:50:38 INFO]: [STDERR]:      at org.bukkit.craftbukkit.Main.main(Main.java:226)
    [10:50:38 INFO]: [STDERR]:      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    [10:50:38 INFO]: [STDERR]:      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    [10:50:38 INFO]: [STDERR]:      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    [10:50:38 INFO]: [STDERR]:      at java.lang.reflect.Method.invoke(Unknown Source)
    [10:50:38 INFO]: [STDERR]:      at com.destroystokyo.paperclip.Paperclip.main(Paperclip.java:239)

@cowwoc
Copy link
Author

cowwoc commented May 11, 2017

@WASasquatch The error you pasted has nothing to do with Windows. Please google NoClassDefFoundError and discuss with the jansi authors if necessary.

@WASasquatch
Copy link

WASasquatch commented May 11, 2017

Same issue with just printing color codes via any program via bash. I was using this as a program-breaking example.

@cowwoc
Copy link
Author

cowwoc commented May 11, 2017

@WASasquatch Can you reproduce this issue without using jansi? If so, can you provide a testcase others can run?

@zadjii-msft
Copy link
Member

@WASasquatch That definitely looks like a java problem, not a conhost or WSL problem to me.

Do you have a specific, simple example of VT sequences not working in WSL? They're a pretty heavily tested part of the system, I doubt they would have broken:

image
worksonmymachine.png

@glenn-slayden
Copy link

glenn-slayden commented Mar 6, 2018

As noted here and here, the ANSI VT can be enabled for SetConsoleMode-oblivious apps, without having to re-compile them, by changing the following registry key.

[HKEY_CURRENT_USER\Console]
"VirtualTerminalLevel"=dword:00000001

This changes the system-wide default for ANSI VT processing from opt-in to opt-out, so it will affect any/all apps that don't express a preference. Works for Windows 10.16257 (and later) (and possibly earlier).

Given the very helpful information in a comment on this page by @zadjii-msft, this registry setting would appear to be the only workaround for apps that can't be re-compiled to include a call to SetConsoleMode. Because yes, cmd.exe does respect this setting while it does the fiddling described in that remark.

HKEY_CURRENT_USER\ConsoleVirtualTerminalLevel -- Registry setting to enable ANSI sequence processing on Windows 10.16257 and later

@E3V3A
Copy link

E3V3A commented Dec 11, 2018

@glenn-slayden

This changes the system-wide default for ANSI VT processing from opt-in to opt-out,..
...this registry setting would appear to be the only workaround for apps that can't be re-compiled to include a call to SetConsoleMode.

Can this be understood as the setting overriding any Get/SetConsoleMode operations?

I'm asking because I get very funny results running on W8.1 where reading the ENABLE_VIRTUAL_TERMINAL_PROCESSING. While I have enabled the registry item above, using a MinGW compiled ConoutMode.exe, tell me it is OFF, while CMD tests seem to indicate that it is ON (ok), while PowerShell is behaving irregularly across versions and returns different results, most of the time no color (for obvious reasons), but returning the escape sequences, so when run via alternative conhost terminal it works...sometimes.

@zadjii-msft
Copy link
Member

zadjii-msft commented Dec 11, 2018

ENABLE_VIRTUAL_TERMINAL_PROCESSING Is only available on Windows 10 and up, in the v2 console.

The v1 console (the only version of the console available on pre-Windows 10) probably doesn't ignore flags that it doesn't support, so you could probably SetConsoleMode with ENABLE_VIRTUAL_TERMINAL_PROCESSING and then have it roundtrip to your next call to GetConsoleMode, but that doesn't imply that it's actually enabled. I know that the v2 console will ignore flags it doesn't understand, but I'm not positive about the v1.

Edit: To add, VirtualTerminalLevel will additionally do nothing on the v1 console.

@ghost
Copy link

ghost commented Feb 2, 2019

Just a note to anybody using Python.

You can enable ANSI using this:

import ctypes

kernel32 = ctypes.windll.kernel32
kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7)

@koryovip
Copy link

jna is ok too

@quantum-hash
Copy link

Use this to enable terminal colors by updating the registry using Java :

public class ENABLE_TERMINAL_COLORS_JAVA {
  public static void main(String[] args)
  {
    try { // TO ENABLE
      Process process = Runtime.getRuntime().exec("reg add HKEY_CURRENT_USER\\Console /v VirtualTerminalLevel /t REG_DWORD /d 0x00000001 /f");
    }
    // TO DISABLE (Put it back to its original value) --> try {Process process = Runtime.getRuntime().exec("reg add HKEY_CURRENT_USER\\Console /v VirtualTerminalLevel /t REG_DWORD /d 0x00000000 /f");}
    catch (Exception e) {
      e.printStackTrace(); // Catch and display exception
    }
  }
} 

Click to follow - GitHub Followers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants