Raw color codes in console output (Windows 7)

[raphinesse]

To clarify the issue, here's the start of the output of nanoc help on my machine

←[1m←[31mNAME←[0m←[0m
    ←[32mnanoc←[0m - nanoc, a static site compiler written in Ruby

←[1m←[31mUSAGE←[0m←[0m
    ←[32mnanoc←[0m command [options] [arguments]

nanoc --version

nanoc 3.6.5 (c) 2007-2013 Denis Defreyne.
Running ruby 1.9.3 (2013-06-27) on i386-mingw32 with RubyGems 1.8.24.

gem list

I suppose it's a nanoc bug since colors work fine when watching my site with guard.

[denisdefreyne]

Do you have a Gemfile? Are you using bundle exec?

[raphinesse]

I do and I am

[denisdefreyne]

Is win32console in the Gemfile (and Gemfile.lock)?

[raphinesse]

It's not in the Gemfile but in Gemfile.lock (as a transitive dependency: guard-nanoc -> guard -> pry (0.9.12.2-x86-mingw32) -> win32console).
Anyhow, the result is the same for calling nanoc help without bundle exec.

[denisdefreyne]

What is the value of RUBY_PLATFORM in Ruby?

[raphinesse]

i386-mingw32

[denisdefreyne]

Hmm, that is odd.

If you want to do some debugging, check out the definition of the method enable_ansi_colors?. When this returns false, color codes will be stripped from the output.

It seems that even with 'Win32/Console/ANSI' required, color support still doesn’t work. Can you verify that?

[raphinesse]

I just tested compilation with an explicit require 'Win32/Console/ANSI' as well as require 'win32console' (via a file in lib) both to no avail.

I also tested color support in irb which works just fine. Odd indeed.

I'll do some debugging with the mentioned method, but I can't imagine what should possibly go wrong in there ^^

[raphinesse]

Ok, it keeps getting stranger. I just modified enable_ansi_colors? like that

def self.enable_ansi_colors?(io)
  return false if !io.tty?

  begin
    puts require 'Win32/Console/ANSI' if RUBY_PLATFORM =~ /mswin|mingw/
  rescue LoadError
    return false
  end

  return true
end

A bundle exec nanoc compile lead to this output. And the error message had colors! I can also confirm that I have colors in $stderr for other errors. o.O

Some things I noticed:

• The puts only failed for false in $stderr
• The error occured in the utf-8-cleaner
• Captain! WeÔÇÖve been hit! was not correctly utf-8-cleaned. I noticed that before but thought: Yeah, same old story. Fuck Windows' console
• I got a correctly escaped Captain! We've been hit! in the error I caused on purpose in a file in lib
• %w( LC_ALL LC_CTYPE LANG ) are all empty on my machine (I guess that ought to be so on Windows)
• Maybe any connection here is far fetched but: I just today noticed that the error I get with every request to the web server ran by nanoc view occurs in the utf-8 stream cleaner too (nanoc view works just fine apart from flooding my console)

[raphinesse]

As a side note, this seems to check for UTF-8 support on Windows correctly:

def self.enable_utf8?(io)
  return true if !io.tty?

  if RUBY_PLATFORM =~ /mswin|mingw/
    `chcp`.include? "65001"
  else
    %w( LC_ALL LC_CTYPE LANG ).any? { |e| ENV[e] =~ /UTF/ }
  end
end

Results:

• UTF-8 output on my console yay
• Still no colors working nay
• nanoc view causes errors in win32console now. I wonder what it's processing with those filters...

[denisdefreyne]

Will take a closer look later this week (it’s getting late here). In the mean time, can you run the nanoc tests locally? (clone the repository and type rake) I’d be interested in seeing whether anything odd turns up there.

[denisdefreyne]

These tests are related to UTF-8 being enabled or not.

Oddly, I don’t have any tests that check for colors.

[raphinesse]

Sure, will do

[raphinesse]

No errors or failures that seem to be related to this problem. Here's the test output.

Had some work to do to get everything up and running on Windows, but the specifics are OT. If you're interested in less stuff exploding when trying to run nanoc's test suite on Windows, I can wrap up a PR 😉

[denisdefreyne]

Fixes for Windows are certainly welcome!

I think we’ve stumbled on two separate issues here: first, nanoc not detecting UTF-8 properly (the chcp trick should fix that), and secondly, colors being printed as raw escape sequences.

The color issue might be related to the fact that nanoc wraps and replaces $stdout and $stderr. Although if that is the case, fixing this might be hard.

[denisdefreyne]

@raphinesse I’m guessing the test fixes on Windows didn’t solve the original color problem yet. Am I right?

[raphinesse]

Yep, still an issue. To summarize:

• When I test color codes with win32console in irb everything works fine
• No colors in $stdout with nanoc, color codes output verbatim instead
• I do have colors on $stderr with nanoc
• The update, create, etc. messages of guard-nanoc are colored as well
• However some messages of guard-nanoc show the same issue as above (those prefixed with [nanoc])

When working on the tests issue I noticed the same faulty behaviour with the messages from other tools.
For example bundle exec rake with missing bluecloth:

←[31mCould not find bluecloth-2.2.0 in any of the sources←[0m
←[33mRun `bundle install` to install missing gems.←[0m

So this might not be a nanoc specific issue after all. This starts to get rather frustrating 😩

[denisdefreyne]

This is very odd.

Win32console replaces $stdout and $stderr (see code), which means nanoc should behave properly, because it only ever uses $stdout and $stderr. ($stderr working properly is also very weird.)

Can you try to print $stdout.instance_eval("@stream"), and same for $stderr, after this line? $stdout and $stderr will be Nanoc::CLI:: CleaningStream instances. The wrapped streams (@stream) should both be Win32::Console::Ansi::IO instances.

[raphinesse]

• p $stdout.instance_eval("@stream") = #<IO:<STDOUT>>
• p $stderr.instance_eval("@stream") = #<Win32::Console::ANSI::IO:fd 2>

This makes it at least totally reasonable that it is not working as expected 😁

[denisdefreyne]

Hmm, very strange. It seems $stdout is modified (to point to STDOUT again, presumably). You’d need a way to figure out where that happens. Presumably not in nanoc itself, but perhaps in a dependency? I verified both nanoc and cri, and neither let $stdout point back to STDOUT.

[raphinesse]

Well, after spending the better part of the evening with looking for a dependency that reset the value of $stdout it turned out that the problem is actually rather simple.

1. An instance of CleaningStream is created, using the current value of $stdout, i.e. STDOUT
2. enable_ansi_colors? is called which causes Win32/Console/ANSI to be required
3. win32console replaces the values of $stdout and $stderr with it's wrappers
4. $stdout is assigned the CleaningStream wrapping STDOUT
5. Everything works fine for $stderr since it has already been wrapped

So to fix this we only need to require "win32console" before step 1. I'll leave the details up to you this time, since I've got no elegant solution ready right now 😁

[denisdefreyne]

Oh, yikes. That was totally non-obvious. Awesome work on the debugging 👍

I’ll whip up a PR and let you review it.

I’ll add you to the release notes for 3.6.7. How would you like to be credited? I usually use the full name, but GitHub alias is fine too.

[denisdefreyne]

@raphinesse See #356, a PR that hopefully fixes this problem. Let me know if this works for you.

[raphinesse]

I'd prefer my full name. Should be in the commits as author.

[denisdefreyne]

PR merged, so I’ll close this issue. It’ll be in the upcoming 3.6.7 release.

[raphinesse]

👍