New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raw color codes in console output (Windows 7) #352

Closed
raphinesse opened this Issue Nov 20, 2013 · 26 comments

Comments

Projects
None yet
2 participants
@raphinesse
Contributor

raphinesse commented Nov 20, 2013

To clarify the issue, here's the start of the output of nanoc help on my machine

←[1m←[31mNAME←[0m←[0m
    ←[32mnanoc←[0m - nanoc, a static site compiler written in Ruby

←[1m←[31mUSAGE←[0m←[0m
    ←[32mnanoc←[0m command [options] [arguments]

nanoc --version

nanoc 3.6.5 (c) 2007-2013 Denis Defreyne.
Running ruby 1.9.3 (2013-06-27) on i386-mingw32 with RubyGems 1.8.24.

gem list

I suppose it's a nanoc bug since colors work fine when watching my site with guard.

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 20, 2013

Member

Do you have a Gemfile? Are you using bundle exec?

Member

ddfreyne commented Nov 20, 2013

Do you have a Gemfile? Are you using bundle exec?

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 20, 2013

Contributor

I do and I am

Contributor

raphinesse commented Nov 20, 2013

I do and I am

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 20, 2013

Member

Is win32console in the Gemfile (and Gemfile.lock)?

Member

ddfreyne commented Nov 20, 2013

Is win32console in the Gemfile (and Gemfile.lock)?

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 20, 2013

Contributor

It's not in the Gemfile but in Gemfile.lock (as a transitive dependency: guard-nanoc -> guard -> pry (0.9.12.2-x86-mingw32) -> win32console).
Anyhow, the result is the same for calling nanoc help without bundle exec.

Contributor

raphinesse commented Nov 20, 2013

It's not in the Gemfile but in Gemfile.lock (as a transitive dependency: guard-nanoc -> guard -> pry (0.9.12.2-x86-mingw32) -> win32console).
Anyhow, the result is the same for calling nanoc help without bundle exec.

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 20, 2013

Member

What is the value of RUBY_PLATFORM in Ruby?

Member

ddfreyne commented Nov 20, 2013

What is the value of RUBY_PLATFORM in Ruby?

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 20, 2013

Contributor

i386-mingw32

Contributor

raphinesse commented Nov 20, 2013

i386-mingw32

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 20, 2013

Member

Hmm, that is odd.

If you want to do some debugging, check out the definition of the method enable_ansi_colors?. When this returns false, color codes will be stripped from the output.

It seems that even with 'Win32/Console/ANSI' required, color support still doesn’t work. Can you verify that?

Member

ddfreyne commented Nov 20, 2013

Hmm, that is odd.

If you want to do some debugging, check out the definition of the method enable_ansi_colors?. When this returns false, color codes will be stripped from the output.

It seems that even with 'Win32/Console/ANSI' required, color support still doesn’t work. Can you verify that?

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 20, 2013

Contributor

I just tested compilation with an explicit require 'Win32/Console/ANSI' as well as require 'win32console' (via a file in lib) both to no avail.

I also tested color support in irb which works just fine. Odd indeed.

I'll do some debugging with the mentioned method, but I can't imagine what should possibly go wrong in there ^^

Contributor

raphinesse commented Nov 20, 2013

I just tested compilation with an explicit require 'Win32/Console/ANSI' as well as require 'win32console' (via a file in lib) both to no avail.

I also tested color support in irb which works just fine. Odd indeed.

I'll do some debugging with the mentioned method, but I can't imagine what should possibly go wrong in there ^^

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 20, 2013

Contributor

Ok, it keeps getting stranger. I just modified enable_ansi_colors? like that

def self.enable_ansi_colors?(io)
  return false if !io.tty?

  begin
    puts require 'Win32/Console/ANSI' if RUBY_PLATFORM =~ /mswin|mingw/
  rescue LoadError
    return false
  end

  return true
end

A bundle exec nanoc compile lead to this output. And the error message had colors! I can also confirm that I have colors in $stderr for other errors. o.O

Some things I noticed:

  • The puts only failed for false in $stderr
  • The error occured in the utf-8-cleaner
  • Captain! WeÔÇÖve been hit! was not correctly utf-8-cleaned. I noticed that before but thought: Yeah, same old story. Fuck Windows' console
  • I got a correctly escaped Captain! We've been hit! in the error I caused on purpose in a file in lib
  • %w( LC_ALL LC_CTYPE LANG ) are all empty on my machine (I guess that ought to be so on Windows)
  • Maybe any connection here is far fetched but: I just today noticed that the error I get with every request to the web server ran by nanoc view occurs in the utf-8 stream cleaner too (nanoc view works just fine apart from flooding my console)
Contributor

raphinesse commented Nov 20, 2013

Ok, it keeps getting stranger. I just modified enable_ansi_colors? like that

def self.enable_ansi_colors?(io)
  return false if !io.tty?

  begin
    puts require 'Win32/Console/ANSI' if RUBY_PLATFORM =~ /mswin|mingw/
  rescue LoadError
    return false
  end

  return true
end

A bundle exec nanoc compile lead to this output. And the error message had colors! I can also confirm that I have colors in $stderr for other errors. o.O

Some things I noticed:

  • The puts only failed for false in $stderr
  • The error occured in the utf-8-cleaner
  • Captain! WeÔÇÖve been hit! was not correctly utf-8-cleaned. I noticed that before but thought: Yeah, same old story. Fuck Windows' console
  • I got a correctly escaped Captain! We've been hit! in the error I caused on purpose in a file in lib
  • %w( LC_ALL LC_CTYPE LANG ) are all empty on my machine (I guess that ought to be so on Windows)
  • Maybe any connection here is far fetched but: I just today noticed that the error I get with every request to the web server ran by nanoc view occurs in the utf-8 stream cleaner too (nanoc view works just fine apart from flooding my console)
@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 20, 2013

Contributor

As a side note, this seems to check for UTF-8 support on Windows correctly:

def self.enable_utf8?(io)
  return true if !io.tty?

  if RUBY_PLATFORM =~ /mswin|mingw/
    `chcp`.include? "65001"
  else
    %w( LC_ALL LC_CTYPE LANG ).any? { |e| ENV[e] =~ /UTF/ }
  end
end

Results:

  • UTF-8 output on my console yay
  • Still no colors working nay
  • nanoc view causes errors in win32console now. I wonder what it's processing with those filters...
Contributor

raphinesse commented Nov 20, 2013

As a side note, this seems to check for UTF-8 support on Windows correctly:

def self.enable_utf8?(io)
  return true if !io.tty?

  if RUBY_PLATFORM =~ /mswin|mingw/
    `chcp`.include? "65001"
  else
    %w( LC_ALL LC_CTYPE LANG ).any? { |e| ENV[e] =~ /UTF/ }
  end
end

Results:

  • UTF-8 output on my console yay
  • Still no colors working nay
  • nanoc view causes errors in win32console now. I wonder what it's processing with those filters...
@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 20, 2013

Member

Will take a closer look later this week (it’s getting late here). In the mean time, can you run the nanoc tests locally? (clone the repository and type rake) I’d be interested in seeing whether anything odd turns up there.

Member

ddfreyne commented Nov 20, 2013

Will take a closer look later this week (it’s getting late here). In the mean time, can you run the nanoc tests locally? (clone the repository and type rake) I’d be interested in seeing whether anything odd turns up there.

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 20, 2013

Member

These tests are related to UTF-8 being enabled or not.

Oddly, I don’t have any tests that check for colors.

Member

ddfreyne commented Nov 20, 2013

These tests are related to UTF-8 being enabled or not.

Oddly, I don’t have any tests that check for colors.

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 20, 2013

Contributor

Sure, will do

Contributor

raphinesse commented Nov 20, 2013

Sure, will do

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 21, 2013

Contributor

No errors or failures that seem to be related to this problem. Here's the test output.

Had some work to do to get everything up and running on Windows, but the specifics are OT. If you're interested in less stuff exploding when trying to run nanoc's test suite on Windows, I can wrap up a PR 😉

Contributor

raphinesse commented Nov 21, 2013

No errors or failures that seem to be related to this problem. Here's the test output.

Had some work to do to get everything up and running on Windows, but the specifics are OT. If you're interested in less stuff exploding when trying to run nanoc's test suite on Windows, I can wrap up a PR 😉

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 21, 2013

Member

Fixes for Windows are certainly welcome!

I think we’ve stumbled on two separate issues here: first, nanoc not detecting UTF-8 properly (the chcp trick should fix that), and secondly, colors being printed as raw escape sequences.

The color issue might be related to the fact that nanoc wraps and replaces $stdout and $stderr. Although if that is the case, fixing this might be hard.

Member

ddfreyne commented Nov 21, 2013

Fixes for Windows are certainly welcome!

I think we’ve stumbled on two separate issues here: first, nanoc not detecting UTF-8 properly (the chcp trick should fix that), and secondly, colors being printed as raw escape sequences.

The color issue might be related to the fact that nanoc wraps and replaces $stdout and $stderr. Although if that is the case, fixing this might be hard.

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 24, 2013

Member

@raphinesse I’m guessing the test fixes on Windows didn’t solve the original color problem yet. Am I right?

Member

ddfreyne commented Nov 24, 2013

@raphinesse I’m guessing the test fixes on Windows didn’t solve the original color problem yet. Am I right?

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 24, 2013

Contributor

Yep, still an issue. To summarize:

  • When I test color codes with win32console in irb everything works fine
  • No colors in $stdout with nanoc, color codes output verbatim instead
  • I do have colors on $stderr with nanoc
  • The update, create, etc. messages of guard-nanoc are colored as well
  • However some messages of guard-nanoc show the same issue as above (those prefixed with [nanoc])

When working on the tests issue I noticed the same faulty behaviour with the messages from other tools.
For example bundle exec rake with missing bluecloth:

←[31mCould not find bluecloth-2.2.0 in any of the sources←[0m
←[33mRun `bundle install` to install missing gems.←[0m

So this might not be a nanoc specific issue after all. This starts to get rather frustrating 😩

Contributor

raphinesse commented Nov 24, 2013

Yep, still an issue. To summarize:

  • When I test color codes with win32console in irb everything works fine
  • No colors in $stdout with nanoc, color codes output verbatim instead
  • I do have colors on $stderr with nanoc
  • The update, create, etc. messages of guard-nanoc are colored as well
  • However some messages of guard-nanoc show the same issue as above (those prefixed with [nanoc])

When working on the tests issue I noticed the same faulty behaviour with the messages from other tools.
For example bundle exec rake with missing bluecloth:

←[31mCould not find bluecloth-2.2.0 in any of the sources←[0m
←[33mRun `bundle install` to install missing gems.←[0m

So this might not be a nanoc specific issue after all. This starts to get rather frustrating 😩

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 24, 2013

Member

This is very odd.

Win32console replaces $stdout and $stderr (see code), which means nanoc should behave properly, because it only ever uses $stdout and $stderr. ($stderr working properly is also very weird.)

Can you try to print $stdout.instance_eval("@stream"), and same for $stderr, after this line? $stdout and $stderr will be Nanoc::CLI:: CleaningStream instances. The wrapped streams (@stream) should both be Win32::Console::Ansi::IO instances.

Member

ddfreyne commented Nov 24, 2013

This is very odd.

Win32console replaces $stdout and $stderr (see code), which means nanoc should behave properly, because it only ever uses $stdout and $stderr. ($stderr working properly is also very weird.)

Can you try to print $stdout.instance_eval("@stream"), and same for $stderr, after this line? $stdout and $stderr will be Nanoc::CLI:: CleaningStream instances. The wrapped streams (@stream) should both be Win32::Console::Ansi::IO instances.

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 24, 2013

Contributor
  • p $stdout.instance_eval("@stream") = #<IO:<STDOUT>>
  • p $stderr.instance_eval("@stream") = #<Win32::Console::ANSI::IO:fd 2>

This makes it at least totally reasonable that it is not working as expected 😁

Contributor

raphinesse commented Nov 24, 2013

  • p $stdout.instance_eval("@stream") = #<IO:<STDOUT>>
  • p $stderr.instance_eval("@stream") = #<Win32::Console::ANSI::IO:fd 2>

This makes it at least totally reasonable that it is not working as expected 😁

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 24, 2013

Member

Hmm, very strange. It seems $stdout is modified (to point to STDOUT again, presumably). You’d need a way to figure out where that happens. Presumably not in nanoc itself, but perhaps in a dependency? I verified both nanoc and cri, and neither let $stdout point back to STDOUT.

Member

ddfreyne commented Nov 24, 2013

Hmm, very strange. It seems $stdout is modified (to point to STDOUT again, presumably). You’d need a way to figure out where that happens. Presumably not in nanoc itself, but perhaps in a dependency? I verified both nanoc and cri, and neither let $stdout point back to STDOUT.

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 25, 2013

Contributor

Well, after spending the better part of the evening with looking for a dependency that reset the value of $stdout it turned out that the problem is actually rather simple.

  1. An instance of CleaningStream is created, using the current value of $stdout, i.e. STDOUT
  2. enable_ansi_colors? is called which causes Win32/Console/ANSI to be required
  3. win32console replaces the values of $stdout and $stderr with it's wrappers
  4. $stdout is assigned the CleaningStream wrapping STDOUT
  5. Everything works fine for $stderr since it has already been wrapped

So to fix this we only need to require "win32console" before step 1. I'll leave the details up to you this time, since I've got no elegant solution ready right now 😁

Contributor

raphinesse commented Nov 25, 2013

Well, after spending the better part of the evening with looking for a dependency that reset the value of $stdout it turned out that the problem is actually rather simple.

  1. An instance of CleaningStream is created, using the current value of $stdout, i.e. STDOUT
  2. enable_ansi_colors? is called which causes Win32/Console/ANSI to be required
  3. win32console replaces the values of $stdout and $stderr with it's wrappers
  4. $stdout is assigned the CleaningStream wrapping STDOUT
  5. Everything works fine for $stderr since it has already been wrapped

So to fix this we only need to require "win32console" before step 1. I'll leave the details up to you this time, since I've got no elegant solution ready right now 😁

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 25, 2013

Member

Oh, yikes. That was totally non-obvious. Awesome work on the debugging 👍

I’ll whip up a PR and let you review it.

I’ll add you to the release notes for 3.6.7. How would you like to be credited? I usually use the full name, but GitHub alias is fine too.

Member

ddfreyne commented Nov 25, 2013

Oh, yikes. That was totally non-obvious. Awesome work on the debugging 👍

I’ll whip up a PR and let you review it.

I’ll add you to the release notes for 3.6.7. How would you like to be credited? I usually use the full name, but GitHub alias is fine too.

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 25, 2013

Member

@raphinesse See #356, a PR that hopefully fixes this problem. Let me know if this works for you.

Member

ddfreyne commented Nov 25, 2013

@raphinesse See #356, a PR that hopefully fixes this problem. Let me know if this works for you.

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 25, 2013

Contributor

I'd prefer my full name. Should be in the commits as author.

Contributor

raphinesse commented Nov 25, 2013

I'd prefer my full name. Should be in the commits as author.

@ddfreyne

This comment has been minimized.

Show comment
Hide comment
@ddfreyne

ddfreyne Nov 27, 2013

Member

PR merged, so I’ll close this issue. It’ll be in the upcoming 3.6.7 release.

Member

ddfreyne commented Nov 27, 2013

PR merged, so I’ll close this issue. It’ll be in the upcoming 3.6.7 release.

@ddfreyne ddfreyne closed this Nov 27, 2013

@raphinesse

This comment has been minimized.

Show comment
Hide comment
@raphinesse

raphinesse Nov 28, 2013

Contributor

👍

Contributor

raphinesse commented Nov 28, 2013

👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment