Encoding error on OS X 10.11.4 Beta (15E33e)

[Freika]

╰─⠠⠵ rubocop -V
0.37.2 (using Parser 2.3.0.5, running on ruby 2.3.0 x86_64-darwin15)

╰─⠠⠵ ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin15]

Command rubocop --format html -o rubocop.html returned an error:

incompatible character encodings: UTF-8 and ASCII-8BIT
(erb):233:in `concat'
(erb):233:in `block (2 levels) in binding'
(erb):225:in `each'
(erb):225:in `block in binding'
(erb):219:in `each'
(erb):219:in `binding'
/Users/frey/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/erb.rb:864:in `eval'
/Users/frey/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/erb.rb:864:in `result'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/formatter/html_formatter.rb:57:in `render_html'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/formatter/html_formatter.rb:49:in `finished'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/formatter/formatter_set.rb:30:in `block (3 levels) in <class:FormatterSet>'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/formatter/formatter_set.rb:30:in `each'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/formatter/formatter_set.rb:30:in `block (2 levels) in <class:FormatterSet>'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/runner.rb:68:in `inspect_files'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/runner.rb:35:in `run'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/lib/rubocop/cli.rb:30:in `run'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/bin/rubocop:14:in `block in <top (required)>'
/Users/frey/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/Users/frey/.rvm/gems/ruby-2.3.0/gems/rubocop-0.37.2/bin/rubocop:13:in `<top (required)>'
/Users/frey/.rvm/gems/ruby-2.3.0/bin/rubocop:23:in `load'
/Users/frey/.rvm/gems/ruby-2.3.0/bin/rubocop:23:in `<main>'
/Users/frey/.rvm/gems/ruby-2.3.0/bin/ruby_executable_hooks:15:in `eval'
/Users/frey/.rvm/gems/ruby-2.3.0/bin/ruby_executable_hooks:15:in `<main>'

I found similar issue about Japanese encoding here: #1617

But my solution was to delete code from gem source:

lib/rubocop/formatter/html_formatter.rb:55

      def render_html
        context = ERBContext.new(files, summary)

        - template = File.read(TEMPLATE_PATH, encoding: Encoding::UTF_8)
        + template = File.read(TEMPLATE_PATH)
        erb = ERB.new(template, nil, '-')
        html = erb.result(context.binding)

        output.write html
      end

After deleting , encoding: Encoding::UTF_8 from 55 line command rubocop --format html -o rubocop.html created correct html file with scan results.

[moritzschepp]

It seems the encoding for the path of the template is the problem: I just had the same issue and I solved it by creating a file rubocop_encoding_fix.rb with

class MyFormatter < RuboCop::Formatter::HTMLFormatter
  TEMPLATE_PATH = RuboCop::Formatter::HTMLFormatter::TEMPLATE_PATH.encode('utf-8')
end

now I call rubocop with

rubocop -r ./rubocop_encoding_fix.rb -f MyFormatter

[moritzschepp]

Unfortunately, I was wrong and my fix doesn't actually fix anything :/ Also, I can't reproduce the problem reliably: When I make changes to the command line switches for rubocop, it works. When I run the command a second time, it throws the error.

[moritzschepp]

... and that lead me to the cache option. with -C false the problem is gone.

[cthulhu]

+1 for same issue

[jonas054]

I've finally been able to reproduce the problem. The JSON.load call here returns a structure of arrays, hashes and strings. It seems that the strings have different encodings in different environments. For me they're UTF-8 but for you, apparently, ASCII-8BIT.

Working on a solution.

[jonas054]

@Freika @moritzschepp @cthulhu Perhaps you can try out changes in my pull request?

[neofreko]

This still happens on rubocop 0.38.0. Same solution from @moritzschepp still applies: rubocop_encoding_fix.rb and run rubocop with -C false

[moritzschepp]

Indeed, with 0.38.0, if I switch on caching, I still get: incompatible character encodings: UTF-8 and ASCII-8BIT. I also deleted the cache under /tmp/1000/rubocop_cache. Then it runs fine once but the second time I get the above error.

[jonas054]

Then I'll just use @moritzschepp's solution even though I can't reproduce the problem and don't fully understand it. PR coming up.

[moritzschepp]

I just played around with rubocop's sources and made a test case that fails. I don't know how to fix it though, I assume the actual problem is how Parser::Source::Range is used but I'm really not sure. @jonas054, would you like a pull request for a failing test without fix? I also post it here, let me know if I can do anything:

it 'saves and reloads utf-8 encoded source code' do
  create_file 'example.rb', ['x = "a non ASCII character: ä"']
  cache.save(offenses)

  cache2 = described_class.new('example.rb', {}, config_store, cache_root)
  saved_offenses = cache2.load
  code = saved_offenses.first.location.source_line
  expect(code.encoding.to_s).to eq('UTF-8')
end

[jonas054]

@moritzschepp Your posting of source code here is enough. Thanks!
I hope to have some time to spare in the next few days to look at this.

[jonas054]

@moritzschepp I have tried to provoke the same problem that you displayed in your example by running rubocop with different encoding settings. It turns out that the source_line of the Range object we create from the cache always has UTF-8 encoding. The only slightly anomalous behavior I've been able to coax out of RuboCop is this:

$ ruby -EISO-8859-1:ISO-8859-1 ./bin/rubocop ex.rb 
Inspecting 1 file
W

Offenses:

ex.rb:3:1: W: Useless assignment to variable - x.
x = "a non ASCII character: �"
^

1 file inspected, 1 offense detected

So we get � instead of ä and that behavior is the same whether the offense comes from the cache or not. Anyway, I think it's as could be expected, and I don't see any real problem here.

If anyone has a better way to show that we do get unexpected behavior in some environments, let me know and I'll revisit the issue.