Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

code converter not found (UTF-8 to UTF-8) (Encoding::ConverterNotFoundError) #2071

Closed
sempervictus opened this Issue · 1 comment

3 participants

@sempervictus

This is a rather odd issue i found when attempting to use Snorby from https://github.com/SgtMalicious/snorby.

To reproduce:
build rbx-head current (as of this morning)
clone https://github.com/SgtMalicious/snorby.git
bundle install, you may need to commend out dm-mysql gem since this is a postgres fork
configure database.yml and snorby_config.yml - ensure you can connect to the postgres DB
run rake snorby:setup
observe the nonsense.

Stack trace for my encounter with the issue is at http://pastebin.com/ySjfj1HS
Temporary workaround is to run setup in MRI, rails app will run in RBX.

@dbackeus

I ran across the same bug today and tried to debug as best I could. Here are my findings...

If an IO object has the same internal_encoding as external_encoding the IO.read_encode method ends up trying to do a conversion between identical encodings.

A simple way of reproducing this error is this:

require 'open3'
Encoding.default_internal = 'utf-8'
sin, sout, serr, thread = Open3.popen3("ls")
sout.internal_encoding # => #<Encoding:UTF-8>
sout.external_encoding # => => #<Encoding:UTF-8>
sout.read # error!

I noticed that I could fix the breakage by monkey patching IO.read_encode to return the original string in case internal was identical to external encoding:

class IO
  def self.read_encode(io, str)
    internal = io.internal_encoding
    external = io.external_encoding || Encoding.default_external

    if external.equal? Encoding::ASCII_8BIT and not internal
      str.force_encoding external
    elsif internal and external
      return str if internal == external # << THIS!
      ec = Encoding::Converter.new external, internal
      ec.convert str
    else
      str.force_encoding external
    end
  end
end

However it seems that the actual error in this case is that the IO returned from popen3 should not have sout.internal_encoding take on the value of Encoding.default_internal. If we run the same example in MRI we get the following results:

require 'open3'
Encoding.default_internal = 'utf-8'
sin, sout, serr, thread = Open3.popen3("ls")
sout.internal_encoding # => nil
sout.external_encoding # => => #<Encoding:UTF-8>

This bug is currently breaking my streamio-ffmpeg gem when run in a rails application since rails sets default encodings to utf8 on initialization.

@dbussink dbussink closed this in fd61b5f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.