Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

code converter not found (UTF-8 to UTF-8) (Encoding::ConverterNotFoundError) #2071

Closed
sempervictus opened this Issue Dec 3, 2012 · 1 comment

Comments

Projects
None yet
3 participants

This is a rather odd issue i found when attempting to use Snorby from https://github.com/SgtMalicious/snorby.

To reproduce:
build rbx-head current (as of this morning)
clone https://github.com/SgtMalicious/snorby.git
bundle install, you may need to commend out dm-mysql gem since this is a postgres fork
configure database.yml and snorby_config.yml - ensure you can connect to the postgres DB
run rake snorby:setup
observe the nonsense.

Stack trace for my encounter with the issue is at http://pastebin.com/ySjfj1HS
Temporary workaround is to run setup in MRI, rails app will run in RBX.

I ran across the same bug today and tried to debug as best I could. Here are my findings...

If an IO object has the same internal_encoding as external_encoding the IO.read_encode method ends up trying to do a conversion between identical encodings.

A simple way of reproducing this error is this:

require 'open3'
Encoding.default_internal = 'utf-8'
sin, sout, serr, thread = Open3.popen3("ls")
sout.internal_encoding # => #<Encoding:UTF-8>
sout.external_encoding # => => #<Encoding:UTF-8>
sout.read # error!

I noticed that I could fix the breakage by monkey patching IO.read_encode to return the original string in case internal was identical to external encoding:

class IO
  def self.read_encode(io, str)
    internal = io.internal_encoding
    external = io.external_encoding || Encoding.default_external

    if external.equal? Encoding::ASCII_8BIT and not internal
      str.force_encoding external
    elsif internal and external
      return str if internal == external # << THIS!
      ec = Encoding::Converter.new external, internal
      ec.convert str
    else
      str.force_encoding external
    end
  end
end

However it seems that the actual error in this case is that the IO returned from popen3 should not have sout.internal_encoding take on the value of Encoding.default_internal. If we run the same example in MRI we get the following results:

require 'open3'
Encoding.default_internal = 'utf-8'
sin, sout, serr, thread = Open3.popen3("ls")
sout.internal_encoding # => nil
sout.external_encoding # => => #<Encoding:UTF-8>

This bug is currently breaking my streamio-ffmpeg gem when run in a rails application since rails sets default encodings to utf8 on initialization.

@dbussink dbussink closed this in fd61b5f Feb 5, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment