New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encoding::ConverterNotFoundError when using Encoding.default_internal #3647

Closed
rykov opened this Issue Feb 5, 2016 · 11 comments

Comments

Projects
None yet
5 participants
@rykov

rykov commented Feb 5, 2016

jRuby versions affected: 9.0.4.0, 9.0.5.0 and possibly earlier ones that I didn't test.

jRuby throws an Encoding::ConverterNotFoundError error when trying to open a BOM:UTF-8 file after setting Encoding.default_internal. This does not mirror the behavior of MRI.

I was able to narrow it down to the following reproduction snippet:

# example.rb
Encoding.default_internal = 'utf-8'
File.open('empty-file', "rb:bom|utf-8") { |f| f.read }
puts "OK!"

Here are the erroneous results:

$ touch empty-file
$ rbenv shell jruby-9.0.4.0 
$ jruby ./example.rb 
Encoding::ConverterNotFoundError: code converter not found (UTF-8 to UTF-8)
...
$ rbenv shell jruby-9.0.5.0 
$ jruby ./example.rb 
Encoding::ConverterNotFoundError: code converter not found (UTF-8 to UTF-8)
...

And here's the same script working with MRI 2.2.4, 2.3.0, and jRuby 1.7.24:

$ rbenv shell 2.2.4 
$ ruby ./example.rb 
OK!
$ rbenv shell 2.3.0 
$ ruby ./example.rb 
OK!
$ rbenv shell jruby-1.7.24 
$ jruby ./example.rb 
OK!

My environment:

$ uname -a
Darwin MBA.local 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64
$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
@rykov

This comment has been minimized.

Show comment
Hide comment
@rykov

rykov Feb 5, 2016

I went back and installed MRI 2.2.3 to match jRuby 9.0.5.0 - it also works:

$ rbenv shell 2.2.3 
$ ruby ./example.rb 
OK!

rykov commented Feb 5, 2016

I went back and installed MRI 2.2.3 to match jRuby 9.0.5.0 - it also works:

$ rbenv shell 2.2.3 
$ ruby ./example.rb 
OK!
@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Feb 5, 2016

Member

I am not positive this is a parser issue but something is amiss. Is this also an issue if that file has a bom and some contents?

Member

enebo commented Feb 5, 2016

I am not positive this is a parser issue but something is amiss. Is this also an issue if that file has a bom and some contents?

@rykov

This comment has been minimized.

Show comment
Hide comment
@rykov

rykov Feb 6, 2016

Works fine with bom (with or without other content):

$ printf '\xFF\xFE' > empty-file
$ jruby ./example.rb 
OK!
$ printf '\xFF\xFE hello' > empty-file
$ jruby ./example.rb 
OK!

rykov commented Feb 6, 2016

Works fine with bom (with or without other content):

$ printf '\xFF\xFE' > empty-file
$ jruby ./example.rb 
OK!
$ printf '\xFF\xFE hello' > empty-file
$ jruby ./example.rb 
OK!
@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Feb 13, 2016

Member

Looking into this.

Member

headius commented Feb 13, 2016

Looking into this.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Feb 13, 2016

Member

Minor bit of code missing from our BOM stripping logic, to clear the external encoding if no BOM could be found. Unexcluded a test from CRuby that's now passing. Yay!

Member

headius commented Feb 13, 2016

Minor bit of code missing from our BOM stripping logic, to clear the external encoding if no BOM could be found. Unexcluded a test from CRuby that's now passing. Yay!

@oozzal

This comment has been minimized.

Show comment
Hide comment

oozzal commented Feb 16, 2016

👍

@pobocks

This comment has been minimized.

Show comment
Hide comment
@pobocks

pobocks Feb 23, 2016

I have jruby-9.1.0.0-SNAPSHOT (JRUBY_REVISION=cea55fb) from RVM, failing with this error, without there being a BOM in sight, as far as I can tell.

Encoding::ConverterNotFoundError: code converter not found (UTF-8 to UTF-8)
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:59:in `ignoring_nonexistent_files'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:14:in `block in load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:46:in `block in with'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:45:in `with'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:13:in `load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-rails-2.1.0/lib/dotenv/rails.rb:29:in `load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-rails-2.1.0/lib/dotenv/rails.rb:44:in `load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-rails-2.1.0/lib/dotenv/rails.rb:20:in `block in Railtie'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:36:in `execute_hook'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:45:in `block in run_load_hooks'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:44:in `run_load_hooks'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/railties-4.2.4/lib/rails/application.rb:148:in `run_load_hooks!'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/railties-4.2.4/lib/rails/application.rb:96:in `instance'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/railties-4.2.4/lib/rails/railtie.rb:123:in `config'
/home/aspace_proc/aspace_processor/releases/20160223175950/config/application.rb:24:in `<class:Application>'
/home/aspace_proc/aspace_processor/releases/20160223175950/config/application.rb:10:in `<module:AspaceProcessor>'
/home/aspace_proc/aspace_processor/releases/20160223175950/config/application.rb:9:in `<top>'
/home/aspace_proc/aspace_processor/releases/20160223175950/Rakefile:1:in `<top>'
/home/aspace_proc/aspace_processor/releases/20160223175950/Rakefile:4:in `(root)'
(See full trace by running task with --trace)

pobocks commented Feb 23, 2016

I have jruby-9.1.0.0-SNAPSHOT (JRUBY_REVISION=cea55fb) from RVM, failing with this error, without there being a BOM in sight, as far as I can tell.

Encoding::ConverterNotFoundError: code converter not found (UTF-8 to UTF-8)
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:59:in `ignoring_nonexistent_files'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:14:in `block in load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:46:in `block in with'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:45:in `with'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-2.1.0/lib/dotenv.rb:13:in `load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-rails-2.1.0/lib/dotenv/rails.rb:29:in `load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-rails-2.1.0/lib/dotenv/rails.rb:44:in `load'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/dotenv-rails-2.1.0/lib/dotenv/rails.rb:20:in `block in Railtie'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:36:in `execute_hook'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:45:in `block in run_load_hooks'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:44:in `run_load_hooks'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/railties-4.2.4/lib/rails/application.rb:148:in `run_load_hooks!'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/railties-4.2.4/lib/rails/application.rb:96:in `instance'
/home/aspace_proc/aspace_processor/shared/bundle/jruby/2.2.0/gems/railties-4.2.4/lib/rails/railtie.rb:123:in `config'
/home/aspace_proc/aspace_processor/releases/20160223175950/config/application.rb:24:in `<class:Application>'
/home/aspace_proc/aspace_processor/releases/20160223175950/config/application.rb:10:in `<module:AspaceProcessor>'
/home/aspace_proc/aspace_processor/releases/20160223175950/config/application.rb:9:in `<top>'
/home/aspace_proc/aspace_processor/releases/20160223175950/Rakefile:1:in `<top>'
/home/aspace_proc/aspace_processor/releases/20160223175950/Rakefile:4:in `(root)'
(See full trace by running task with --trace)
@rykov

This comment has been minimized.

Show comment
Hide comment
@rykov

rykov Feb 23, 2016

@pobocks Can you try running the example code in the original post to confirm it's the same bug

rykov commented Feb 23, 2016

@pobocks Can you try running the example code in the original post to confirm it's the same bug

@pobocks

This comment has been minimized.

Show comment
Hide comment
@pobocks

pobocks Feb 23, 2016

Will do - putting a BOM in my .env allowed the process to continue, so it
seems likely to be the same kind of thing, for sure.
On Feb 23, 2016 1:33 PM, "Michael Rykov" notifications@github.com wrote:

@pobocks https://github.com/pobocks Can you try running the example
code in the original post to confirm it's the same bug


Reply to this email directly or view it on GitHub
#3647 (comment).

pobocks commented Feb 23, 2016

Will do - putting a BOM in my .env allowed the process to continue, so it
seems likely to be the same kind of thing, for sure.
On Feb 23, 2016 1:33 PM, "Michael Rykov" notifications@github.com wrote:

@pobocks https://github.com/pobocks Can you try running the example
code in the original post to confirm it's the same bug


Reply to this email directly or view it on GitHub
#3647 (comment).

@pobocks

This comment has been minimized.

Show comment
Hide comment
@pobocks

pobocks Feb 23, 2016

Confirmed:

$ touch empty-file
$ rvm jruby-head do ruby example.rb 
Encoding::ConverterNotFoundError: code converter not found (UTF-8 to UTF-8)
                 read at org/jruby/RubyIO.java:2972
                 read at org/jruby/RubyIO.java:2950
  block in example.rb at example.rb:2
                 open at org/jruby/RubyIO.java:1124
                <top> at example.rb:2

pobocks commented Feb 23, 2016

Confirmed:

$ touch empty-file
$ rvm jruby-head do ruby example.rb 
Encoding::ConverterNotFoundError: code converter not found (UTF-8 to UTF-8)
                 read at org/jruby/RubyIO.java:2972
                 read at org/jruby/RubyIO.java:2950
  block in example.rb at example.rb:2
                 open at org/jruby/RubyIO.java:1124
                <top> at example.rb:2
@pobocks

This comment has been minimized.

Show comment
Hide comment
@pobocks

pobocks Feb 23, 2016

Replicated on an Amazon linux box (probably whatever their Red Hat one is) and OSX 10.9.5 local.

pobocks commented Feb 23, 2016

Replicated on an Amazon linux box (probably whatever their Red Hat one is) and OSX 10.9.5 local.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment