Encoding::UndefinedConversionError when IO.copy_streaming binary files

[janko]

We're trying to make Refile compatible with JRuby. We've almost made it (and it was fairly easy, I have to admit!), but we have one remaining bug.

In Refile we're using file wrappers, that is, we wrap an actual IO into a class which responds to #read, #eof?, #close and #size. This is all it takes to act as a real IO, because we can now pass them to IO.copy_stream, and it will work. However, when we wrap images (which are binary), IO.copy_stream raises an Encoding::UndefinedConversionError. The error doesn't happen if we pass the image IO directly, only when we use a wrapper (ActionDispatch::Http::UploadedFile or Refile::File, they both fail).

We weren't able to reproduce the problem (I posted an attempt of reproducing it), so we were hoping you could help us out. The pull request is refile/refile#187. I think you only need to run bundle install to set up the test suite, the branch is jruby on my fork.

[nirvdrum]

Is this against JRuby 1.7.x, 9k, or both?

[janko]

I tested against 9.0.0.0.rc1.

[nirvdrum]

Do you mean 9.0.0.0.pre1? Or did you build from source? We haven't released rc1 yet.

[janko]

Yes, sorry, that's the one I meant, 9.0.0.0.pre1.

[nirvdrum]

Great. I can't say for certain that anything has been resolved, but a lot of encoding issues have been resolved since pre1. If you can test against jruby-head, that would be helpful. But pre2 should be releasing in the next couple weeks if you'd prefer to wait for a release.

[janko]

Yeah, I guess it's not that important that we have this ASAP. Tried compiling JRuby from source, and did /path/to/bin/jruby -S bundle, but it cannot install Refile itself (which it shouldn't, because gemspec should make Bundler just take the Refile locally from the current directory). But it doesn't matter, we'll wait for the release.

[headius]

Well your issue exposed a bug in autoload I have now fixed, but the branch still appears to try to activate the sqlite3 gem. Can you give me exact steps to reproduce the copy_stream issue?

[janko]

Hmm, that's strange, I thought I set the Gemfile correctly, with JRuby's version of sqlite3. This setup worked for 9.0.0.0.pre1, but I think it also didn't for me on JRuby master. Maybe instead of

gem "sqlite3", platforms: [:ruby]
gem "activerecord-jdbcsqlite3-adapter", platforms: [:jruby]

it should be

unless RUBY_PLATFORM == "java"
  gem "sqlite3"
else
  gem "activerecord-jdbcsqlite3-adapter"
end

In the PR on Refile I explained what was happening and included a backtrace, and I included an isolated example I thought was going to reproduce the bug, but doesn't. You should be able to simply run bundle install and bundle exec rspec to run the test suite.

[headius]

@janko-m I will try the alternate gemfile.

[headius]

Seems like using sqlite3 in spec/refile/active_record_helper.rb causes the sqlite3 gem to be triggered, and I'm not sure how to stop that from happening. I did manage to bundle install everything ok.

[headius]

@janko-m Please try to reduce your case to a single executable script. I have not been able to get your branch to run tests on JRuby yet.

[janko]

Yeah, the only reason why I linked you the PR was because I wasn't able to compose a single executable script which fails (I tried to set all encodings as those in the tests, but it didn't work, the script passed). I will try some more.

[scarfacedeb]

@janko-m @headius I was able to reproduce the issue using your example in jruby-9.0.0.0.rc1:

require "forwardable"

# This is the crucial part
Encoding.default_internal = Encoding::UTF_8

IoWrapper = Struct.new(:io) do
  extend Forwardable
  delegate [:read, :eof?, :close, :size] => :io
end

io = File.open('spec/refile/fixtures/image.jpg', encoding: "ascii-8bit") # == 'rb' mode
# Now io.external_encoding is equal to ASCII-8BIT, just like in the tests.
io_wrapper = IoWrapper.new(io)

IO.copy_stream(io_wrapper, "foo.jpg")

When I set Encoding.default_internal = nil, the problem disappears and it works as expected.

In a similar way, I was getting the same error when I tried to save a binary file with refile.
Applying the same fix with Encoding resolved that issue as well.

---

The original code example was working with jruby, because Encoding.default_internal == nil, but when I tried to require 'rails/all', the error reappeared.

By trial and error, I've found out that rails was setting Encoding.default_internal in rails/railties/lib/rails.rb which caused that error.

[headius]

I'm going to close this in hopes that it has been fixed by the thousands of commits since 2015.