Encoding::UndefinedConversionError from UTF-8 to US-ASCII while minifying JS in production #76

Open
kgrz opened this Issue Feb 4, 2013 · 14 comments

7 participants

@kgrz

The assets part of my app:

assets {                                                                                                                                                                                       
  serve '/javascripts', from: 'public/javascripts'
  serve '/stylesheets', from: 'public/stylesheets'
  serve '/images', from: 'public/images'

  js :foundation, '/javascripts/foundation.js',[
    '/javascripts/foundation/*.js'
  ]

  js :application, '/javascripts/application.js',[
    '/javascripts/application.js'
  ]

  css :application, '/stylesheets/application.css', [
    '/stylesheets/app.css',
  ]

  js_compression :yui
  css_compression :yui

I'm using slim as my templating engine. The app is a "modular" Sinatra app. Not sure if these change the behaviour.

The offending part is at the line https://github.com/rstacruz/sinatra-assetpack/blob/master/lib/sinatra/assetpack/package.rb#L101

This is unrelated to the yui compressor since changing that to the default compressor also triggers this error. The issue with this error is when compiling/compressing the assets on a per-request basis, the javascripts files won't get sent to the browser:

Screen Shot 2013-02-04 at 2 18 14 PM

Changing the entire method to the below snippet corrects this behaviour.

if result.body.respond_to?(:force_encoding)
  result.body.force_encoding("ISO-8859-1").encode("UTF-8")  if result.status == 200                                                                                    
else
  result.body  if result.status == 200
end

Tested this with 1.8.7 (Mac system ruby), 1.9.3-p327, rubinius 1.2.4. Here's the commit in my repo kgrz@4810b71 Will send a PR if this gets tested thoroughly. The tests pass without any error but there are failures which were there before changing this method.

I'm guessing the reason for this happening only in production is that the combined method is not used in other environments.

Another point to note is that this setup ( assets in public directory) cannot used if you want to build the assets first since by default, the built assets get copied into public directory.

Here's the stacktrace:

Encoding::UndefinedConversionError - U+FEFF from UTF-8 to US-ASCII:
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:101:in `encode'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:101:in `block in combined'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:97:in `map'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:97:in `combined'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:83:in `minify'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/class_methods.rb:28:in `block (3 levels) in add_compressed_routes!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/tilt-1.3.3/lib/tilt.rb:127:in `fetch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/class_methods.rb:27:in `block (2 levels) in add_compressed_routes!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:1293:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:1293:in `block in compile!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `[]'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `block (3 levels) in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:876:in `route_eval'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `block (2 levels) in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:897:in `block in process_route'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:895:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:895:in `process_route'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:859:in `block in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:858:in `each'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:858:in `route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:963:in `block in dispatch!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `block in invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:960:in `dispatch!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:794:in `block in call!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `block in invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:794:in `call!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:780:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/path_traversal.rb:16:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/json_csrf.rb:17:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/nulllogger.rb:9:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/head.rb:11:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:124:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:in `block in pre_process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `pre_process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in `process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:in `receive_data'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/handler/thin.rb:16:in `run'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/server.rb:264:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/server.rb:141:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/bin/rackup:4:in `<top (required)>'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/bin/rackup:23:in `load'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/bin/rackup:23:in `<main>'
@j15e
Collaborator

Thanks for the detailed report.

I think the encoding issue might be related to querying via http to get assets, we should get them from the disk directly when possible. Might be related to #68 too

@j15e
Collaborator

(and or detect & deal with encoding properly, I think your fix will work only for files encoded in ISO-8859-1)

@kgrz

ASCII-8BIT to UTF-8 conversion can be done this way without errors. So, files encoded in ISO-8859-1, US-ASCII, ASCII-8BIT and UTF-8 can be converted to UTF-8 encoding without errors.

@j15e
Collaborator

Unfortunately force_encoding is not (1.8.7 compatible](http://ruby-doc.org/core-1.8.7/String.html) and I doubt doing this is the real solution as force_encoding does not change anything. We should add a test I adress this issue with a cleaner solution.

@shioyama

Just bit by the same bug. Any progress?

@j15e
Collaborator

I haven't work on this issue, but would sure accept a pull request on it. I doubt replacing the package snippet with force_encoding would solve all cases.

@doits

just stumbled upon this right now, too. why is it trying to convert from utf-8 to ascii anyway? everything is written in utf-8 on my server (hopefully all vendor js, too), output to clients is utf-8. why does it want to convert from utf-8 to ascii?

@kgrz

@doits I suppose this has to do with backwards compatibility. (@j15e mentioned this earlier :)

@j15e
Collaborator

I do not have any update on this issue but it should indeed be resolved and we have to find a clean solution for this issue.

@doits

I could get rid of the error by setting the correct LANG env var. By doing LANG=en_US.UTF-8 it changed Encoding.default_external to UTF-8. Therefore, assetpack does not try to convert things to ASCII, but to UTF-8 and everything works.

I could do this for apache with phusion passenger by SetEnv LANG en_US.UTF-8.

@Soulou

I'm also encountering this issue, is there something new? (Setting the env var LANG didn't work in my case.

@jamesrwhite jamesrwhite referenced this issue in jamesrwhite/minicron Apr 16, 2014
Closed

UTF-8 error #64

@jarredholman

Can an option be provided to specify which encoding should be used instead of forcing it to the Encoding.default_external?

@willmoss

I encountered this issue today, it would be nice to get a fix thanks

@j15e
Collaborator

Could you report your full stacktrace @willmoss ? Is it exactly the same as the original issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment