Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

"invalid byte sequence in #{__ENCODING__}" because of directive_processor #412

Closed
wants to merge 3 commits into from

2 participants

Philipp Tessenow Joshua Peek
Philipp Tessenow

When I concatenate assets using the directive_processor, which are UTF-8 encoded, the processor returns a #{ENCODING} (e.g. US-ASCII) encoded String.

The attached Pull Requests lets the processor return a string in the encoding specified by environment.default_external_encoding.

Philipp Tess... added some commits
Philipp Tessenow Update lib/sprockets/directive_processor.rb
Let's create the directive_processor output string in the environment's encoding
f4d3a31
Philipp Tessenow Update lib/sprockets/directive_processor.rb
handle non encoding aware rubies
988ee1a
Lewis Marshall Add test for encoding aware DirectiveProcessor
This is a cherry-picked commit from Lewis Marshall.
The following was his original commit message:

Make the DirectiveProcessor encoding aware

In Ruby 1.9, the directive_processor.rb source file defaults to US-ASCII
as it doesn't contain a magic comment, meaning the String initialised in
@result gets a US-ASCII encoding.

This means that when content is concatenated onto @result (e.g. in
process_source), Ruby will maintain the US-ASCII encoding, provided the
content being concatenated contains no non-ascii chars.

The changes in this commit force the encoding of the resulting string to
be that of the body, because conceptually that is the string we are
"concatenating" with.

Conflicts:
	lib/sprockets/directive_processor.rb
bd1aa6b
Philipp Tessenow

I don't get why the tests fail in jruby. The changes are cherry-picket from #419 and work there.

Philipp Tessenow

It might be a jruby/rubygems problem. see: rubygems/rubygems#504

Joshua Peek
Collaborator

Merged #419

Joshua Peek josh closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 22, 2013
  1. Update lib/sprockets/directive_processor.rb

    Philipp Tessenow authored
    Let's create the directive_processor output string in the environment's encoding
  2. Update lib/sprockets/directive_processor.rb

    Philipp Tessenow authored
    handle non encoding aware rubies
Commits on Mar 11, 2013
  1. Philipp Tessenow

    Add test for encoding aware DirectiveProcessor

    Lewis Marshall authored tessi committed
    This is a cherry-picked commit from Lewis Marshall.
    The following was his original commit message:
    
    Make the DirectiveProcessor encoding aware
    
    In Ruby 1.9, the directive_processor.rb source file defaults to US-ASCII
    as it doesn't contain a magic comment, meaning the String initialised in
    @result gets a US-ASCII encoding.
    
    This means that when content is concatenated onto @result (e.g. in
    process_source), Ruby will maintain the US-ASCII encoding, provided the
    content being concatenated contains no non-ascii chars.
    
    The changes in this commit force the encoding of the resulting string to
    be that of the body, because conceptually that is the string we are
    "concatenating" with.
    
    Conflicts:
    	lib/sprockets/directive_processor.rb
This page is out of date. Refresh to see the latest.
6 lib/sprockets/directive_processor.rb
View
@@ -91,7 +91,11 @@ def prepare
def evaluate(context, locals, &block)
@context = context
- @result = ""
+ @result = if context.environment.respond_to?(:default_external_encoding) #encoding aware ruby?
+ "".encode(context.environment.default_external_encoding)
+ else
+ ""
+ end
@has_written_body = false
process_directives
1  test/fixtures/context/utf8.js.erb
View
@@ -0,0 +1 @@
+console.log("Snowman: <%= "\xe2\x98\x83" %>")
13 test/test_directive_processor.rb
View
@@ -108,6 +108,19 @@ def directives(offset = 0)
end
end
+if "".respond_to?(:encoding)
+ class TestDirectiveProcessorEncoding < Sprockets::TestCase
+ def setup
+ @env = Sprockets::Environment.new
+ @env.append_path(fixture_path('context'))
+ end
+
+ test "encoding of evaluated string" do
+ assert_equal Encoding::UTF_8, @env['utf8.js'].source.encoding
+ end
+ end
+end
+
class TestCustomDirectiveProcessor < Sprockets::TestCase
def setup
@env = Sprockets::Environment.new
Something went wrong with that request. Please try again.