UTF-8 Support Again

[ryanwood]

It looks like both #185 and #192 were both reverted so I'm opening this issue up again. I'm running Ruby 2.0 and Liquid 2.6.0.

I have a drop that includes:

{ 'payment' => { 'name' => 'Céline Smith'}}

I get the following error on rendering the template:

Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT
    from /app/vendor/bundle/ruby/2.0.0/gems/liquid-2.6.0/lib/liquid/block.rb:123:in `join'
    from /app/vendor/bundle/ruby/2.0.0/gems/liquid-2.6.0/lib/liquid/block.rb:123:in `render_all'
    from /app/vendor/bundle/ruby/2.0.0/gems/liquid-2.6.0/lib/liquid/block.rb:82:in `render'
    from /app/vendor/bundle/ruby/2.0.0/gems/liquid-2.6.0/lib/liquid/template.rb:128:in `render'
    from /app/app/models/notification.rb:47:in `render'

When I examine the object after render it looks like this:

{ 'payment' => { 'name' => "C\xC3\xA9line Smith"}}

If I test the encoding of the string beforehand it is UTF-8 yet after the render fails it is ASCII-8BIT. Any ideas?

[fw42]

Hm I can't reproduce this.

flo@flombp ~code/liquid $ git checkout 2-6-stable
Switched to branch '2-6-stable'
Your branch is up-to-date with 'origin/2-6-stable'.
flo@flombp ~code/liquid [2-6-stable] $ ruby -v
ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-darwin13.0.0]
flo@flombp ~code/liquid [2-6-stable] $ cat test.rb
require "liquid"
before = "Céline Smith"
after = Liquid::Template.parse("{{ name }}").render("name" => before)
puts before.encoding
p after
puts after.encoding
flo@flombp ~code/liquid [2-6-stable] $ ruby -I lib test.rb
UTF-8
"Céline Smith"
UTF-8

Am I missing something?

[ryanwood]

So weird. That code works for me as well. The only place it is breaking is on Heroku. When I run your code locally or on the Heroku console, it work as well. However, within the app itself (which is considerably more complex) it breaks. Obviously not a Liquid issue. Thanks for looking into it.

[cmager]

I have the same issue with ruby 2.1.1, rails 4.0.4 and liquid master/3.0.0. Did you solve your problem? What was the issue there?

[ryanwood]

That one was a bear to figure out. I was using liquid to generate emails. It turned out to be a problem with the mail gem. When I created a friendly email address, the mail gem would mutate the argument and convert it from UTF-8 to ASCII-8BIT. Then liquid would get the mutated argument breaking the rendering process. Here's the offending code that I changed to fix it:

Old:

     def build_address(email, name)
       address = Mail::Address.new(email)
       address.display_name = name
       address.format
     end

New:

     def build_address(email, name)
       address = Mail::Address.new(email)
       # Need dup as this call mutates the string for some reason
       address.display_name = name.dup
       address.format
     end

See details here: mikel/mail#621. It is currently still an open issue.

[cmager]

thanks for your reply. I'm rendering HTML templates with liquid tags that are stored in a database. If i am running my code on the rails console everything is fine, but in the browser it's giving me that exception. Usually when i have öäü's or so in the text. That happened after a rails 3.2 -> 4 migraton.

I'll try your advice with dup and give feedback!