Encoding::InvalidByteSequenceError - "\xE2" on US-ASCII

[abaird]

I'm trying to run the pizza server in some environment other than my local machine to do some load testing on it. After not being able to get it running on (AWS linux|AWS ubuntu)/nginx/unicorn, I decided to try AWS Elastic Beanstalk. I'm getting the following stack from Puma:

2015-08-29 20:17:07 - Encoding::InvalidByteSequenceError - "\xE2" on US-ASCII:
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `encode'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `initialize'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `new'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `parse'
        /var/app/current/api/v1/pizzerias.rb:52:in `geojson_data'
        /var/app/current/api/v1/pizzerias.rb:25:in `block (3 levels) in <class:API>'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `block in compile!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `[]'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (3 levels) in route!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:993:in `route_eval'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (2 levels) in route!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1014:in `block in process_route'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `catch'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `process_route'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:972:in `block in route!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `each'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `route!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1084:in `block in dispatch!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1081:in `dispatch!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `block in call!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `call!'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:894:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/nulllogger.rb:9:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:181:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:2021:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1486:in `block in call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1795:in `synchronize'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1486:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/puma-2.10.2/lib/puma/configuration.rb:74:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/puma-2.10.2/lib/puma/server.rb:492:in `handle_request'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/puma-2.10.2/lib/puma/server.rb:363:in `process_client'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/puma-2.10.2/lib/puma/server.rb:254:in `block in run'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/puma-2.10.2/lib/puma/thread_pool.rb:101:in `call'
        /opt/rubies/ruby-2.2.2/lib/ruby/gems/2.2.0/gems/puma-2.10.2/lib/puma/thread_pool.rb:101:in `block in spawn_thread'

I tracked this down to several special characters in the pizza_map.geojson. For example

{
      "type": "Feature",
      "properties": {
        "city": "Onsala",
        "pizzeria": "Onsala Pizzeria",
        "website": "https:\/\/www.youtube.com\/watch?v=ytXzO8_BMrQ",
        "address": "Kopparv\u00e4gen 30",
        "marker-size": "medium",
        "marker-color": "ffff00",
        "marker-symbol": "restaurant"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          12.0151975,
          57.4085344
        ]
      }
    },

I'm researching fixes for this, but I'm posting here to see if anyone knows how to fix this faster than I do.

[wwest4]

The geojson data is UTF-8. It looks like you're inheriting a default encoding of C/US-ASCII from the EB environment.

Best solution here is probably to just set the API's default encoding on startup, e.g. Encoding.default_external = 'UTF-8'

[abaird]

Bill - thanks for the help! Where would I do that, in pizza/api/v1/pizzerias.rb or somewhere else?

[wwest4]

Yep, I'd probably put it right after the last require in pizzerias.rb

[abaird]

Awesome sauce! that worked - thanks! Please close.

[wwest4]

👍

[alanbsmith]

Should this be something added to the repo in a PR, @abaird?

[abaird]

yep, I'll push it out shortly

[abaird]

Actually, I submitted a PR to clean up the indentation in the pizzerias.rb file. What is the best way to proceed? Should I submit a PR to change the external encoding assuming the indentation PR is accepted or something else?

[alanbsmith]

Merged that in. Feel free to submit and thanks!