Encoding::UndefinedConversionError

[michaelshobbs]

Is there anything we can do in chef-handler-sns to "fix" this?

[2015-07-09T21:34:37+00:00] INFO: Running queued delayed notifications before re-raising exception
[2015-07-09T21:34:37+00:00] ERROR: Running exception handlers
[2015-07-09T21:34:37+00:00] ERROR: Report handler Chef::Handler::Sns raised #<Encoding::UndefinedConversionError: "\xE2" from ASCII-8BIT to UTF-8>
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/aws-sdk-v1-1.53.0/lib/aws/sns/topic.rb:272:in `encode'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/aws-sdk-v1-1.53.0/lib/aws/sns/topic.rb:272:in `to_json'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/aws-sdk-v1-1.53.0/lib/aws/sns/topic.rb:272:in `publish'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-handler-sns-1.2.0/lib/chef/handler/sns.rb:37:in `report'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:226:in `run_report_unsafe'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:214:in `run_report_safely'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:118:in `block in run_exception_handlers'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:117:in `each'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:117:in `run_exception_handlers'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:127:in `block in <class:Handler>'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:122:in `call'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:122:in `block in run_failed'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:121:in `each'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:121:in `run_failed'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:431:in `rescue in do_run'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:441:in `do_run'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:200:in `block in run'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:194:in `fork'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:194:in `run'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application.rb:215:in `run_chef_client'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application/client.rb:314:in `block in run_application'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application/client.rb:306:in `loop'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application/client.rb:306:in `run_application'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application.rb:66:in `run'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/releases/20150629124612_3422-20150629124612/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/bin/chef-client:26:in `<top (required)>'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/current/bin/chef-client:16:in `load'
[2015-07-09T21:34:37+00:00] ERROR: /opt/aws/opsworks/current/bin/chef-client:16:in `<main>'
[2015-07-09T21:34:37+00:00] ERROR: Exception handlers complete

[zuazo]

@michaelshobbs, I'm not sure. I think setting the locale to UTF-8 should fix this error. There are a lot of complains out there about this type of encoding errors in Chef.

Could you give me more information to know how to reproduce the error?

[michaelshobbs]

Well that's a might bit tricky I think. I use this project to notify me on failed deployments on AWS OpsWorks using dokku. I think it must be it's failure output or that of a failed plugin. Not 100% certain. Could also be that combined with upgrading to Chef 11.10. I can reproduce it in my production environment consistently though.

[michaelshobbs]

@zuazo when you say set the locale, to what specifically are you referring?

[zuazo]

Something like LC_ALL=en_US.UTF-8. Configuring it in the system or running chef-client with it.

[michaelshobbs]

I set that but to no avail.

root@strudel:/var/lib/aws/opsworks/chef# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=en_US.utf8


root@strudel:/var/lib/aws/opsworks/chef# env | grep LC
LC_ALL=en_US.utf8

[zuazo]

That's strange. Are you sure chef is running with this locale? Perhaps some kind of LC_ALL=C in the init script.

What is the message body?

[michaelshobbs]

I set LC_ALL in /etc/environment but I have no real control over how the OpsWorks chef client is invoked.

Not sure how to get at the message body. Any pointers?

[zuazo]

The message body is the content of the email you should receive through SNS. That is what is causing the error.

I guess you are using the default body template. Maybe there is an exception to report that has some kind of encoding error/UTF-8 character.

I think maybe we should treat everything as UTF-8 and allow to change the charset.

[michaelshobbs]

My template is different than the one to which you linked but yeah I'm sure that is the issue. The message is sent successfully in all cases except when there is a failure in my deployment. Below is my template and a gist of a failed deployment using my patch from #7.

<%# file "app/files/default/chef_handler_sns_body.erb" %>

Node Name: <%= node.name %>
<% if node.attribute?("fqdn") -%>
Hostname: <%= node.fqdn %>
<% end -%>

Chef Run List: <%= node.run_list.to_s %>
Chef Environment: <%= node.chef_environment %>

<% if node.attribute?("opsworks") -%>
######  OpsWorks Summary  ######
Operating System: <%= `head -1 /etc/issue | sed -e 's/ \\\\.*//'`.chomp %>
OpsWorks Instance: <%= node[:opsworks][:instance][:hostname] %>
OpsWorks Instance ID: <%= node[:opsworks][:instance][:id] %> (Instance ID: <%= node[:opsworks][:instance][:aws_instance_id] %>)
OpsWorks Layers: <%= node[:opsworks][:instance][:layers].map{|id| node[:opsworks][:layers][id][:name] }.join(', ') %>
OpsWorks Stack: <%= node[:opsworks][:stack][:name] %>
EC2 Region: <%= node[:opsworks][:instance][:region] %>
EC2 Availability Zone: <%= node[:opsworks][:instance][:availability_zone] %>
<% end -%>

<% if node.attribute?("ec2") -%>
Instance Id: <%= node.ec2.instance_id %>
<% if node.attribute?("ec2.public_hostname") -%>
Instance Public Hostname: <%= node.ec2.public_hostname %>
<% end -%>
Instance Hostname: <%= node.ec2.hostname %>
<% if node.attribute?("ec2.public_ipv4") -%>
Instance Public IPv4: <%= node.ec2.public_ipv4 %>
<% end -%>
Instance Local IPv4: <%= node.ec2.local_ipv4 %>
<% end -%>

Chef Client Elapsed Time: <%= elapsed_time.to_s %>
Chef Client Start Time: <%= start_time.to_s %>
Chef Client End Time: <%= end_time.to_s %>

<% if exception -%>
Exception: <%= run_status.formatted_exception %>
Stacktrace:
<%= Array(backtrace).join("\n") %>

<% end -%>

report with UTF-8: https://gist.github.com/michaelshobbs/5b6bc933fb3dc9eb96c6

[zuazo]

Fixed in #7. Thanks for all your help!

[michaelshobbs]

This is still an issue even with the patch.

[2015-07-23T17:49:00+00:00] INFO: Running queued delayed notifications before re-raising exception
[2015-07-23T17:49:00+00:00] ERROR: Running exception handlers
[2015-07-23T17:49:00+00:00] ERROR: Report handler Chef::Handler::Sns raised #<Encoding::UndefinedConversionError: "\xE2" from ASCII-8BIT to UTF-8>
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/aws-sdk-v1-1.53.0/lib/aws/sns/topic.rb:272:in `encode'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/aws-sdk-v1-1.53.0/lib/aws/sns/topic.rb:272:in `to_json'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/aws-sdk-v1-1.53.0/lib/aws/sns/topic.rb:272:in `publish'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-handler-sns-1.2.0/lib/chef/handler/sns.rb:37:in `report'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:226:in `run_report_unsafe'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:214:in `run_report_safely'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:118:in `block in run_exception_handlers'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:117:in `each'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:117:in `run_exception_handlers'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/handler.rb:127:in `block in <class:Handler>'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:122:in `call'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:122:in `block in run_failed'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:121:in `each'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:121:in `run_failed'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:431:in `rescue in do_run'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:441:in `do_run'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:200:in `block in run'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:194:in `fork'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/client.rb:194:in `run'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application.rb:215:in `run_chef_client'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application/client.rb:314:in `block in run_application'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application/client.rb:306:in `loop'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application/client.rb:306:in `run_application'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/lib/chef/application.rb:66:in `run'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/releases/20150709092700_3424-20150709092700/vendor/bundle/ruby/2.0.0/gems/chef-11.10.4/bin/chef-client:26:in `<top (required)>'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/current/bin/chef-client:16:in `load'
[2015-07-23T17:49:00+00:00] ERROR: /opt/aws/opsworks/current/bin/chef-client:16:in `<main>'
[2015-07-23T17:49:00+00:00] ERROR: Exception handlers complete
[2015-07-23T17:49:00+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out

[michaelshobbs]

👍

[zuazo]

@michaelshobbs, sorry for the delay and thanks for all your help and feedback!

Current master released in 2.0.0. I hope this bug is already fixed.

[michaelshobbs]

Thanks!! I'll test 2.0.0 in the new year.