Encoding issues when loading JSON files

[lardcanoe]

Only getting this in prod, not in my sandbox, so not sure what is different.

2014-08-05_20:10:28.87487 ""\xE2"" on US-ASCII
2014-08-05_20:10:28.87845 org/jruby/RubyString.java:7573:in `encode'
2014-08-05_20:10:28.87847 json/ext/Parser.java:175:in `initialize'
2014-08-05_20:10:28.87848 json/ext/Parser.java:151:in `new'
2014-08-05_20:10:28.87849 /home/cp-workers/cp-workers/releases/5cd0d7968260bdf46c00ae95f05d2fa6a1631371/vendor/bundle/jruby/1.9/gems/json-1.8.1-java/lib/json/common.rb:155:in `parse'
2014-08-05_20:10:28.87850 /home/cp-workers/cp-workers/releases/5cd0d7968260bdf46c00ae95f05d2fa6a1631371/vendor/bundle/jruby/1.9/gems/multi_json-1.3.7/lib/multi_json/adapters/json_common.rb:7:in `load'
2014-08-05_20:10:28.87851 /home/cp-workers/cp-workers/releases/5cd0d7968260bdf46c00ae95f05d2fa6a1631371/vendor/bundle/jruby/1.9/gems/multi_json-1.3.7/lib/multi_json.rb:96:in `load'
2014-08-05_20:10:28.87851 /home/cp-workers/cp-workers/releases/5cd0d7968260bdf46c00ae95f05d2fa6a1631371/vendor/bundle/jruby/1.9/bundler/gems/aws-sdk-core-ruby-d6d8ddd2fd4d/lib/aws/service_builder.rb:51:in `load_json'

Checked which json files have non-ascii chars. There are a few. Can you clean them up?

➜  apis 1.9.3 git:(master) ✗ grep --color='auto' -P -n "[\x80-\xFF]" *.json

glacier-2012-06-01.api.json:1357:          "documentation":"<p>The archive description that you are uploading in parts.</p> <p>The part size must be a megabyte (1024 KB) multiplied by a power of 2��for example, 1048576 (1 MB), 2097152 (2 MB), 4194304 (4 MB), 8388608 (8 MB), and so on. The minimum allowable part size is 1 MB, and the maximum is 4 GB (4096 MB).</p>"

route53domains-2014-05-15.api.json:508:          "documentation":"<p>Whether the domain name is available for registering.</p> <p>Type: String</p> <p>Valid values:</p> <ul> <li> <code>AVAILABLE</code> �� The domain name is available.</li> <li> <code>AVAILABLE_RESERVED</code> �� The domain name is reserved under specific conditions.</li> <li> <code>AVAILABLE_PREORDER</code> �� The domain name is available and can be preordered.</li> <li> <code>UNAVAILABLE</code> �� The domain name is not available.</li> <li> <code>UNAVAILABLE_PREMIUM</code> �� The domain name is not available.</li> <li> <code>UNAVAILABLE_RESTRICTED</code> �� The domain name is forbidden.</li> <li> <code>RESERVED</code> �� The domain name has been reserved for another person or organization.</li> </ul>"

storagegateway-2013-06-30.api.json:404:      "documentation":"<p>This operation deletes a gateway. To specify which gateway to delete, use the Amazon Resource Name (ARN) of the gateway in your request. The operation deletes the gateway; however, it does not delete the gateway virtual machine (VM) from your host computer.</p> <p>After you delete a gateway, you cannot reactivate it. Completed snapshots of the gateway volumes are not deleted upon deleting the gateway, however, pending snapshots will not complete. After you delete a gateway, your next step is to remove it from your environment.</p> <important> <p>You no longer pay software charges after the gateway is deleted; however, your existing Amazon EBS snapshots persist and you will continue to be billed for these snapshots.�You can choose to remove all remaining Amazon EBS snapshots by canceling your Amazon EC2 subscription.� If you prefer not to cancel your Amazon EC2 subscription, you can delete your snapshots using the Amazon EC2 console. For more information, see the <a href=\"http://aws.amazon.com/storagegateway\"> AWS Storage Gateway Detail Page</a>. </p> </important>"

swf-2012-01-25.api.json:386:      "documentation":"<p> Returns the list of domains registered in the account. The results may be split into multiple pages. To retrieve subsequent pages, make the call again using the nextPageToken returned by the initial call. </p> <p><b>Access Control</b></p> <p>You can use IAM policies to control this action's access to Amazon SWF resources as follows:</p> <ul> <li>Use a <code>Resource</code> element with the domain name to limit the action to only specified domains. The element must be set to <code>arn:aws:swf::AccountID:domain/*\"</code>, where ��AccountID\" is the account ID, with no dashes.</li> <li>Use an <code>Action</code> element to allow or deny permission to call this action.</li> <li>You cannot use an IAM policy to constrain this action's parameters.</li> </ul> <p>If the caller does not have sufficient permissions to invoke the action, or the parameter values fall outside the specified constraints, the action fails by throwing <code>OperationNotPermitted</code>. For details and example IAM policies, see <a href=\"http://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dev-iam.html\">Using IAM to Manage Access to Amazon SWF Workflows</a>.</p>"

swf-2012-01-25.api.json:3257:      "documentation":"<p>Provides details of the <code>ScheduleActivityTask</code> decision.</p> <p><b>Access Control</b></p> <p>You can use IAM policies to control this decision's access to Amazon SWF in much the same way as for the regular API:</p> <ul> <li>Use a <code>Resource</code> element with the domain name to limit the decision to only specified domains.</li> <li>Use an <code>Action</code> element to allow or deny permission to specify this decision.</li> <li>Constrain the following parameters by using a <code>Condition</code> element with the appropriate keys. <ul> <li> <code>activityType.name</code>: String constraint. The key is ��swf:activityType.name��.</li> <li> <code>activityType.version</code>: String constraint. The key is ��swf:activityType.version��.</li> <li> <code>taskList</code>: String constraint. The key is ��swf:taskList.name��.</li> </ul> </li> </ul> <p>If the caller does not have sufficient permissions to invoke the action, or the parameter values fall outside the specified constraints, the action fails. The associated event attribute's <b>cause</b> parameter will be set to OPERATION_NOT_PERMITTED. For details and example IAM policies, see <a href=\"http://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dev-iam.html\">Using IAM to Manage Access to Amazon SWF Workflows</a>.</p>"

swf-2012-01-25.api.json:3488:      "documentation":"<p>Provides details of the <code>StartChildWorkflowExecution</code> decision.</p> <p><b>Access Control</b></p> <p>You can use IAM policies to control this decision's access to Amazon SWF in much the same way as for the regular API:</p> <ul> <li>Use a <code>Resource</code> element with the domain name to limit the decision to only specified domains.</li> <li>Use an <code>Action</code> element to allow or deny permission to specify this decision.</li> <li>Constrain the following parameters by using a <code>Condition</code> element with the appropriate keys. <ul> <li> <code>tagList.member.N</code>: TBD</li> <li> <code>taskList</code>:String constraint. The key is \"swf:taskList.name\".</li> <li> <code>workflowType.name</code>: String constraint. The key is ��swf:workflowType.name��.</li> <li> <code>workflowType.version</code>: String constraint. The key is ��swf:workflowType.version��.</li> </ul> </li> </ul> <p>If the caller does not have sufficient permissions to invoke the action, or the parameter values fall outside the specified constraints, the action fails. The associated event attribute's <b>cause</b> parameter will be set to OPERATION_NOT_PERMITTED. For details and example IAM policies, see <a href=\"http://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dev-iam.html\">Using IAM to Manage Access to Amazon SWF Workflows</a>.</p>"

[trevorrowe]

I can verify that there are UTF-8 characters into the JSON API documents. Is there a particular reason this is a problem? AFAIK, UTF-8 is valid inside JSON.

On a separate note, I've got a work-in-progress branch that separates the API reference documentation into separate JSON documents from the API. The primary purpose for this change is to make it simple to exclude the documentation when building the gem. In my local branch, the final gem size is reduced from ~730KB to ~247KB. If these files were not loaded at runtime, would that make a difference to you?

[trevorrowe]

I'm sorry. I see the issue you are having. I can update the SDK to load the files, forcing the proper encoding.

[trevorrowe]

Can you see if this commit resolves the issue?

[trevorrowe]

This fix went out with the 2.0.0.rc15 release. This should resolve the issue. Please re-open the issue if the problem persists.

[lardcanoe]

Fixed. Thanks!