Consider removing unprintable characters?

[swanson]

We've encountered some issues with HTML containing unprintable characters (namely \u2028\u2029) over on the Stringer project (stringer-rss/stringer#295). The issue manifests itself further down the chain once we try to load the HTML with unprintable characters as JSON.

We've resolved our issues by adding a gsub(/[^[:print:]]/, '') call after running the contents through loofah. I think this makes sense to add this as part of loofah's sanitizing process - any thoughts before I take a stab at a PR?

[flavorjones]

In principle this feature makes sense to me; however I'm curious as to why
it matters whether these characters are present or not. Can you provide a
bit of context to help me understand?
On Apr 6, 2014 6:00 PM, "matt swanson" notifications@github.com wrote:

We've encountered some issues with HTML containing unprintable characters
(namely \u2028\u2029) over on the Stringer project (stringer-rss/stringer#295stringer-rss/stringer#295).
The issue manifests itself further down the chain once we try to load the
HTML with unprintable characters as JSON.

We've resolved our issues by adding a gsub(/[^[:print:]]/, '') call after
running the contents through loofah. I think this makes sense to add this
as part of loofah's sanitizing process - any thoughts before I take a
stab at a PR?

Reply to this email directly or view it on GitHubhttps://github.com//issues/65
.

[swanson]

Sure - the problem occurs downstream. In our particular case, we are parsing RSS feeds and then sanitizing the post content and storing it in a database. Later on, we are converting that post content to JSON (no problems yet, the unprintable characters are valid JSON) and then using a client-side Javascript framework (Backbone.js in our case) to render the data.

The issues occur when we try to run those unprintable characters through the Javascript parser. More details here: http://timelessrepo.com/json-isnt-a-javascript-subset but the long and short of it: Javascript cannot handle these unprintable characters even though JSON can. "No string in JavaScript can contain a literal U+2028 or a U+2029."

[flavorjones]

That's crazy! OK, got it loud and clear.

How about this proposed usage:

doc = Loofah.fragment(thing)
doc.scrub!(:unprintable)

so that the non-printable characters can be removed by chaining .scrub!(:unprintable) to the end of the currently-used scrubbers.

If that usage makes sense, then are you able to take a first whack at an implementation? If not, I'll try to get to it.

[flavorjones]

Possibly useful reference: here's how Rack::JSONP addressed this issue:

https://github.com/rack/rack-contrib/pull/37/files

[swanson]

👍 sounds good to me. I can probably get going on a pull request next week.

[flavorjones]

Let me know if you have questions on where to start. Should be as easy as
adding a new class to lib/loofah/scrubbers.rb, and a corresponding entry
in MAP at the bottom of that file.

On Wed, Apr 9, 2014 at 12:25 PM, matt swanson notifications@github.comwrote:

[image: 👍] sounds good to me. I can probably get going on a pull
request next week.

Reply to this email directly or view it on GitHubhttps://github.com//issues/65#issuecomment-39984807
.