Should strip non-space controls and noncharacters from input prior to parsing

[cloudy9101]

Hello, I think Sanitize#preprocesses not filter "\b" , then Nokogiri make it a �. Is it a mistake or a normal behavior?

[rgrove]

I'm not sure what character you mean by "\b". Can you please specify the actual hex value of the Unicode code point you're referring to?

In any event, if you're seeing a "�" somewhere, that's a good sign that either the input was not valid UTF-8 or the output is not being properly displayed or processed as UTF-8.

[cloudy9101]

Emm, for "\b" I mean backspace, \u0008 .

[rgrove]

Thanks for clarifying. I understand the problem now.

It looks like Sanitize should probably strip all non-space control characters and non-characters from the input prior to parsing, as described in the HTML standard. We currently strip some, but not all, of these characters, since we were following older guidelines that have since been withdrawn.

[ixti]

And now Sanitize do not strips out U+2028 is that intentional?

[rgrove]

Yes. I'm not aware of any requirement in the HTML standard for parsers to strip U+2028. Have I missed something?

Previously Sanitize was stripping some characters that shouldn't have been stripped and was not stripping some characters that should have been stripped. Now it only strips characters that actually aren't allowed in HTML.

[ixti]

I'm not saying you've missed something. Was just double checking that it was intentional (before changing expectations in my app).

In any case your answer clears things up. Thank you.