" " returns space after sanitize instead of returning same " "

[GoogleCodeExporter]

I can not say this is bug but may be the policy we configure is wrong.


On the string if have html entites " " than after sanitize it show (empty 
space) but not return " "  while for other example "&lt", "&gt" shows 
correctly after sanitize.

example,

final String test = " >";

final PolicyFactory policy = Sanitizers.FORMATTING.and(
Sanitizers.BLOCKS).and(Sanitizers.STYLES);
final String safeHTML = policy.sanitize(test);

System.out.println("Before:" +test);
System.out.println("After:" +safeHTML);

Result:
-------
Before: >
After: >

Actually we need   after sanitize so can your provide guidance on this how 
to achieve.

Thx in advance!

Kr,
Urvish

Original issue reported on code.google.com by urvishmp...@gmail.com on 23 May 2014 at 1:17

[Rancho007]

We have an another issue example, if you enter ’ sanitizer covert this to html character (´) which is not recognized by rendered engine (UTF-8).

Actually we are not understanding why santizer required to change html entities to characters. This happend to only some entites not for all (example < is not converted)

Kr,
Urvish

[Rancho007]

Hi,

In fact if use different encoding rather than UTF-8 there is problem.
If use ISO-8859-1 it does not convert HTMLentities well to character.
But we are not understanding why it is required to convert HTML entities to character.
Example (default file encoding ISO-8599-1),
Before sanitize: &#8217
After sanitize: ? (if we use encoding UTF-8 it gives ´(SINGLE RIGHT QUOTATION MARK))

According to us this is incorrect?

Kr,
U

[mikesamuel]

I'm not going to expand the space of configurations to specify different policies on entity encoding.

Expanding the configuration space reduces the mean test coverage per configuration and the minimum test coverage per configuration.

Noone has yet outlined a use-case that requires this extra configuration.

I have too many other demands on time to write and maintain lots of extra tests for configurations that don't meet a required use-case.

I can't recommend this library for use in production if the minimum test coverage per configuration is low.

I want to be able to recommend this library for use in production.

CJK languages compress better when arbitrary code-points are not encoded as character reference.

Therefore, until I see a clear and well-argued use-case,
I will not complicate the code that decides whether to encode a code-point as a character reference.

[tibistibi]

i can understand that you do not have time but that should not be an argument to close this issue. you can give it a low priority maybe some one else can take this one.

you need a clear and well argued use case. ok here is mine.

i have a front-end where, via a rich text editor, a user can add content. which will be stored in the db and shown to users on some other page.
i want to make this secure so the editor can't add tags which are not allowed but he should still be able to add all the text he wants.
i also want to give clear error messages.

so ideally i would receive the added text. i would run it through the sanitizer to remove tags i don't want. then i would compare the added text with the cleaned text. if this is different i return an error telling the editor he added wrong input.
second spaces should still be kept as &nbsp because multiple &nbsp is different presented in html than multiple spaces (they are compressed to one space).

the strange thin is i have all the options to configure which tags i want with even which arguments but i have no option at all to toggle the encoding.

both scenario's are not possible because the text is encoded when i sanitize. encoding and sanitation are 2 different steps which should be two different steps in the code. this will not add you more unit tests it just would make the lib more modular.

i won't mind helping out 😄

[tibistibi]

from #43 to put all the examples in 1 issue
test case:

    @Test
    public void testSpace() {
        String text = "L   L";
        assertEquals(text, Sanitizers.FORMATTING.sanitize(text));
     }

why:
1)
a   is something different than a space. when i get a   from my richttext editor i want to preserve it. in the above example when i would add the sanitized text into an html page it would look like L L instead of L   L.
2)
i want to know if the user added something wrong and present an error:

    if (!StringUtils.equals(text, clean)) {
        addFieldError("wrong input! please check cleaned text");
    }

i don't want this to happen after a spcae replacement.

todo:
remove the   to space part or make it optional.

work around:
replace the   by spaces. do the sanitizing and checking and re add the  

[mikesamuel]

i can understand that you do not have time but that should not be an argument to close this issue. you can give it a low priority maybe some one else can take this one.

It's not just that I don't have a lot of time, but also that I've seen nothing that leads me to believe that it serves a purpose ; I don't just prioritize it low, I can't distinguish its priority from Priority:Worthless.

[mikesamuel]

  is something different than a space

Agreed. That's why I don't normalize it as space (U+20) and instead re-encode it as non-breaking space (U+A0).

[tibistibi]

ok but for me this tool is worthless now because i need to sanitize and not decode/encode.

somehow it is not important for you to preserve added text. how can i sanitize and keep the text equal. how can i tell my users that they added wrong tags?

[mikesamuel]

Testing string equality is extremely brittle -- there are many HTML strings with the same semantics.

What does HtmlChangeListener not provide that you need to do your check?

[tibistibi]

yes i was looking into that one interesting that i get an call for every change well every sanitize change. the decode encode changes are not send to a listener.
but its not a problem that html strings have the same semantics when it stays the same i do not have to report any thing to the user.
but i agree with the listener i think i can report back to the user.

so how can i keep the text equal?