Non-english characters are encoded

[danroth27]

Reported as aspnet/Razor#409 by @anfomin

Every non-english character is encoded. For example:

@{
    string s = "Добрый день"; // this is Russian text
}
@s

The result is something like:

---

Characters displayed correct, but page size increased significantly 👎 Also MVC 5 rendered unicode symbol correctly in HTML source.

[blowdart]

Actually MVC 5 would have done the same if you'd switched to AntiXSS as the encoder.

By design.

[Tratcher]

What API are you complaining about?

[Tratcher]

Ah, IHtmlEncoder

[dougbu]

@blowdart however in MVC 5 it was a choice and the default HTML encoder provided readable source of a reasonable size.

[blowdart]

The standard Russian characters should have been safe listed with the new encoder.
You'll need to examine the whitelist generation to see why they're not.

[dougbu]

@blowdart from the HtmlEncoder source, the default escaped range is UnicodeRanges.BasicLatin which has this convenient comment:

/// A <see cref="UnicodeRange"/> corresponding to the 'Basic Latin' Unicode block (U+0000..U+007F).

Are you suggesting a different default?

[blowdart]

I though it was changed, but ok. And no, let's be honest, I'd encode everything by default to avoid certain utf7 badness. Customer can use the Default property to widen their safe list as needed.

[anfomin]

Every character encoding brings another problems. Sometimes we use <script> generation in razor pages like:

<script>
    $(function() {
        var someVariable = "@thisIsDotNetVariable";
        // other code goes here
    });
</script>

So JavaScript will not decode this value and will display unicode numbers.

[blowdart]

Don't do that for javascript. It's simply wrong.

@ encoding is HTML encoding. It's not JavaScript encoding. Using HTML encoding inside javascript means some of the characters that can cause javascript injection are not encoded.

The most foolproof way to inject variable values into javascript is to have a div or span or other tag, and use a data attribute - using @ encoding there is correct, and you're protected. Then you'd retrieve it within the javascript.

[anfomin]

@blowdart OK, this it good explanation not to use @ with JS. But encoded-text increase page size 7 times (each character converted into 7). I do not think this is good behavior by default.

Also there is no attack can be done using Russian or any other language characters so they do not require HTML encoding. This characters do not have any "special" behavior compared to English ones.

[blowdart]

Actually you can attack via some characters depending on the underlying page encoding and other badness. Utf7 is a great example of this - http://openmya.hacker.jp/hasegawa/public/20071107/s6/h6.html?file=datae.txt has some details.

We're discussing internally what we want to do - but the encoders are used by things other than razor, and defaulting to encoding everything over and above ASCII is what makes it 100% safe for the other things to consume, so I'm not minded to change the default encoder configuration. With gzip compression which most servers support these days the actually transport size increase shouldn't be that painful, but I realise if you're not in a default English speaking country it's probably not a cost you want to take (being a Brit, it annoys me when we end up assuming everything speaks US english).

You could just whitelist everything yourself, either by

HtmlEncoder.Default = new HtmlEncoder(UnicodeRanges.All);

before the encoder gets registered in DI, or via

serviceColl.AddWebEncoders(o => { 
    o.CodePointFilter = new CodePointFilter(UnicodeRanges.All);
});

If you wanted to narrow it down, the you can create ranges and pass them into the encoder which match the Unicode code charts which contain the characters you need (Unicode doesn't do languages, so you will need to mix and match - you can find the code charts at http://www.unicode.org/charts/)

[dougbu]

Note setting HtmlEncoder.Default is likely not what you want. That changes only the HTML encoder, leaving the JavaScript and URL encoders with their (ASCII) defaults.

Recommend

  services.Configure<WebEncoderOptions>(options => new CodePointFilter(UnicodeRanges.All));

because (today) nothing configures WebEncoderOptions but much of the system adds the web encoders.