Add UTF-7 to replacement encoding list? / Encoding sniffing

[jungshik]

Maybe this was discussed before, but I couldn't find a bug on this. What do you think of treating UTF-7 the same way as ISO-2022-{KR,CN}, HZ-GB, etc?

When decoding, the whole input is replaced by U+FFFD. When encoding, use UTF-8.

Background: Blink began to use Compact Encoding Detector ( google/compact_enc_det ) when no encoding label is found (http, meta). When 7-bit encoding detection is on, it detects ISO-2022-{KR,CN}, HZ-GB AND UTF-7 in addition to ISO-2022-JP. 7-bit encoding detection is ON for ISO-2022-JP, but we want to suppress other 7-bit encodings. I think the best way to 'suppress' (unsupport) them is to turn the whole input to U+FFFD.

[annevk]

If that is compatible, sounds fine. I know @hsivonen wants to greatly extend the list of replacement labels with all kinds of legacy encodings that might cause problems. Main issue is doing the research to see whether it is safe.

[hsivonen]

Blink began to use Compact Encoding Detector ( google/compact_enc_det ) when no encoding label is found (http, meta).

Whoa! Do you mean Blink now uses more unspecified heuristics than before even for non-Japanese locales? Why more heuristics? Why without a spec? ಠ_ಠ

When 7-bit encoding detection is on, it detects ISO-2022-{KR,CN}, HZ-GB AND UTF-7 in addition to ISO-2022-JP. 7-bit encoding detection is ON for ISO-2022-JP, but we want to suppress other 7-bit encodings. I think the best way to 'suppress' (unsupport) them is to turn the whole input to U+FFFD.

It seems to me that it's bad for Blink to adopt a library that doesn't do what Blink needs to do as a black box and then tweak the output of the black box as opposed to writing code that does what's needed in the first place.

As for the issue of mapping the label "utf-7" to the replacement encoding generally, I think it's a matter of determining if real Web content relies on the label "utf-7" being unrecognized so as to fall back on an ASCII-compatible encoding.

If Blink is willing to experiment with shipping a mapping from "utf-7" to replacement and assessing breakage, that would be cool.

[hsivonen]

Also, does existing Web content actually require ISO-2022-JP to be heuristically detected (as opposed to heuristically detecting between Shift_JIS and EUC-JP)? Wouldn't in be a security improvement not to detect ISO-2022-JP and not to let it be chosen from the menu (i.e. support it only when labeled)?

[hsivonen]

Also, letting a detector decide that input should be decoded as the replacement encoding is most likely a bad idea. Consider an unlabeled site whose markup template and site-provided content are all ASCII and that accepts comments. A user could cause denial of service by submitting a comment that looks like a UTF-7 or HZ byte sequence.

[jungshik]

Chromium got rid of the encoding menu completely. So, the encoding cannot be chosen via UI any more.

Blink did not add a new heuristic, but just replaced ICU encoding detector with Compact Encoding Detector. It only kicks in when there is no encoding label (meta / http C-T charset).

[annevk]

@jungshik if the web requires such a detector, we should work towards standardizing it. Would Google be willing to work with us on that?

[jungshik]

FYI, CED is open-sourced and available in github. See https://github.com/google/compact_enc_det

BTW, Blink was changed recently to treat auto-detected (unlabelled) 7-bit encodings (other than ISO-2022-JP) as ASCII. We came across @hsivonen's scenario ( #68 (comment) )

[jungshik]

does existing Web content actually require ISO-2022-JP to be heuristically detected
Wouldn't in be a security improvement not to detect ISO-2022-JP ?

Both are good questions. I've just filed https://bugs.chromium.org/p/chromium/issues/detail?id=647582

[annevk]

And by "as ASCII" I guess you mean windows-1252?

[hsivonen]

@jungshik, I have some more questions:

• Did you explore guessing the encoding from the top-level domain name instead of guessing it from content?
• How many bytes do you feed to the detector?
• What happens if the network stalls before that many bytes have been received? Do you wait or do you stop detecting based on a timer? (Timers have their own problems, and Firefox got rid of the timer in Firefox 4.)
• Once the detector has seen as many bytes as you usually feed it, do you continue feeding data to the detector to allow it to revise its guess? If yes, what do you do if the detector revises its guess?
• Other than ISO-2022-JP (per your previous comment) and, I'm extrapolating, EUC-JP, can the detector (as used in Blink) guess an encoding that wasn't previously the default for any Chrome localization? That is, does the detector make pages "work" when they previously didn't for any localization defaults?
• Can the detector (as used in Blink) guess UTF-8? That is, can Blink's use of the detector inadvertently encourage the non-labeling of newly-created UTF-8 content?

(Also, "is open-sourced and available in github" doesn't really answer @annevk's question about willingness to work towards standardizing.)

[JinsukKim]

Realized these questions better be answered by the person who ported the new detector to Blink (which is me):

Did you explore guessing the encoding from the top-level domain name instead of guessing it from content?

TLD is utilized in the way that it is used to set the default encoding which is chosen for the encoding of the document if auto-detection fails to work or doesn't get triggered for whatever reason. So it is involved in the overall decoding process but not exactly in detection step itself.

How many bytes do you feed to the detector?

To be precise, that's out of the scope of the detector but rather how Blink decoder is designed. It feeds the first chunk of the document received from network, which in almost most cases big enough for the detector to come up with the right guess.

What happens if the network stalls before that many bytes have been received? Do you wait or do you stop detecting based on a timer? (Timers have their own problems, and Firefox got rid of the timer in Firefox 4.)
Once the detector has seen as many bytes as you usually feed it, do you continue feeding data to the detector to allow it to revise its guess? If yes, what do you do if the detector revises its guess?

As answered above, Blink is designed to feed the detector with the first chunk of the data only. After that, no more data is fed to it, therefore no revising its guess either. There is a way to artificially force the first chunk to be small so as not to have significant part of the document content but only have tags and scripts that can be viewed in ascii, which will trick the detector into thinking it is in plain ascii. But it would rarely happen in real life.
One possible bug I'm aware of it that it is possible that a document is written in plain ascii in most of its text but has a couple of symbols in UTF-8, say, at the end of it. The detector guesses it is in ascii because it never sees the symbols if it is not in the first chunk. https://bugs.chromium.org/p/chromium/issues/detail?id=233002

Other than ISO-2022-JP (per your previous comment) and, I'm extrapolating, EUC-JP, can the detector (as used in Blink) guess an encoding that wasn't previously the default for any Chrome localization? That is, does the detector make pages "work" when they previously didn't for any localization defaults?

Not sure if I got the question right - AFAICT, Blink respects the detected encoding and uses it for decoding documents regardless of the localization default.

Can the detector (as used in Blink) guess UTF-8? That is, can Blink's use of the detector inadvertently encourage the non-labeling of newly-created UTF-8 content?

Yes it can also detect UTF-8. I wouldn't view it 'inadvertently encourage' non-labelling though. Most of the modern web sites go with UTF-8, and label the documents well. I don't have the stats on how it actually contributes to them neglecting labelling properly. The detector is more for legacy web sites.

[hsivonen]

Realized these questions better be answered by the person who ported the new detector to Blink

Thank you for the answer.

TLD is utilized in the way that it is used to set the default encoding which is chosen for the encoding of the document if auto-detection fails to work or doesn't get triggered for whatever reason.

Have you estimated the success rate of looking at the TLD only versus looking at the first network buffer and falling back to looking at the TLD? How significant a win is it to look inside the first network buffer when you have the capability to look at the TLD?

Do you guess windows-1252 for the new TLDs, like Gecko does?

It feeds the first chunk of the document received from network, which in almost most cases big enough for the detector to come up with the right guess.

I'm quite unhappy about Blink reintroducing a dependency on network buffer boundaries in text/html handling without discussion at the WHATWG prior to making the change.

For historical context, prior to Firefox 4 text/html handling in Firefox was dependent on where in the input stream the buffer boundaries fell. This was bad, and getting rid of this, thanks to the inside from WebKit, was one of the accomplishments of the HTML parsing standardization effort at the WHATWG.

There is a way to artificially force the first chunk to be small so as not to have significant part of the document content but only have tags and scripts that can be viewed in ascii, which will trick the detector into thinking it is in plain ascii. But it would rarely happen in real life.

Unfortunately, the Web is vast enough for rare cases to be real.

One possible bug I'm aware of it that it is possible that a document is written in plain ascii in most of its text but has a couple of symbols in UTF-8, say, at the end of it. The detector guesses it is in ascii because it never sees the symbols if it is not in the first chunk.

When the guess is "ASCII", does it result in a windows-1252 decoder being instantiated? Or the decoder being chosen based on the TLD? Or something else?

https://bugs.chromium.org/p/chromium/issues/detail?id=233002

Even if the HTML parser has only seen ASCII by the time it revises its encoding detection guess, the encoding may already have been inherited to CSS, JS and iframes, so revising the guess without causing a full reload seems like a bad idea.

Other than ISO-2022-JP (per your previous comment) and, I'm extrapolating, EUC-JP, can the detector (as used in Blink) guess an encoding that wasn't previously the default for any Chrome localization? That is, does the detector make pages "work" when they previously didn't for any localization defaults?

Not sure if I got the question right - AFAICT, Blink respects the detected encoding and uses it for decoding documents regardless of the localization default.

I meant that there is a set of encodings that were the default in some localization and a set of encodings that were not to the default in any localization. When the guess is from the first set, the behavior that ensues matches the behavior of some pre-existing default configuration. When the guess is from the second set, a new Web-exposed behavior has been introduced. Apart from Japanese encodings, does Blink ever guess an encoding from the second set (e.g. ISO-8859-13)?

Yes it can also detect UTF-8. I wouldn't view it 'inadvertently encourage' non-labelling though.

The vast majority of Web authors have not read your comment here and don't know how Blink works. Since Web authors do what seem to work, they don't know that UTF-8 sniffing is unreliable until it fails them. As noted in your comment about the case where the first network buffer is all ASCII, UTF-8 sniffing in Blink is unreliable. Some Web authors are going to feel free not to label UTF-8 as long as they haven't experienced the failure yet (i.e. they only test in Blink and e.g. use a language that makes it probable for <title> to have some non-ASCII). This is bad.

(Detecting UTF-8 for file:-URL HTML by looking at the entire file is fine, IMO, and I want to implement that in a Gecko in due course.)

[JinsukKim]

Please note that I'm not working on Blink. And I ported CED as a drop-in replacement of ICU encoding detector for any Blink-based Web browser to benefit from it. I'm not entirely familiar with all the terms used in your questions.

Do you guess windows-1252 for the new TLDs, like Gecko does?

What are 'new' TLDs? What I can say is Blink makes use of a typical set of TLDs and mappings to default encoding which can be found in Chromium/Android resource. Mapping for countries whose legacy, default encoding is not obvious (ge-Georgian, id-Indonesian, for instance) are not included.

When the guess is "ASCII", does it result in a windows-1252 decoder being instantiated? Or the decoder being chosen based on the TLD? Or something else?

Yes it goes with window-1252.

I meant that there is a set of encodings that were the default in some localization and a set of encodings that were not to the default in any localization. When the guess is from the first set, the behavior that ensues matches the behavior of some pre-existing default configuration. When the guess is from the second set, a new Web-exposed behavior has been introduced. Apart from Japanese encodings, does Blink ever guess an encoding from the second set (e.g. ISO-8859-13)?

I don't know what belongs to the second set. Maybe you can see it for yourself here about what the detector can return: https://cs.chromium.org/chromium/src/third_party/ced/src/compact_enc_det/compact_enc_det_unittest.cc?l=4733

The vast majority of Web authors have not read your comment here and don't know how Blink works. Since Web authors do what seem to work, they don't know that UTF-8 sniffing is unreliable until it fails them. As noted in your comment about the case where the first network buffer is all ASCII, UTF-8 sniffing in Blink is unreliable. Some Web authors are going to feel free not to label UTF-8 as long as they haven't experienced the failure yet (i.e. they only test in Blink and e.g. use a language that makes it probable for <title> to have some non-ASCII). This is bad.

I can see your point. What would be your suggestion?