[Breaking change request] Change UTF-8 encoder and decoder to match the WHATWG encoding standard

[askeksa-google]

Summary

Change encoding and decoding of UTF-8 to conform to the WHATWG encoding standard. This means that it will never emit invalid UTF-8, only accept valid UTF-8 and be compatible with the TextEncoder and TextDecoder classes in JavaScript.

Related issues: #7046, #22330, #28832, #31370, #31954

What is changing:

• When decoding UTF-8 data with the Utf8Codec or Utf8Decoder class, the input is considered malformed if it contains an encoded surrogate character (code point in the range U+D800-U+DFFF, encoded in UTF-8 as a 3-byte character encoding where the first byte is 0xED and the second byte is in the range 0xA0-0xBF).
• When encoding a string as UTF-8 with the Utf8Codec or Utf8Encoder class, and the string contains an unpaired surrogate, that surrogate is emitted as a replacement character (U+FFFD, encoded in UTF-8 as 0xEF, 0xBF, 0xBD) instead of an encoded surrogate (which is invalid UTF-8). For chunked conversion, if a chunk ends with a high surrogate and the next chunk starts with a low surrogate, these surrogates are considered properly paired and are combined, like before.
• When decoding malformed UTF-8 data with allowMalformed set to true, the number of replacement characters emitted will sometimes differ from the number currently emitted. Specifically, the decoder will emit one replacement character for each maximal sequence of input bytes that is either
  1. a prefix of a valid encoding of a character, or
  2. a single byte that is not a prefix of a valid encoding of a character.
• When decoding malformed UTF-8 data with allowMalformed set to false, the offset in the resulting FormatException will point to the first byte from which the decoder can conclude that the sequence is malformed, rather than the first byte that was not decoded successfully. Also, the message of the FormatException will sometimes be different from what it is currently. If the input contains more than one error, the FormatException may point to a different error than before.

Why is this changing?

Dart strings (like JS and Java strings) may contain unpaired surrogates. The current strategy of allowing surrogates when encoding and decoding UTF-8 ensures that any Dart string can be encoded as UTF-8 (actually, WTF-8) and decoded back into the original string.

This strategy has a number of drawbacks:

• The output of the UTF-8 encoder is sometimes not valid UTF-8, which can be problematic when this data needs to be consumed by other programs.
• When UTF-8 data is read, and that data contains encoded unpaired surrogates, this may cause problems much later, when the string is processed, rather that catching the invalid encoding up front.
• The Dart behavior deviates from JS, which means that when Dart code is translated to JS, UTF-8 encoding and decoding can't directly use the JS TextEncoder and TextDecoder classes. It must do some or all of the conversion in Dart code, which has a significant performance cost.
• Retaining exact compatibility with the current error behavior complicates some planned optimizations to UTF-8 decoding in the Dart VM.

The purpose of the change is thus to:

• ensure that Dart programs don't inadvertently produce or accept invalid UTF-8.
• enable faster UTF-8 encoding and decoding for both JS and VM targets.

Expected impact

Programs manipulating strings through usual string operations are unlikely to be affected.

A program may be affected by this change if it does any of the following:

1. Manipulates strings in a way that may introduce unpaired surrogates, encodes these strings as UTF-8, decodes them again and expects the string contents to be preserved.
2. Encodes arbitrary substrings as UTF-8 without regard to surrogate pairs, decodes them again, concatenates them (before or after decoding) and expects the string contents to be preserved.
3. Relies on the exact offsets and/or error messages from decoding invalid UTF-8 data or which error is reported in case of multiple errors.
4. Relies on the number of replacement characters produced by decoding invalid UTF-8 data.

Mitigation

For the scenarios listed above:

1. UTF-8, being an interchange format, is unsuited for representing such broken strings. Consider a different representation.
2. For encoding in multiple chunks, use the chunked conversion API.
3. Adjust the program to detect specific errors in a different way, or adapt it to the new errors.
4. Same as 3.

Variations

An optional allowSurrogates parameter could be added to the encoder and decoder to support the round-trip use case. To obtain the performance benefits, it should default to false. This could introduce further breakage for programs implementing the Utf8Codec interface (unless we only put the flag on the constructors).

If the surrogate change is considered too risky, the error and replacement character changes on their own can still ease the VM optimizations and possibly improve the performance of JS when allowMalformed is set to true.

[franklinyow]

Please send the announce
https://github.com/dart-lang/sdk/blob/master/docs/process/breaking-changes.md

The change will need to wait for the beta branch snapped before it can be submitted.

[mkustermann]

The breaking change has been announced on dart-announce, see announcement.

[mkustermann]

The change will need to wait for the beta branch snapped before it can be submitted.

The beta branch has been cut, so this is not blocked anymore.

We would like to move forward with this.

@franklinyow Can you help us getting the LGTM from all veto powers?

[franklinyow]

cc @Hixie @matanlurey @dgrove @vsmenon for review and approval.

[vsmenon]

Is there an impact on prod code size for web or native? ( @rakudrama )

[askeksa-google]

@rakudrama and I just discussed the web situation. There is currently a 4k size increase and a significant speed regression for small inputs, but we have ideas for fixing both issues.

For native there will probably be a small (a few k) constant size increase.

[askeksa-google]

One detail that we need to decide on: for JS, when allowMalformed is set to true, we can choose either to fall back to the Dart implementation like we do currently, or we can use the TextDecoder (when available).

Using TextDecoder is significantly faster, but it might result in different output between browsers, since not all browsers follow the WHATWG standard precisely. Experiments indicate that:

• Firefox follows the standard.
• Chrome (and Chromium-based browsers): When an unfinished sequence is interrupted by end-of-input, or when an unfinished 4-byte sequence is interrupted after exactly 2 bytes, then one replacement character is emitted per byte instead of just one.
• Safari: When a valid first byte of a multi-byte sequence is followed by fewer bytes than necessary to complete the sequence and then end-of-input, a replacement character is emitted and the rest of the input is discarded, even if it contains valid sequences.

@lrhn @rakudrama @vsmenon WDYT?

[lrhn]

Use TextDecoder and file bugs against Chrome and Safari.
(And mark the corresponding tests as failing on those browsers).

I don't think the exact behavior on malformed inputs is that important. We don't otherwise try to work around browser bugs.

[rakudrama]

Using TextDecoder for short inputs is slower than the Dart implementation, so we will likely use a hybrid approach, using the backup Dart implementation for short inputs, where the threshold depends on some arguments.
@lrhn How do you feel about the bug existing for some inputs but not others?

@askeksa-google, is the following statement true: All cases of non-standard behaviour have an output containing the replacement character.
If so, we have a relatively simple remedy: in the allowMalformed case, check the output for the replacement character and if present, redo the conversion with the backup dart code.
It might be less work to do this than split the tests to keep good test coverage.

We do work around some other browser bugs - for example, fixing the unusual exponential form in IE11's Number.toString(radix), and various differences in the DOM APIs.

[askeksa-google]

That statement is true. A stricter statement which is also true is: All cases of non-standard behaviour have an output containing the replacement character as the last character or two replacement characters in a row.

The "when an unfinished 4-byte sequence is interrupted after exactly 2 bytes" deviation in Chromium is unfortunate, since without that (assuming only the other observed deviations) we would only have to check the last character of the output.

For mostly-ASCII inputs, it might very well be faster to always fall back than to scan the result. At least the cross-over point will be quite high.

[lrhn]

@lrhn How do you feel about the bug existing for some inputs but not others?

I feel fine. File bug reports against the browsers and hope they get around to fixing it.

It's still only error handling, it's not like they do something different for a well-formed input.

[Hixie]

Having standards-compliant error handling can be as important as handling valid input. For one, variance in error handling is often a vector in vulnerabilities. For another, most people don't care if input is valid or not. They just care that all their software interoperates.

[lrhn]

Having consistent behavior is a good thing, but there is a limit to how much work should be spent on mitigating browser bugs. I don't think the impact here is big enough to warrant adding detection and workarounds for something which is, arguably, already an error situation.

[askeksa-google]

I benchmarked a post-scan for replacement characters. Here it adds about 6ns per byte on top of the TextDecoder time of 2-6ns per byte. In other words, it is a 4x slowdown for ASCII strings and 2x for the most complex strings. Still faster than the Dart version, though (for long strings).