UTF-8 decoder produces incorrect number of replacement characters

[chykon]

Dart SDK version: 2.18.0 (stable) (Fri Aug 26 10:22:54 2022 +0000) on "linux_x64"

At https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt you can find a fairly well-known stress test for the UTF-8 decoder. If you try to decode this file using Dart, then in paragraph 3.3.7 there will be an insufficient number of replacement characters. Each malformed byte is expected to be replaced by a single U+FFFD character.

Minimal example available: https://dartpad.dev/?id=36ce63525a3b532328afd12de900d45a

[rakudrama]

The Dart UTF-8 decoder used to generate one replacement character per malformed byte. We changed it to match the web standard. See #41100 for more details.

Since the current behavior is a deliberate choice, I suggest closing this issue as "by design".

/cc @lrhn @askeksa-google

[lrhn]

I concur.

From the linked UTF-8-test.txt:

This file is not meant to be a conformance test. It does not prescribe any particular outcome.