`UTF8.getu` should validate that it's input is UTF-8.

[Skyb0rg007]

Description

When decoding UTF8, it is important to avoid possible security issues, as detailed in UTR #36.

The UTF8 module provided by the SML/NJ library is meant to be used for working with UTF-8 encoded strings.
However it fails to facilitate secure coding standards in regards to the Unicode Security Considerations linked above.
Side note: while this is a feature request, it may want to be moved to a bug fix, since the use of the decoding functions as they exist is potentially unsafe.

Non-shortest form

From UTR #36:

3.1 UTF-8 Exploits

There are three equivalent encoding forms for Unicode: UTF-8, UTF-16, and UTF-32. UTF-8 is commonly used in XML and HTML; UTF-16 is the most common in program APIs; and UTF-32 is the best for representing single characters. While these forms are all equivalent in terms of the ability to express Unicode, the original usage of UTF-8 was open to a canonicalization exploit.

Originally, Unicode forbade the generation of "non-shortest form" UTF-8, but not the interpretation of "non-shortest form" UTF-8. This was fixed in Unicode 3.0, because security issues can arise when software does interpret the non-shortest forms. For example:

Process A performs security checks, but does not check for non-shortest forms.
Process B accepts the byte sequence from process A, and transforms it into UTF-16 while interpreting non-shortest forms.
The UTF-16 text may then contain characters that should have been filtered out by process A.

For example, the backslash character "\" can often be a dangerous character to let through a gatekeeper, because it can be used to access different directories. Thus a gatekeeper might specifically prevent it from getting through. The backslash is represented in UTF-8 as the byte sequence <5C>. However, as a non-shortest form, backslash could also be represented as the byte sequence<C1 9C>. When a gatekeeper does not check for non-shortest form, this situation can lead to a severe security breach.

To address this issue, the Unicode Technical Committee modified the definition of UTF-8 in Unicode 3.1 to forbid conformant implementations from interpreting non-shortest forms for BMP characters, and clarified some of the conformance clauses.

This is not implemented in the UTF8 module:

- UTF8.implode (UTF8.explode "\193\156"); (* C1, 9C *)
val it = "\\" : string

Surrogate codepoints

Page 125 of the Unicode Standard

Because surrogate code points are not Unicode scalar values, any UTF-8 byte
sequence that would otherwise map to code points U+D800..U+DFFF is ill-formed.

This is not implemented in the UTF8 module:

- UTF8.explode "\237\160\128"; (* ED, A0, 80 *)
val it = [0wxD800] : UTF8.wchar list

Out-of-range

Page 125 of the Unicode Standard

Table 3-7 lists all of the byte sequences that are well-formed in UTF-8. A range of byte val-
ues such as A0..BF indicates that any byte from A0 to BF (inclusive) is well-formed in that
position. Any byte value outside of the ranges listed is ill-formed. For example:
• The byte sequence is ill-formed, because C0 is not well-formed in the
“First Byte” column.
• The byte sequence <E0 9F 80> is ill-formed, because in the row where E0 is
well-formed as a first byte, 9F is not well-formed as a second byte.
• The byte sequence <F4 80 83 92> is well-formed, because every byte in that
sequence matches a byte range in a row of the table (the last row).
In Table 3-7, cases where a trailing byte range is not 80..BF are shown in bold italic to draw
attention to them. These exceptions to the general pattern occur only in the second byte of
a sequence.

Table 3-7. Well-Formed UTF-8 Byte Sequences

Code Points	First Byte	Second Byte	Third Byte	Fourth Byte
U+0000..U+007F	00..7F
U+0080..U+07FF	C2..DF	80..BF
U+0800..U+0FFF	E0	A0..BF	80..BF
U+1000..U+CFFF	E1..EC	80..BF	80..BF
U+D000..U+D7FF	ED	80..9F	80..BF
U+E000..U+FFFF	EE..EF	80..BF	80..BF
U+10000..U+3FFFF	F0	90..BF	80..BF	80..BF
U+40000..U+FFFFF	F1..F3	80..BF	80..BF	80..BF
U+100000..U+10FFFF	F4	80..8F	80..BF	80..BF

This is not implemented in the UTF8 module:

- UTF8.explode "\247\191\191\191"; (* F7, BF, BF, BF *)
val it = [0wx1FFFFF] : UTF8.wchar list
- UTF8.maxCodePoint;
val it = 0wx10FFFF : UTF8.wchar list

[JohnReppy]

This issue has been addressed by pull request #277. The fix will also be merged into the legacy version.