Wrong length of surrogate Unicode codepoints

[mlocati]

Description

The following code:

<?php

echo 'SUN BEHIND CLOUD: ', mb_strlen("\u{26C5}"), "\n";

echo 'Non Private Use High Surrogate, First: ', mb_strlen("\u{D800}"), "\n";
echo 'Non Private Use High Surrogate, Last: ', mb_strlen("\u{DB7F}"), "\n";

echo 'Private Use High Surrogate, First: ', mb_strlen("\u{DB80}"), "\n";
echo 'Private Use High Surrogate, Last: ', mb_strlen("\u{DBFF}"), "\n";

echo 'Low Surrogate, First: ', mb_strlen("\u{DC00}"), "\n";
echo 'Low Surrogate, Last: ', mb_strlen("\u{DFFF}"), "\n";

In PHP 8.1 & 8.2 we have:

SUN BEHIND CLOUD: 1
Non Private Use High Surrogate, First: 1
Non Private Use High Surrogate, Last: 1
Private Use High Surrogate, First: 1
Private Use High Surrogate, Last: 1
Low Surrogate, First: 1
Low Surrogate, Last: 1

In PHP 8.3 we instead have:

SUN BEHIND CLOUD: 1
Non Private Use High Surrogate, First: 3
Non Private Use High Surrogate, Last: 3
Private Use High Surrogate, First: 3
Private Use High Surrogate, Last: 3
Low Surrogate, First: 3
Low Surrogate, Last: 3

PHP Version

8.3.0-dev (81e236c)

Operating System

Ubuntu 22.04.3

[alexdowad]

What is the value of mb_internal_encoding()?

[mlocati]

I have these lines in my bootstrap file:

mb_regex_encoding('UTF-8');
ini_set('mbstring.script_encoding', 'pass');
mb_internal_encoding('UTF-8');
mb_substitute_character('none');

[alexdowad]

Just quickly stepped through in a debugger.

U+D800 is a reserved, "illegal" codepoint which never appears in valid UTF-8 text. When our new UTF-8-to-codepoint conversion code sees the illegal sequence 0xEDA080, it emits an error marker (mb_substitute_char()) and then tries to reprocess 0xA080... which results in more error markers, thus giving you the output value 3.

If I remember well, I think this behavior originates from when we started complying with the W3C's specification for handling of UTF-8 text: https://www.w3.org/International/docs/encoding/#utf-8-decoder

[alexdowad]

Ah, yes... this is a result of this rule in the W3C specification:

If byte is 0xED, set utf-8 upper boundary to 0x9F.

Then the second byte in the sequence, 0xA0, will be over the "UTF-8 upper boundary", and the specification requires us to reprocess it as the start of a new character.

[alexdowad]

I think what may be needed here is to add a note in UPGRADING that UTF-8 text handling now obeys the W3C specification, and this may result in different output for invalid UTF-8 strings.

[edegaudenzi]

Fairly old issue, but I've stepped today on something similar with actual utf chars. mb_strlen() returns different lengths depending on the php version. Not sure to understand what's going on here.

<?php
echo 'mb_internal_encoding: '.mb_internal_encoding().PHP_EOL;
echo 'strlen(我): '.strlen('我').PHP_EOL;
echo 'mb_strlen(我): '.mb_strlen('我').PHP_EOL;
echo 'mb_strlen(我, utf16): '.mb_strlen('我', 'UTF-16').PHP_EOL;

[alexdowad]

@edegaudenzi, you are trying to check the length of a UTF-8 string while setting the $encoding parameter to UTF-16. UTF-8 and UTF-16 are completely different text encodings and a UTF-8 string is not valid UTF-16 text.

To avoid bugs (and in some cases, even security vulnerabilities), it is advisable to check that strings are valid in your desired encoding before operating on them. You can do this with mb_check_encoding.

If you find that mb_strlen returns different values for a valid UTF-16 string, please let me know.