Skip to content

Require ext-mbstring, which the parser already needs #278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Amoifr wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Require ext-mbstring, which the parser already needs #278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions composer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
],
"require" : {
"ext-dom": "*",
"ext-mbstring": "*",
"php" : ">=7.4"
},
"require-dev": {
Expand Down
58 changes: 16 additions & 42 deletions src/HTML5/Parser/UTF8Utils.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

use Masterminds\HTML5\Exception;

class UTF8Utils
{
Expand All @@ -38,28 +37,15 @@ class UTF8Utils

/**
* Count the number of characters in a string.
* UTF-8 aware. This will try (in order) iconv, MB, and finally a custom counter.
* UTF-8 aware.
*
* @param string $string
*
* @return int
*/
public static function countChars($string)
{
// Get the length for the string we need.
if (function_exists('mb_strlen')) {
return mb_strlen($string, 'utf-8');
}

if (function_exists('iconv_strlen')) {
return iconv_strlen($string, 'utf-8');
}

$count = count_chars($string);

// 0x80 = 0x7F - 0 + 1 (one added to get inclusive range)
// 0x33 = 0xF4 - 0x2C + 1 (one added to get inclusive range)
return array_sum(array_slice($count, 0, 0x80)) + array_sum(array_slice($count, 0xC2, 0x33));
return mb_strlen($string, 'utf-8');
}

/**
Expand Down Expand Up @@ -88,32 +74,20 @@ public static function convertToUTF8($data, $encoding = 'UTF-8')
// details for the bug are on http://us1.php.net/manual/en/function.iconv.php#108643
// which contains links to the actual but reports as well as work around
// details.
if (function_exists('mb_convert_encoding')) {
// mb library has the following behaviors:
// - UTF-16 surrogates result in false.
// - Overlongs and outside Plane 16 result in empty strings.

// Before we run mb_convert_encoding we need to tell it what to do with
// characters it does not know. This could be different than the parent
// application executing this library so we store the value, change it
// to our needs, and then change it back when we are done. This feels
// a little excessive and it would be great if there was a better way.
$save = mb_substitute_character();
mb_substitute_character('none');
$data = mb_convert_encoding($data, 'UTF-8', $encoding);
mb_substitute_character($save);
}
// @todo Get iconv running in at least some environments if that is possible.
elseif (function_exists('iconv') && 'auto' !== $encoding) {
// fprintf(STDOUT, "iconv found\n");
// iconv has the following behaviors:
// - Overlong representations are ignored.
// - Beyond Plane 16 is replaced with a lower char.
// - Incomplete sequences generate a warning.
$data = @iconv($encoding, 'UTF-8//IGNORE', $data);
} else {
throw new Exception('Not implemented, please install mbstring or iconv');
}
//
// mb library has the following behaviors:
// - UTF-16 surrogates result in false.
// - Overlongs and outside Plane 16 result in empty strings.

// Before we run mb_convert_encoding we need to tell it what to do with
// characters it does not know. This could be different than the parent
// application executing this library so we store the value, change it
// to our needs, and then change it back when we are done. This feels
// a little excessive and it would be great if there was a better way.
$save = mb_substitute_character();
mb_substitute_character('none');
$data = mb_convert_encoding($data, 'UTF-8', $encoding);
mb_substitute_character($save);

/*
* One leading U+FEFF BYTE ORDER MARK character must be ignored if any are present.
Expand Down