Not encoding UTF-8 correctly #66

shtse8 · 2017-03-01T20:20:09Z

I am a Chinese developer and making Chinese website.

Code to reproduce:

$html = <<<EOF
<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
<body>
网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
</body>
</html>
EOF;
$dom = FluentDOM::QueryCss($html, 'text/html');
echo $dom;

Result:

<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8"><title>&#32593;&#21451;&#32456;&#20110;&#32905;&#25628;&#20986;&#12300;&#33539;&#20912;&#20912;&#12301;&#23478;&#26063;&#29031;&#29255;&#65292;&#27809;&#24819;&#21040;&#30475;&#35265;&#22905;&#22902;&#22902;&#25165;&#21457;&#29616;&#12300;&#33539;&#20912;&#20912;&#26159;&#20840;&#23478;&#26368;&#38590;&#30475;&#30340;&#12301;&#65281;</title></head><body>
&#32593;&#21451;&#32456;&#20110;&#32905;&#25628;&#20986;&#12300;&#33539;&#20912;&#20912;&#12301;&#23478;&#26063;&#29031;&#29255;&#65292;&#27809;&#24819;&#21040;&#30475;&#35265;&#22905;&#22902;&#22902;&#25165;&#21457;&#29616;&#12300;&#33539;&#20912;&#20912;&#26159;&#20840;&#23478;&#26368;&#38590;&#30475;&#30340;&#12301;&#65281;
</body></html>

Expected Result:

<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
<body>
网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
</body></html>

It is a known bug of PHP DomDocument. Here is the reference:
http://stackoverflow.com/questions/8218230/php-domdocument-loadhtml-not-encoding-utf-8-correctly

We should get the UTF-8 result instead of getting HTML-ENTITIES result. It doesn't make sense to get the final html with full of encoded utf-8 and making the size much larger.

The text was updated successfully, but these errors were encountered:

ThomasWeinert · 2017-03-02T18:23:33Z

Right, I will check and try to fix this.

shtse8 · 2017-03-03T11:52:13Z

Not only encoded utf-8 is returned. Garbled code is returned using html()

Reproduce:

$html = '<div><p>Paragraph 1</p> <p>Paragraph 2</p><p>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</p></div>';
$doc = FluentDOM($html, 'html-fragment');
echo $doc->html();

Expected:

<p>Paragraph 1</p> <p>Paragraph 2</p><p>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</p>

Actual:

<p>Paragraph 1</p> <p>Paragraph 2</p>
<p>ç½�å��ç»�äº�è��æ��å�ºã��è��å�°å�°ã��å®¶æ��ç�§ç��ï¼�æ²¡æ�³å�°ç��è§�å¥¹å¥¶å¥¶æ��å��ç�°ã��è��å�°å�°æ�¯å�¨å®¶æ��é�¾ç��ç��ã��ï¼�</p>

and also with text()

echo $doc->text();

Result:

Paragraph 1 Paragraph 2ç½�å��ç»�äº�è��æ��å�ºã��è��å�°å�°ã��å®¶æ��ç�§ç��ï¼�æ²¡æ�³å�°ç��è§�å¥¹å¥¶å¥¶æ��å��ç�°ã��è��å�°å�°æ�¯å�¨å®¶æ��é�¾ç��ç��ã��ï¼�

while:

echo $doc;

Returns encoded utf-8:

<div><p>Paragraph 1</p> <p>Paragraph 2</p><p>&ccedil;&frac12;&#145;&aring;&#143;&#139;&ccedil;&raquo;&#136;&auml;&ordm;&#142;&egrave;&#130;&#137;&aelig;&#144;&#156;&aring;&#135;&ordm;&atilde;&#128;&#140;&egrave;&#140;&#131;&aring;&#134;&deg;&aring;&#134;&deg;&atilde;&#128;&#141;&aring;&reg;&para;&aelig;&#151;&#143;&ccedil;&#133;&sect;&ccedil;&#137;&#135;&iuml;&frac14;&#140;&aelig;&sup2;&iexcl;&aelig;&#131;&sup3;&aring;&#136;&deg;&ccedil;&#156;&#139;&egrave;&sect;&#129;&aring;&yen;&sup1;&aring;&yen;&para;&aring;&yen;&para;&aelig;&#137;&#141;&aring;&#143;&#145;&ccedil;&#142;&deg;&atilde;&#128;&#140;&egrave;&#140;&#131;&aring;&#134;&deg;&aring;&#134;&deg;&aelig;&#152;&macr;&aring;&#133;&uml;&aring;&reg;&para;&aelig;&#156;&#128;&eacute;&#154;&frac34;&ccedil;&#156;&#139;&ccedil;&#154;&#132;&atilde;&#128;&#141;&iuml;&frac14;&#129;</p></div>

shtse8 · 2017-03-03T12:00:35Z

But it is fine using text('some strings'), correct utf-8 is returned.

Code:

$doc = FluentDOM('<div></div>', 'html-fragment');
$doc->text('网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！');
echo $doc->text();

Result:

网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！

but html('some string') is still failed.

Code:

$doc = FluentDOM('<div></div>', 'html-fragment');
$doc->html('网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！');
echo $doc->html();

Result:

ç½�å��ç»�äº�è��æ��å�ºã��è��å�°å�°ã��å®¶æ��ç�§ç��ï¼�æ²¡æ�³å�°ç��è§�å¥¹å¥¶å¥¶æ��å��ç�°ã��è��å�°å�°æ�¯å�¨å®¶æ��é�¾ç��ç��ã��ï¼�

I really like this library very much. It is powerful and provide lots of methods to handle each different DOM. But unluckily, because of this encoding issue, I cannot use it because all my html and html-fragment are full of Chinese. Hope it can be fixed soon. Thanks for your contribution.

ThomasWeinert · 2017-03-03T14:04:27Z

Give me one or two weekends please :-)

I will try to add the issues as unit tests to make them reproducible.

ThomasWeinert · 2017-03-04T09:58:14Z

I just pushed FluentDOM/FluentDOM@370e98f

This is not the final fix, but it should improve the behavior.

shtse8 · 2017-03-06T17:32:04Z

I want to try your new push, but I cannot use the fluentdom and selectors-phpcss in the same project.
It seems selectors-phpcss specify a version to make my composer not to install the new push of fluentdom. Do you have any idea to update to the latest push using composer?

"require": {
		"fluentdom/fluentdom": "dev-master#e2a47d5",
		"fluentdom/selectors-phpcss": "^1.0"
    }

when composer update:

Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Conclusion: remove fluentdom/fluentdom dev-master
    - fluentdom/selectors-phpcss 1.0.0 requires fluentdom/fluentdom ^5.3 -> satisfiable by fluentdom/fluentdom[5.3.x-dev].
    - fluentdom/selectors-phpcss 1.0.1 requires fluentdom/fluentdom ^5.3||^6.0 -> satisfiable by fluentdom/fluentdom[5.3.x-dev, 6.0.x-dev, 6.1.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 5.3.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 6.0.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 6.1.x-dev].
    - Installation request for fluentdom/fluentdom dev-master#e2a47d5 -> satisfiable by fluentdom/fluentdom[dev-master].
    - Installation request for fluentdom/selectors-phpcss ^1.0 -> satisfiable by fluentdom/selectors-phpcss[1.0.0, 1.0.1].

I can install the new push without the selectors-phpcss, but I need it to test.

shtse8 · 2017-03-06T17:43:27Z

Anyway, I have downloaded the lastest clone and replaced the whole folder manually. I can test now.
It seems the behavior is the same using FluentDOM::QueryCss, but it is fine using new HTML().

shtse8 · 2017-03-19T18:49:46Z

Hi, Any updates?

ThomasWeinert · 2017-03-26T11:36:35Z

I added some RegEx to the HTML loader to fetch the encoding from the meta tags - default is UTF-8 now. Additionally I did a lot of rework on the load/save process from HTML - testing it with Chinese characters. The changes are pushed to the 6.1 branch and I added the version to the CSS Selector package, so a composer install allowing the dev versions should work now.

If you could test it out and send me (small) examples that do not work as expected, I would appreciate it.

shtse8 · 2017-03-26T17:17:19Z

@ThomasWeinert Hi, could you give me a sample of composer.json to include the dev version? I have tried the following but without any luck.

{
	"repositories": [
      {
        "type": "vcs",
        "url": "https://github.com/FluentDOM/FluentDOM"
      }
    ],
	"require": {
		"fluentdom/fluentdom": "dev-master#6.1",
		"fluentdom/selectors-phpcss": "^1.0"
    }
}

Result:

Loading composer repositories with package information                                                                                                                                 Updating dependencies (including require-dev)         Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Conclusion: don't install fluentdom/fluentdom dev-master
    - fluentdom/selectors-phpcss 1.0.1 requires fluentdom/fluentdom ^5.3||^6.0 -> satisfiable by fluentdom/fluentdom[5.3.x-dev, 6.0.x-dev, 6.1.x-dev].
    - fluentdom/selectors-phpcss 1.0.0 requires fluentdom/fluentdom ^5.3 -> satisfiable by fluentdom/fluentdom[5.3.x-dev].
    - fluentdom/selectors-phpcss 1.0.1 requires fluentdom/fluentdom ^5.3||^6.0 -> satisfiable by fluentdom/fluentdom[5.3.x-dev, 6.0.x-dev, 6.1.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 5.3.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 6.0.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 6.1.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 5.3.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 6.0.x-dev].
    - Can only install one of: fluentdom/fluentdom[dev-master, 6.1.x-dev].
    - Installation request for fluentdom/fluentdom dev-master#6.1 -> satisfiable by fluentdom/fluentdom[dev-master].
    - Installation request for fluentdom/selectors-phpcss ^1.0 -> satisfiable by fluentdom/selectors-phpcss[1.0.0, 1.0.1].

and, I also tried.

{
	"require": {
		"fluentdom/fluentdom": "^6.1",
		"fluentdom/selectors-phpcss": "^1.0"
    }
}

Result:

Loading composer repositories with package information                                                                                                                                 Updating dependencies (including require-dev)         Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package fluentdom/fluentdom could not be found in any version, there may be a typo in the package name.
  Problem 2
    - fluentdom/selectors-phpcss 1.0.1 requires fluentdom/fluentdom ^5.3||^6.0 -> no matching package found.
    - fluentdom/selectors-phpcss 1.0.0 requires fluentdom/fluentdom ^5.3 -> no matching package found.
    - fluentdom/selectors-phpcss 1.0.1 requires fluentdom/fluentdom ^5.3||^6.0 -> no matching package found.
    - Installation request for fluentdom/selectors-phpcss ^1.0 -> satisfiable by fluentdom/selectors-phpcss[1.0.0, 1.0.1].

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.

Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

and, I also tried.

{
	"repositories": [
      {
        "type": "vcs",
        "url": "https://github.com/FluentDOM/FluentDOM"
      }
    ],
	"require": {
		"fluentdom/fluentdom": "dev-6.1",
		"fluentdom/selectors-phpcss": "^1.0.1"
    }
}

Result:

Loading composer repositories with package information                                                                                                                                 Updating dependencies (including require-dev)         Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package fluentdom/fluentdom dev-6.1 could not be found.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.

Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

shtse8 · 2017-03-26T19:10:13Z

Okay, I can use it now with the following composer.json.

{
	"repositories": [
      {
        "type": "vcs",
        "url": "https://github.com/FluentDOM/FluentDOM"
      }
    ],
	"require": {
		"fluentdom/fluentdom": "dev-master as 6.1",
		"fluentdom/selectors-phpcss": "^1.0.1"
    }
}

shtse8 · 2017-03-26T19:20:59Z

Code to reproduce

$doc = FluentDOM('<div></div>', 'html-fragment');
$doc->html(FluentDOM('网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！', 'html-fragment'));
echo $doc;

Expected Result:

<div>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</div>

Actual Result:

<div>Ã§Â½Â�Ã¥Â�Â�Ã§Â»Â�Ã¤ÂºÂ�Ã¨Â�Â�Ã¦Â�Â�Ã¥Â�ÂºÃ£Â�Â�Ã¨Â�Â�Ã¥Â�Â°Ã¥Â�Â°Ã£Â�Â�Ã¥Â®Â¶Ã¦Â�Â�Ã§Â�Â§Ã§Â�Â�Ã¯Â¼Â�Ã¦Â²Â¡Ã¦Â�Â³Ã¥Â�Â°Ã§Â�Â�Ã¨Â§Â�Ã¥Â¥Â¹Ã¥Â¥Â¶Ã¥Â¥Â¶Ã¦Â�Â�Ã¥Â�Â�Ã§Â�Â°Ã£Â�Â�Ã¨Â�Â�Ã¥Â�Â°Ã¥Â�Â°Ã¦Â�Â¯Ã¥Â�Â¨Ã¥Â®Â¶Ã¦Â�Â�Ã©Â�Â¾Ã§Â�Â�Ã§Â�Â�Ã£Â�Â�Ã¯Â¼Â�
</div>

shtse8 · 2017-03-26T19:23:19Z

Code to reproduce:

$html = '<div><p>Paragraph 1</p> <p>Paragraph 2</p><p>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</p></div>';
$doc = FluentDOM($html, 'html-fragment');
echo $doc->html();

Expected Result:

<div><p>Paragraph 1</p> <p>Paragraph 2</p><p>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</p></div>

Actaul Result:

<p>Paragraph 1</p> <p>Paragraph 2</p>
<p>ç½�å��ç»�äº�è��æ��å�ºã��è��å�°å�°ã��å®¶æ��ç�§ç��ï¼�æ²¡æ�³å�°ç��è§�å¥¹å¥¶å¥¶æ��å��ç�°ã��è��å�°å�°æ�¯å�¨å®¶æ��é�¾ç��ç��ã��ï¼�</p>

shtse8 · 2017-04-19T07:17:45Z

any news on this?

ThomasWeinert · 2017-04-19T11:00:16Z

I just pushed a fix for html-fragments: FluentDOM/FluentDOM@9b0daec

shtse8 · 2017-04-22T05:22:38Z

@ThomasWeinert Thanks for your updates.
But setting a html-fragment as a innerHtml of a html-fragment result is still wrong.

Code to reproduce

$doc = FluentDOM('<div></div>', 'html-fragment');
$doc->html(FluentDOM('网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！', 'html-fragment'));
echo $doc;

ExpectedResult:

<div>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</div>

Actual Result:

<div>ç½�å��ç»�äº�è��æ��å�ºã��è��å�°å�°ã��å®¶æ��ç�§ç��ï¼�æ²¡æ�³å�°ç��è§�å¥¹å¥¶å¥¶æ��å��ç�°ã��è��å�°å�°æ�¯å�¨å®¶æ��é�¾ç��ç��ã��ï¼�
</div>

ThomasWeinert · 2017-04-22T08:26:58Z

That uses a different method of the HTML loader (load() vs loadFragment()). Added test, fixed and pushed: FluentDOM/FluentDOM@28c53f1

shtse8 · 2017-04-25T17:33:20Z

Thanks. It works GREAT!
But why is the closing div in the next line?

Code to reproduce

$doc = FluentDOM('<div></div>', 'html-fragment');
$doc->html(FluentDOM('hihi', 'html-fragment'));
echo $doc;

Expected Result:

<div>hihi</div>

Actual Result:

<div>hihi
</div>

ThomasWeinert · 2017-06-28T21:28:14Z

I will move the formatting problem to a new issue

ThomasWeinert self-assigned this Mar 2, 2017

ThomasWeinert added the bug label Mar 2, 2017

ThomasWeinert added this to the 6.1 milestone Mar 2, 2017

ThomasWeinert closed this as completed Jun 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not encoding UTF-8 correctly #66

Not encoding UTF-8 correctly #66

shtse8 commented Mar 1, 2017 •

edited

Loading

ThomasWeinert commented Mar 2, 2017

shtse8 commented Mar 3, 2017 •

edited

Loading

shtse8 commented Mar 3, 2017 •

edited

Loading

ThomasWeinert commented Mar 3, 2017

ThomasWeinert commented Mar 4, 2017

shtse8 commented Mar 6, 2017 •

edited

Loading

shtse8 commented Mar 6, 2017

shtse8 commented Mar 19, 2017

ThomasWeinert commented Mar 26, 2017

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Apr 19, 2017

ThomasWeinert commented Apr 19, 2017

shtse8 commented Apr 22, 2017 •

edited

Loading

ThomasWeinert commented Apr 22, 2017

shtse8 commented Apr 25, 2017 •

edited

Loading

ThomasWeinert commented Jun 28, 2017

Not encoding UTF-8 correctly #66

Not encoding UTF-8 correctly #66

Comments

shtse8 commented Mar 1, 2017 • edited Loading

ThomasWeinert commented Mar 2, 2017

shtse8 commented Mar 3, 2017 • edited Loading

shtse8 commented Mar 3, 2017 • edited Loading

ThomasWeinert commented Mar 3, 2017

ThomasWeinert commented Mar 4, 2017

shtse8 commented Mar 6, 2017 • edited Loading

shtse8 commented Mar 6, 2017

shtse8 commented Mar 19, 2017

ThomasWeinert commented Mar 26, 2017

shtse8 commented Mar 26, 2017 • edited Loading

shtse8 commented Mar 26, 2017 • edited Loading

shtse8 commented Mar 26, 2017 • edited Loading

Code to reproduce

Expected Result:

Actual Result:

shtse8 commented Mar 26, 2017 • edited Loading

Code to reproduce:

Expected Result:

Actaul Result:

shtse8 commented Apr 19, 2017

ThomasWeinert commented Apr 19, 2017

shtse8 commented Apr 22, 2017 • edited Loading

Code to reproduce

ExpectedResult:

Actual Result:

ThomasWeinert commented Apr 22, 2017

shtse8 commented Apr 25, 2017 • edited Loading

Code to reproduce

Expected Result:

Actual Result:

ThomasWeinert commented Jun 28, 2017

shtse8 commented Mar 1, 2017 •

edited

Loading

shtse8 commented Mar 3, 2017 •

edited

Loading

shtse8 commented Mar 3, 2017 •

edited

Loading

shtse8 commented Mar 6, 2017 •

edited

Loading

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Mar 26, 2017 •

edited

Loading

shtse8 commented Apr 22, 2017 •

edited

Loading

shtse8 commented Apr 25, 2017 •

edited

Loading