Encoding Issues Driving Me Insane!

[glenelkins1984]

Hi

I like this library but i am going insane with the encoding. Pulling 5000 emails from GMAIL works fine, but a lot of the html has encoding problems. I can't even seem to convert it all to proper UTF-8 with mbstring or utf8_encode, i just still have weird characters all over the place. I also can't just insert the html directly as the utf8 database wont' accept it.

There must be a way to have the emails displaying properly! Can someone help me here, ideally i should be able to just get the encoding from the message object, something like:

/** @var PhpImap\IncomingMail */
$mesaage->getEncoding()

But there doesn't seem to be such a thing on the IncomingMail object, if i could know what the encoding was then i could use mbstring to convert it properly.

[bapcltd-marv]

I think the issue with that approach is different chunks of the message can be encoded differently- you'd need to be able to inspect
DataPartInfo::$charset on all IncomingMail::$dataInfo entries.

[glenelkins1984]

Thanks for the reply.

So how do i do that then? What you're describing is some static properties, but i'm looping over IncomingMail objects and there is no $mailObj->charset there is also no property $mailObj->dataInfo ?

This is after pulling a mail in like so:

$remoteEmail = $imap->getMail($mailId,false);

So in my example $remoteEmail->charset and $remoteEmail->dataInfo both do not exist.

[bapcltd-marv]

Thanks for the reply.

So how do i do that then? What you're describing is some static properties, but i'm looping over IncomingMail objects and there is no $mailObj->charset there is also no property $mailObj->dataInfo ?

This is after pulling a mail in like so:

$remoteEmail = $imap->getMail($mailId,false);

So in my example $remoteEmail->charset and $remoteEmail->dataInfo both do not exist.

I'm referring to class properties, not static properties- specifically:

php-imap/src/PhpImap/IncomingMail.php

Line 35 in 7ca3d19

protected $dataInfo = [[], []];

php-imap/src/PhpImap/DataPartInfo.php

Line 38 in 7ca3d19

public $charset;

[glenelkins1984]

Thanks, so $dataInfo is protected so how am i supposed to access that information? I don't see a public method that returns it.

[bapcltd-marv]

There must be a way to have the emails displaying properly! Can someone help me here, ideally i should be able to just get the encoding from the message object, something like:

/** @var PhpImap\IncomingMail */
$mesaage->getEncoding()

You proposal would need to be modified (at least in part) to either make these methods properties directly accessible, or to add a method into IncomingMail to collate the various DataPartInfo::$charset properties (although I'm unsure if they'd be impacted by DataPartInfo::$encoding)

A property would need to be routed through magic getters, and a method's return type would need to be defined for single body + single charset, multiple body + single charset, and multiple body + multiple charset.

The proposed technique would also likely need to account for non-inline text-based attachments.

[glenelkins1984]

ok, i was hoping the library would literally just have the html body encoding set on it without all this messing around.

[hrst]

I think I found the issue: decodeMimeStr() is called via convertEncodingAfterFetch(). However, at this point in time, the string no longer has the MIME header info. Hence, using imap_mime_header_decode() in decodeMimeStr() will not work and it then does not convert the encoding.

php-imap/src/PhpImap/DataPartInfo.php

Line 111 in 7ca3d19

$this->data = $this->mail->decodeMimeStr(

php-imap/src/PhpImap/Mailbox.php

Line 1347 in 7ca3d19

$elements = \imap_mime_header_decode($string);

I fixed it like this:

Mailbox.php

    public function decodeMimeStr(string $string): string
    {
        $newString = '';
        /** @var list<object{charset?:string, text?:string}>|false */
        $elements = \imap_mime_header_decode($string);

        if (false === $elements) {
            return $string;
        }

        foreach ($elements as $element) {
            $newString .= $this->convertToUtf8($element->text, $element->charset);
        }

        return $newString;
    }

    public function convertToUtf8(string $string, string $fromCharset): string
    {
		$fromCharset = mb_strtolower($fromCharset);
		$newString = '';

		if ('default' === $fromCharset) {
			$fromCharset = $this->decodeMimeStrDefaultCharset;
		}

		switch ($fromCharset) {
			case 'default': // Charset default is already ASCII (not encoded)
			case 'utf-8': // Charset UTF-8 is OK
				$newString .= $string;
				break;
			default:
				// If charset exists in mb_list_encodings(), convert using mb_convert function
				if (\in_array($fromCharset, $this->lowercase_mb_list_encodings(), true)) {
					$newString .= \mb_convert_encoding($string, 'UTF-8', $fromCharset);
				} else {
					// Fallback: Try to convert with iconv()
					$iconv_converted_string = @\iconv($fromCharset, 'UTF-8', $string);
					if (!$iconv_converted_string) {
						// If iconv() could also not convert, return string as it is
						// (unknown charset)
						$newString .= $string;
					} else {
						$newString .= $iconv_converted_string;
					}
				}
				break;
		}

        return $newString;
    }

DataPartInfo.php

    protected function convertEncodingAfterFetch(): string
    {
        if (isset($this->charset) and !empty(\trim($this->charset))) {
            /*$this->data = $this->mail->decodeMimeStr(
                (string) $this->data // Data to convert
            );*/
            $this->data = $this->mail->convertToUtf8(
                (string) $this->data, // Data to convert
				$this->charset
            );
        }

        return (null === $this->data) ? '' : $this->data;
    }

[dwalczyk]

@hrst merge request, your fix is working for me

[jmbianca]

@hrst Hi, your fix works well for me too. Is there any plan to merge it soon?

[Sebbo94BY]

I've created a merge request with those changes and merged it into the develop branch. Feel free to test the changes using the develop branch.

[Tanktiger]

@Sebi94nbg When will there be a new release with this fix in it?

[hrst]

I've just updated and the new version actually re-introduces the problem:

php-imap/src/PhpImap/DataPartInfo.php

Lines 111 to 114 in f401d85

	$this->data = $this->mail->decodeMimeStr(
	(string) $this->data, // Data to convert
	\trim($this->charset)
	);

decodeMimeStr() is called with a second parameter, but the method does not even have it?

Also here:

php-imap/src/PhpImap/Mailbox.php

Line 1262 in fb8be23

$fileName = $this->decodeMimeStr($fileName, $this->serverEncoding);

I'm very confused on what the intention here was. Am I missing something and a second decodeMimeStr() method exists somewhere that does have the second parameter? I could not find it.

See the method here:

php-imap/src/PhpImap/Mailbox.php

Line 1385 in f401d85

public function decodeMimeStr(string $string): string

I suspect you'd want to use convertToUtf8() in both cases. I'll update my code accordingly.

[Sebbo94BY]

The develop branch has been merged into the master and released as version 4.3.0 six days ago. Please check again, if this issue still exists.

[hrst]

The current version has the same issues again, please check this comment for a fix:
#657 (comment)

[Sebbo94BY]

I've just released a new version 4.5.3 with some more improvements regarding these encoding issues and some unit tests.

Relates to #657.

Feel free to reopen this issue, if you should be still able to reproduce your issue(s) or just open a new ticket.