Strange characters in HTML mails on iOS

[evrpress]

This isnt a real issue rather a question.

I'm using phpMailer in a WordPress plugin and send HTML mails. This works very well. But unfortunataly some users have these strange characters on german Umlauts and the   for instance.

This normally happens when using a different charset rather than UTF-8 but this is not the case. I don't allow users to change that and as long they don't change the code (what they don't do) there are no hooks or filters to change it too.

The strangest thing is when they send me the exact same message to me and I have no problems on my iPhone

Has anyone experienced similar? I hardly get this support request but every time I have to say that I don't know an answer.

[Synchro]

This is bound to be an encoding issue somewhere, whether in the page where users enter their content, how it's submitted to PHPMailer, or the settings you're using in PHPMailer. PHPMailer defaults to the ISO-8859-1 charset so unless you're changing that you're probably looking at a latin1 to UTF-8 mismatch.
I'm quite happily using 4-byte UTF-8 unicode characters in PHPMailer without any issues, though I know that wordpress can't cope with that by default.
It's somewhat telling that the example character you posted didn't appear...

[evrpress]

Thanks for the quick response!

Users can edit the content via the built in WordPress editor (TinyMCE). This causes often some strange markup especially when pasting from Word or similar.

But I'm really curios why the exact mail looks ok on my iphone while it breaks on the users one? I can't change encoding on my device

[Synchro]

On 14 May 2013, at 16:11, revaxarts notifications@github.com wrote:

Users can edit the content via the built in WordPress editor (TinyMCE). This causes often some strange markup especially when pasting from Word or similar.

Pasting from Word is generally a recipe for disaster. It's very likely to include strange characters from the wrong encoding.
But I'm really curios why the exact mail looks ok on my iphone while it breaks on the users one? I can't change encoding on my device

Well iPhone has better character set support than many other email clients. I know it supports things like Emojis which will work perfectly on there and break in many other places, but I doubt your users are entering 4-byte chars.

I'd say the most likely source of the problem is TinyMCE. Try doing some experiments with specific characters in different encodings and see how they come out.

Marcus

[evrpress]

Ok, I'll do!

Thanks Marcus! You did a great job!

[ooxi]

Hi,

Which character are displayed? What User Agent Do your customers use? Do you specify the mime Type charset?

-- ooxi

Violetland — An open source cross-platform game similar to Crimsonland — http://violetland.github.com

revaxarts notifications@github.com schrieb:

This isnt a real issue rather a question.

I'm using phpMailer in a WordPress plugin and send HTML mails. This works very well. But unfortunataly some users have these strange characters on german Umlauts and the   for instance.

This normally happens when using a different charset rather than UTF-8 but this is not the case. I don't allow users to change that and as long they don't change the code (what they don't do) there are no hooks or filters to change it too.

The strangest thing is when they send me the exact same message to me and I have no problems on my iPhone

Has anyone experienced similar? I hardly get this support request but every time I have to say that I don't know an answer.

---

Reply to this email directly or view it on GitHub:
#59

[evrpress]

Mostly these "A" characters - (http://ticksy_attachments.s3.amazonaws.com/5765101190.png)

Which user agent? It's an iPhone 4 with iOS 6.1.x

As mentioned the same email looks fine on my (same) device

[Synchro]

Those characters indicate that you're displaying UTF-8 encoded text in an 8-bit Windows-1252 charset. See this page for a map: http://www.i18nqa.com/debug/utf8-debug.html

For example in the title you have the sequence ✔, which is hex E29C94 in Windows-1252, which is apparently meant to be the character "U+2714 HEAVY CHECK MARK", which should look like: ✔. Is that what you see on your phone?

Interesting carrier you have!

[evrpress]

It's the phone and the screenshot from a user (obviously an iPhone 5 but have others with same issue). The "A"s should be   and are invisible on my phone (and other clients).

I'm not that much familiar with different charsets and thought utf-8 is the way to go without problems

[ooxi]

Hi,

UTF-8 is the way to go. Are you sure you Set the correct Content Type (Text/plain; charset=utf-8)?

-- ooxi

Violetland — An open source cross-platform game similar to Crimsonland — http://violetland.github.com

revaxarts notifications@github.com schrieb:

It's the phone and the screenshot from a user (obviously an iPhone 5 but have others with same issue). The "A"s should be   and are invisible on my phone (and other clients).

I'm not that much familiar with different charsets and thought utf-8 is the way to go without problems

---

Reply to this email directly or view it on GitHub:
#59 (comment)

[evrpress]

Every html mail has this header:
```

I set Charset to utf-8:
$this->CharSet = 'UTF-8';

I use isHTML to set $ContentType to text/html

[Synchro]

That may be true, but if your'e providing it with content that uses the Windows-1252 charset, you'll get the problem you're seeing. Try using mb_convert_encoding to convert your content to UTF-8 first - that's not something that is up to PHPMailer.

[evrpress]

Ok thanks Marcus! I'm using this snippet now:

if(function_exists('mb_convert_encoding')){
    $encoding = mb_detect_encoding($content, 'auto');
    $content = mb_convert_encoding($content, $encoding, 'UTF-8');
}

let's see if this works

[evrpress]

Seems the above method doesn't work. I have again a customer with this issue.

Here is the customers screenshot
Here is my device

He sad if he switches to turkish the email is fine but not with English. My phone is running with English as well.

I'm sorry to post this as an issue cause it'S probably not related to phpMailer. but I hope I'll find an answer here.

[Synchro]

Somewhere you're still using the Windows charset. You need to track that down.

For the most part, UTF-8 doesn't care what language your'e using, it's all just characters; having it fail when switching to a non-matching language suggests there's an 8-bit charset in use, which will fail as you're seeing.

When comparing devices, it's best to use BCC to send the messages to multiple recipients rather than sending them separate messages, then you know they are receiving copies of the exact same message, which are thus comparable.

[ooxi]

Could you please post the message itself here? It sould look something like

Delivered-To: XXX
Received: by A.B.C.D with SMTP id XXX;
        Wed, 22 May 2013 17:15:20 -0700 (PDT)
X-Received: by A.B.C.D with SMTP id XXX;
        Wed, 22 May 2013 17:15:19 -0700 (PDT)
Return-Path: <XXX>
Received: from XXX (XXX. [A.B.C.D])
        by mx.google.com with ESMTP id XXX
        for <XXX>;
        Wed, 22 May 2013 17:15:19 -0700 (PDT)
Received-SPF: neutral (google.com: A.B.C.D is neither permitted nor denied by best guess record for domain of XXX) client-ip=A.B.C.D;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: A.B.C.D is neither permitted nor denied by best guess record for domain of XXX) smtp.mail=XXX
Received: by XXX (Postfix, from userid 33)
    id XXX; Thu, 23 May 2013 02:15:19 +0200 (CEST)
To: XXX
Subject: XXX
MIME-Version: 1.0
From: XXX
Reply-To: XXX
Message-Id: <XXX@XXX>
Date: Thu, 23 May 2013 02:15:19 +0200 (CEST)


Your message here

[evrpress]

Delivered-To: XXX
Received: by XXX with SMTP id XXX;
        Wed, 22 May 2013 07:27:17 -0700 (PDT)
X-Received: by XXX with SMTP id XXX;
        Wed, 22 May 2013 07:27:17 -0700 (PDT)
Return-Path: <XXX>
Received: from XXX (XXX [XXX])
        by mx.google.com with ESMTP id XXX
        for <XXX>;
        Wed, 22 May 2013 07:27:17 -0700 (PDT)
Received-SPF: neutral (google.com: XXX is neither permitted nor denied by best guess record for domain of XXX) client-ip=XXX;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: XXX is neither permitted nor denied by best guess record for domain of XXX) smtp.mail=XXX
Received: from XXX (unknown [XXX])
    by XXX (Postfix) with ESMTP id XXX
    for <XXX>; Wed, 22 May 2013 16:52:19 +0300 (EEST)
Received: from XXX (XXX [XXX])
    by XXX (Postfix) with ESMTP id XXX
    for <XXX>; Wed, 22 May 2013 17:39:04 +0300 (EEST)
Date: Wed, 22 May 2013 14:27:10 +0000
To: XXX
From: XXX <XXX>
Reply-To: XXX
Subject: Please check the mail in iPhone Mail App ticket ID 55503
Message-ID: <ba5330e8fde51d0915e1f6c0e15a94fc@XXX>
X-Priority: 3
X-Mailer: MyMail 1.5.3.2 (5.2.6) by revaxarts.com (http://rxa.li/mymail)
MIME-Version: 1.0
Content-Type: multipart/alternative;
    boundary="b1_ba5330e8fde51d0915e1f6c0e15a94fc"

--b1_ba5330e8fde51d0915e1f6c0e15a94fc
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit


//text part


--b1_ba5330e8fde51d0915e1f6c0e15a94fc
Content-Type: multipart/related;
    boundary="b2_ba5330e8fde51d0915e1f6c0e15a94fc"

--b2_ba5330e8fde51d0915e1f6c0e15a94fc
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

//HTML version


--b2_ba5330e8fde51d0915e1f6c0e15a94fc
Content-Type: image/png; name="XXX.png"
Content-Transfer-Encoding: base64
Content-ID: <11c2943ac6b04ccf2d2deab6d3e49599@phpmailer.0>
Content-Disposition: inline; filename=XXXlogotransparentwhite.png

//embedded images stuff

--b2_ba5330e8fde51d0915e1f6c0e15a94fc--


--b1_ba5330e8fde51d0915e1f6c0e15a94fc--

[ooxi]

The real content of //text part and //HTML version is quite important ;)

[evrpress]

seem legit :) I've just copied and paste parts of the content:

Text

Tobago�da hayalindeki evini in�a etmi�tir ve burada Karayip dansçıların kalabilmesini sa�lamı�tır, böylece Afro-Karayip sanatının en önemli temsilcilerinden biri haline gelmi�tir. Heyecan verici bir �ekilde kendi deneyimlerini payla�arak hayallerini gerçe�e çevirebilen ki�ilerdendir. 

HTML

<p style="margin:1em 0"><a href="XXX/" target="_blank" style="color:#F22734;text-decoration:none"><strong><span style="font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; line-height: 16px;">Salı/ Tuesday 21.05.2013 Saat/Time: 20.00 - �cretsiz - Free</span></strong></a></p>
<p style="margin:1em 0">Ba�dat Caddesi No: 350-A Erenköy 34728 Istanbul<br><br style="font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; line-height: 16px;"><span style="font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; line-height: 16px;">Trailer: </span><a style="color:#F22734;cursor: pointer;font-family: 'lucida grande', tahoma, verdana, arial, sans-serif;line-height: 16px;text-decoration:none" href="XXX" target="_blank" rel="nofollow nofollow">XXX</a></p>
</div>

[Synchro]

Lets play "spot the 8-bit character set"...

[evrpress]

I saw them, I just tested it with ÄÖÜ and got Ã�Ã�Ã�

Why this doesn't help?

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

[ooxi]

This might not be a problem now but I suggest using Content-Transfer-Encoding Base64 or quoted printable even for text/plain and text/html parts. Not all mail servers are 8bit stable now. Could you post the hex values of ErenköyÂ?

[evrpress]

How should I do that?

I'll offer an option to set CharSet and Encoding in the next update. I think this should solve this issue.
I always thought that UTF-8 can handle all characters but it seems it doesn't.

I think we can close this issue for now. I'll test it with these particular users.

Thanks so much for your effort! I really appreciate that!

[ooxi]

I'll offer an option to set CharSet and Encoding in the next update. I think this should solve this issue.
I always thought that UTF-8 can handle all characters but it seems it doesn't.

UTF-8 can handle (almost) all characters but your input has to be valid UTF-8. You cannot just send an arbitrary blob labelled UTF-8 and hope the conversion will be done by magic gnomes ;)