Add IDN (non-ASCII domain names) support

[swiftmailer]

When i use special characters in e-mails which is valid these days you get an error:

Address in mailbox given [Tëst@intermeshdev.nl] does not comply with RFC 2822, 3.6.2.

Original creation date: 2009-05-22T11:56:20Z
Original reporter: Maciej Lisiewski
Original ticket: http://swiftmailer.lighthouseapp.com/projects/21527/tickets/93

[swiftmailer]

Ignore this one. Utf-8 is not valid but people seem to be trying to use this :)

Original creation date: 2009-05-22T12:23:00Z
Original reporter: Merijn

[swiftmailer]

It should be possible to have UTF-8 characters in the e-mail. Domains with UTF-8 like ość.pl exist and swift can't mail to them!

Original creation date: 2009-12-17T15:12:42Z
Original reporter: Merijn

[swiftmailer]

UTF-8 domains violate RFC 2822, but since they now exist we should certainly slacken to regular expression which checks for the domain portion of the email.

e: chris@w3style.co.uk
t (en): http://twitter.com/d11wtq
t (it): http://twitter.com/cosadici

Il giorno 18/dic/2009, alle ore 02.12, Lighthouse ha scritto:

Original creation date: 2009-12-17T23:14:03Z
Original reporter: Chris Corbyn

[swiftmailer]

Do you have a patch for this?

Original creation date: 2009-12-18T08:11:46Z
Original reporter: Merijn

[swiftmailer]

At the moment there is no "true" UTF-8 in domains - IDNs [ http://en.wikipedia.org/wiki/Internationalized_domain_name ] are still ASCII only, but with UTF-8 characters encoded in Punycode [ http://en.wikipedia.org/wiki/Punycode ]. I will submit my UTF-8->Punycode converter once I find it (give me a couple of hours) and I'll leave it up to you when you want to use it (pre-validation, or post-validation - the lated needing changing the regexp to accommodate for UTF-8 in domain name). I am not entirely sure whenever I made one that does Punycode->UTF-8 as I wrote mine to take care of no IDN support in PHPMailer.

Original creation date: 2009-12-21T06:49:57Z
Original reporter: Maciej Lisiewski

[swiftmailer]

Update: will add code before the end of the month - I have found it, it works, but also it's totally unreadable, commentless and with variable names such as $_abc2 - It has been a quick and dirty fix and once I make it readable I'll post it here.

Original creation date: 2009-12-23T04:24:17Z
Original reporter: Maciej Lisiewski

[swiftmailer]

As promised I include my cleaned up converter function. Function accepts prevalidated UTF-8 domain names (not emails!) - extra explode will be required. There shouldn't be much of a performance impact if this function is called for each email - for ASCII only domain names it doesn't do much.

Original creation date: 2010-01-02T18:36:05Z
Original reporter: Maciej Lisiewski

[swiftmailer]

Correct file attached

Original creation date: 2010-01-02T18:38:32Z
Original reporter: Maciej Lisiewski

[swiftmailer]

Thanks! I'll try it. But shouldn't this function be built into the Swift library?

Original creation date: 2010-01-04T07:50:02Z
Original reporter: Merijn

[swiftmailer]

Yes, I think it should.

Original creation date: 2010-01-04T08:36:35Z
Original reporter: Maciej Lisiewski

[swiftmailer]

Oh, and since I haven't added any licence info in the file: feel free to include any part of my code in swift mailer

Original creation date: 2010-01-19T21:15:14Z
Original reporter: Maciej Lisiewski

[swiftmailer]

We'll check if we can rely on the http://www.php.net/manual/en/function.idn-to-ascii.php function too :)

I'll check how to integrate this in the Address Headers

Original creation date: 2011-03-12T22:54:04Z
Original reporter: xdecock

[swiftmailer]

idn_to_ascii is nice, but is PHP >= 5.3.0 only and requires 2 PECL packages to be installed installed: PECL intl >= 1.0.2 and PECL idn >= 0.1.

Original creation date: 2011-03-13T10:31:11Z
Original reporter: Maciej Lisiewski

[swiftmailer]

Hi,

I just added your code in my branch, however, for now, it encodes every domain, without the STD3-ASCII Rules, i'll check if it's possible to add this validation, but for this, i need to understand what's the problem exactly.

Can you confirm we have permission to embed it as GPL inside Swift?

Original creation date: 2011-03-22T21:33:39Z
Original reporter: xdecock

[swiftmailer]

I think all domain names will have to pass through this:
To encode only non-ascii domain names you'd have to be able to determine if the domain name contains any non-ascii characters - check if each and every character in domain name is <= 0x7F. This is the very first step of the encoding - all the characters that are actually <= 0x7F are just copied - additional check before string if passed to be encoded will just duplicate this step. The only real optimization I can see here is getting rid of ord() I have in my code.
I doubt there is any additional check necessary - even with ord it will encode google.com 100 000 times in 1.1s on old xeon I use for testing. Encoding 6 letter IDN domain with 3 non-ascii characters is only 0.3s slower on 100 000 encodes.

As for adding it to Swift feel free to include it - I am fine with GPL/LGPL/MIT/BSD or any compatible licence(s).

Original creation date: 2011-03-23T00:50:26Z
Original reporter: Maciej Lisiewski

[swiftmailer]

OK, thanks for the permission.

The validation problem is that domains such as héllo..wòrld are encoded altough it's not a valid domain.

• some code points must be handled as dot. but it's a pretty good base :)

Original creation date: 2011-03-23T00:55:56Z
Original reporter: xdecock

[swiftmailer]

I think we shouldn't look to deeply into domain being valid or not - rules vary a lot between TLDs and the resulting regexp would be enormous.
Better idea would be to just encode all domains without any checks BEFORE resulting email@encoded-idn-domain.tld is passed to email address validation. Current regexp will accept encoded IDN domains (fails on non-encoded) and reject all obviously invalid addresses (such as your example "héllo..wòrld") - the only false positives would be domains not following TLD-speciffic rules.

Original creation date: 2011-03-23T02:31:18Z
Original reporter: Maciej Lisiewski

[swiftmailer]

Please adjust the milestone setting of this bug. It's assigned to 4.1, but meanwhile 4.1.1 is out already :-)
BTW: +1 for the bug itself - I'd need that functionality too.

Original creation date: 2011-07-25T16:43:49Z
Original reporter: Thomas Landauer

[swiftmailer]

Until Swift Mailer supports IDN natively, you can use
http://www.phpclasses.org/package/1509-PHP-Convert-from-and-to-IDNA-Punycode-domain-names.html
to convert an IDN domain name to punycode.
Not very elegant tough - but still better than an error message ;-)

Original creation date: 2011-07-27T21:48:44Z
Original reporter: Thomas Landauer

[swiftmailer]

Or you can use idn->punycode converter code I have attached few posts earlier - instead of class a single 115 line function.

Original creation date: 2011-08-09T19:09:25Z
Original reporter: Maciej Lisiewski

[MickL]

any news on this?

[mvar]

status update, please

[dakira]

@fabpot: let's finally fix this and #541. I propose we don't add the intl dependency and use true/php-punycode instead for IDN/punycode conversion.

The way I see it, the quickest solution would be to generally convert all email addresses containing multibyte chars to punycode (before validation).

Comments?

Edit: Another option could be to replace validation with voku/email-check and only convert for sending.

[ossinkine]

I've just published plugin for Swiftmailer which converts domain name to punycode before sending https://github.com/ossinkine/swiftmailer-punycode-plugin. Maybe someone will find it useful while we have no official support.

[c960657]

Fixed in #1044. Will be included in the upcoming release 6.1.0.

This ticket should be closed.

[MickL]

This took only 6 years

[fabpot]

@MickL Indeed, you could have contributed that feature 6 years ago, but you did not.

Big thanks to @c960657 who took the time to investigate the issue, worked on the implementation, tested it, submitted a pull request, worked on making it better based on feedback. Thank you @c960657 in the name of all developers out there who will be able to benefit from this important feature for free. Of course, they don't care how and by whom it was implemented. Some won't even realized that you worked on this for free, without being paid, without any incentives. You deserve one more thank you from me, the maintainer who was too lazy to dedicate a few free days to work on this to please developers who are never satisfied with what they get for free.