Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IDN (non-ASCII domain names) support #63

Closed
swiftmailer opened this issue Sep 2, 2011 · 27 comments

Comments

Projects
None yet
7 participants
@ghost
Copy link
Collaborator

commented Sep 2, 2011

When i use special characters in e-mails which is valid these days you get an error:

Address in mailbox given [Tëst@intermeshdev.nl] does not comply with RFC 2822, 3.6.2.

Original creation date: 2009-05-22T11:56:20Z
Original reporter: Maciej Lisiewski
Original ticket: http://swiftmailer.lighthouseapp.com/projects/21527/tickets/93

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Ignore this one. Utf-8 is not valid but people seem to be trying to use this :)

Original creation date: 2009-05-22T12:23:00Z
Original reporter: Merijn

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

It should be possible to have UTF-8 characters in the e-mail. Domains with UTF-8 like ość.pl exist and swift can't mail to them!

Original creation date: 2009-12-17T15:12:42Z
Original reporter: Merijn

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

UTF-8 domains violate RFC 2822, but since they now exist we should certainly slacken to regular expression which checks for the domain portion of the email.

e: chris@w3style.co.uk
t (en): http://twitter.com/d11wtq
t (it): http://twitter.com/cosadici

Il giorno 18/dic/2009, alle ore 02.12, Lighthouse ha scritto:

Original creation date: 2009-12-17T23:14:03Z
Original reporter: Chris Corbyn

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Do you have a patch for this?

Original creation date: 2009-12-18T08:11:46Z
Original reporter: Merijn

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

At the moment there is no "true" UTF-8 in domains - IDNs [ http://en.wikipedia.org/wiki/Internationalized_domain_name ] are still ASCII only, but with UTF-8 characters encoded in Punycode [ http://en.wikipedia.org/wiki/Punycode ]. I will submit my UTF-8->Punycode converter once I find it (give me a couple of hours) and I'll leave it up to you when you want to use it (pre-validation, or post-validation - the lated needing changing the regexp to accommodate for UTF-8 in domain name). I am not entirely sure whenever I made one that does Punycode->UTF-8 as I wrote mine to take care of no IDN support in PHPMailer.

Original creation date: 2009-12-21T06:49:57Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Update: will add code before the end of the month - I have found it, it works, but also it's totally unreadable, commentless and with variable names such as $_abc2 - It has been a quick and dirty fix and once I make it readable I'll post it here.

Original creation date: 2009-12-23T04:24:17Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

As promised I include my cleaned up converter function. Function accepts prevalidated UTF-8 domain names (not emails!) - extra explode will be required. There shouldn't be much of a performance impact if this function is called for each email - for ASCII only domain names it doesn't do much.

Original creation date: 2010-01-02T18:36:05Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Correct file attached

Original creation date: 2010-01-02T18:38:32Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Thanks! I'll try it. But shouldn't this function be built into the Swift library?

Original creation date: 2010-01-04T07:50:02Z
Original reporter: Merijn

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Yes, I think it should.

Original creation date: 2010-01-04T08:36:35Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Oh, and since I haven't added any licence info in the file: feel free to include any part of my code in swift mailer

Original creation date: 2010-01-19T21:15:14Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

We'll check if we can rely on the http://www.php.net/manual/en/function.idn-to-ascii.php function too :)

I'll check how to integrate this in the Address Headers

Original creation date: 2011-03-12T22:54:04Z
Original reporter: xdecock

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

idn_to_ascii is nice, but is PHP >= 5.3.0 only and requires 2 PECL packages to be installed installed: PECL intl >= 1.0.2 and PECL idn >= 0.1.

Original creation date: 2011-03-13T10:31:11Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Hi,

I just added your code in my branch, however, for now, it encodes every domain, without the STD3-ASCII Rules, i'll check if it's possible to add this validation, but for this, i need to understand what's the problem exactly.

Can you confirm we have permission to embed it as GPL inside Swift?

Original creation date: 2011-03-22T21:33:39Z
Original reporter: xdecock

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

I think all domain names will have to pass through this:
To encode only non-ascii domain names you'd have to be able to determine if the domain name contains any non-ascii characters - check if each and every character in domain name is <= 0x7F. This is the very first step of the encoding - all the characters that are actually <= 0x7F are just copied - additional check before string if passed to be encoded will just duplicate this step. The only real optimization I can see here is getting rid of ord() I have in my code.
I doubt there is any additional check necessary - even with ord it will encode google.com 100 000 times in 1.1s on old xeon I use for testing. Encoding 6 letter IDN domain with 3 non-ascii characters is only 0.3s slower on 100 000 encodes.

As for adding it to Swift feel free to include it - I am fine with GPL/LGPL/MIT/BSD or any compatible licence(s).

Original creation date: 2011-03-23T00:50:26Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

OK, thanks for the permission.

The validation problem is that domains such as héllo..wòrld are encoded altough it's not a valid domain.

  • some code points must be handled as dot. but it's a pretty good base :)

Original creation date: 2011-03-23T00:55:56Z
Original reporter: xdecock

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

I think we shouldn't look to deeply into domain being valid or not - rules vary a lot between TLDs and the resulting regexp would be enormous.
Better idea would be to just encode all domains without any checks BEFORE resulting email@encoded-idn-domain.tld is passed to email address validation. Current regexp will accept encoded IDN domains (fails on non-encoded) and reject all obviously invalid addresses (such as your example "héllo..wòrld") - the only false positives would be domains not following TLD-speciffic rules.

Original creation date: 2011-03-23T02:31:18Z
Original reporter: Maciej Lisiewski

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Please adjust the milestone setting of this bug. It's assigned to 4.1, but meanwhile 4.1.1 is out already :-)
BTW: +1 for the bug itself - I'd need that functionality too.

Original creation date: 2011-07-25T16:43:49Z
Original reporter: Thomas Landauer

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Until Swift Mailer supports IDN natively, you can use
http://www.phpclasses.org/package/1509-PHP-Convert-from-and-to-IDNA-Punycode-domain-names.html
to convert an IDN domain name to punycode.
Not very elegant tough - but still better than an error message ;-)

Original creation date: 2011-07-27T21:48:44Z
Original reporter: Thomas Landauer

@ghost

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 2, 2011

Or you can use idn->punycode converter code I have attached few posts earlier - instead of class a single 115 line function.

Original creation date: 2011-08-09T19:09:25Z
Original reporter: Maciej Lisiewski

@MickL

This comment has been minimized.

Copy link

commented Dec 10, 2013

any news on this?

@mvar

This comment has been minimized.

Copy link

commented Feb 12, 2015

status update, please

@dakira

This comment has been minimized.

Copy link

commented Jan 18, 2017

@fabpot: let's finally fix this and #541. I propose we don't add the intl dependency and use true/php-punycode instead for IDN/punycode conversion.

The way I see it, the quickest solution would be to generally convert all email addresses containing multibyte chars to punycode (before validation).

Comments?

Edit: Another option could be to replace validation with voku/email-check and only convert for sending.

@ossinkine

This comment has been minimized.

Copy link

commented Jan 18, 2018

I've just published plugin for Swiftmailer which converts domain name to punycode before sending https://github.com/ossinkine/swiftmailer-punycode-plugin. Maybe someone will find it useful while we have no official support.

fabpot added a commit that referenced this issue Jan 24, 2018

feature #1044 Basic IDN support (c960657)
This PR was squashed before being merged into the 6.0-dev branch (closes #1044).

Discussion
----------

Basic IDN support

| Q             | A
| ------------- | ---
| Bug fix?      | no
| New feature?  | yes
| Doc update?   | no
| BC breaks?    | no
| Deprecations? | no
| Fixed tickets | #63, #541
| License       | MIT

IDN domains have been around for more than a decade and are now widely used. Swiftmailer should support internationalised email addresses. Even if you don't use an IDN domain yourself, you may need to send email to people who do.

Calling `setTo()` with a non-ASCII email address does not throw an error (it does throw, if the email address is otherwise malformed). The [egulias/EmailValidator](https://github.com/egulias/EmailValidator) package accepts addresses containing non-ASCII characters both in the local-part (the part of the address left of @) and domain (after @). However, when you try to send the email, the SMTP server rejects the address for being syntactically invalid.

Handling non-ASCII characters in local-part and domain are two different issues. Non-ASCII characters in the domain is supported by all email servers, as long as the domain is IDN-encoded by the sender. Non-ASCII characters in local-part requires support for the `SMTPUTF8` SMTP extension (see [RFC 6531](https://tools.ietf.org/html/rfc6531)).

This PR adds basic support for email addresses using IDN domains. It does not add support for `SMTPUTF8` but simply IDN-encodes the domain, so mail can use the existing infrastructure. The implementation is backwards-compatible fashion.

Future work should add support for `SMTPUTF8` in order to support for email addresses with non-ASCII characters in local-part also. This would probably require BC-breaking changes, so it needs to go into a new major release of Swiftmailer.

Commits
-------

6a87efd Basic IDN support
@c960657

This comment has been minimized.

Copy link
Contributor

commented Feb 23, 2018

Fixed in #1044. Will be included in the upcoming release 6.1.0.

This ticket should be closed.

@MickL

This comment has been minimized.

Copy link

commented Feb 23, 2018

This took only 6 years

@fabpot

This comment has been minimized.

Copy link
Member

commented Feb 24, 2018

@MickL Indeed, you could have contributed that feature 6 years ago, but you did not.

Big thanks to @c960657 who took the time to investigate the issue, worked on the implementation, tested it, submitted a pull request, worked on making it better based on feedback. Thank you @c960657 in the name of all developers out there who will be able to benefit from this important feature for free. Of course, they don't care how and by whom it was implemented. Some won't even realized that you worked on this for free, without being paid, without any incentives. You deserve one more thank you from me, the maintainer who was too lazy to dedicate a few free days to work on this to please developers who are never satisfied with what they get for free.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.