[Proposal] More robust IMAP autodiscovery

[chenseanxy]

Describe the solution you'd like

Currently Wino gets IMAP settings from the emailsettings.firetrust.com API, but the contents of these datasets are not configurable by email admins. There are some ways site admins can configure autodiscovery for their sites: (in brackets: standards established by)

• SRV Record: _autodiscover (Microsoft) Record points to a domain that contains autodiscover.xml
• SRV Record: _imap/_imaps/_submission (RFC6186) Record directly stores IMAP / SMTP host and port info
• /autodiscover/autodiscover.xml (Microsoft)
• autoconfig.domain.tld/mail/config-v1.1.xml (Thunderbird)
• /email.mobileconfig (Apple)

At this point it's difficult to know of which clients currently supports which standards, and there are probably more private databases that some clients use to do autodiscovery (like Microsoft's AutoDetect service).

I'm not sure which ones to implement, but I think it'd be crucial that at least one of these admin-configurable autodiscover sources get implemented (and admins can make sure they configure that one specifically for Wino).

Is your proposal implemented in Windows Mail?

It's difficult to know definitively, what Windows Mail uses.

However from my experience, Wino sometimes fails autodetect when Windows Mail doesn't. At least for the case of our org that has SRV _autodiscover configured, Windows Mail was able to autodiscover settings based on that (or it's somehow Microsoft's AutoDetect service), but not Wino.

Additional context

Protocols:

• Microsoft Plain Old XML
• Thunderbird Autoconfiguration
• That RFC probably no one is using

Helpful writeups:

• Zoho's setup guide: MS and Thunderbird
• Writeup on Microsoft AutoDetect which I think we can also use, but might be problematic if we gave it a high priority

[bkaankose]

Nicely done investigation.

AutoDiscoveryService definitely lacks lots of capabilities right now. I wanted to include Thunderbird Autoconfig at some point but never had the time for it.

This would speed up things for a lot of new users since most of them don't want to search for the IMAP settings they've been offered from their provider.

[chenseanxy]

I think what we can do, currently, is the plain XML versions of Thunderbird and MSExchange, maybe later on we can add SRV support of both of these.

Implementing both of these should be simple enough:

• Try sending request to multiple possible domains that contain the autodiscovery response (eg, MS: domain.tld or autodiscover.domain.tld, Thunderbird: autoconfig.domain.tld and domain.tld/.well-known)
• On 200, deserialize and return the discovered settings

Now we're introducing multiple sources, there's the consideration about priority:

• MS and Thunderbird admin-defined sources should get higher priority than the central DB / API sources, so admins can override bad info in the centralized sources
• Between MS and thunderbird it shouldn't matter

So to implement this, here's what I'm thinking:

• Every source gets its own model, as well as provider that handles request, timeout, and deserialize logic (try multiple domains), and returns AutoDiscoverySettings
• For AutoDiscoveryService, it awaits the providers one by one according to priority, returns if settings are found
  • maybe parallize? but the providers should be pretty fast, only kinda-slow thing is gonna be cold DNS lookups

If it's OK you can assign this and I'll start working on it this week. (Sorry for writing too much, I'm usually overcautious in new codebases)

[bkaankose]

I think you sum up pretty well. As long as we return AutoDiscoverySettings you don't need a lot of changes in the current code.

If you have something better in mind, you may completely discard the existing code and rewrite the module for auto discovery as well. I don't mind it since current implementation lacks a lot of things.

Assigned to @chenseanxy