Add Autoconfig, Autodiscover. Import exchangelib for Microsoft compatibility.

[chris001]

Following up from #507 on the topic of auto-detecting which type of IMAP server we are talking to.

Let user enter only the email address. Most of the time, Offlineimap can use the standard autoconfig/autodiscover method to obtain the IMAP email server details.

• There's in fact 2 very similar automated email client setup standards.

1. autoconfig (Thunderbird Linux open source servers). Docs here: https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration
2. autodiscover (Microsoft outlook.com hotmail.com office365.com ) docs: https://technet.microsoft.com/en-us/library/bb124251(v=exchg.160).aspx

• These 2 web service standards automate the configuration of the mail client (Offlineimap),
• All you need for input data is 1. the email address and 2. the domain (extracted from the email address).
• This standard is supported by popular linux hosting panels (virtualmin, cpanel, etc) and Microsoft Oulook.com Hotmail and Office365.
  This project is php example implementation of automating the email account setup, with only email address as input data.
  https://github.com/Thorarin/MailClientAutoConfig
• Offlineimap should be updated to use this autoconfig/autodiscover email account parameter detection method by default, because it works probably 95% of the time. It depends only on the mail server having this feature enabled. 99% of microsoft servers have it enabled. 90% of linux servers have it enabled. For the rare times isn't enabled, the user can add their parameters to offlineimap config file manually.

1. Visit http://autoconfig.example.com/mail/config-v1.1.xml?emailaddress=info@example.com (where info@example.com is a valid email address delivered to a single local mailbox).
   You should see an XML file in which you recognize the settings you've provided.
   If you get an empty response, check the error log of your web server.
2. Visit https://autoconfig.example.com/mail/config-v1.1.xml?emailaddress=info@example.com to test your SSL setup.
   You should see the same file.
   If this does not work, check your SSL setup.
   If everything appears to work, test by adding the mail account to Thunderbird and Outlook itself.

[nicolas33]

I'm not sure about this. This would mean maintaining a website and we don't have the money for this.

Also, our users usually are advanced enough to configure the client.

About auto-filtering this should be supported in the client first.

[chris001]

No website is necessary?
User wants to add email address to OfflineIMAP: user@emaildomain.tld.
The majority of email providers have a subdomain for this, either autoconfig.emaildomain.tld or autodiscover.emaildomain.tld

1. OfflineIMAP would first try to get http://autoconfig.emaildomain.tld/mail/config-v1.1.xml?emailaddress=user@emaildomain.tld. If it works, then use the data in the XML document to auto config the IMAP server hostname, port, TLS/SSL, etc.
2. If the previous attempt failed, OfflineIMAP could use ecederstrand microsoft exchange web services (EWS) client library, to try to autodiscover microsoft mail server IMAP parameters:
   https://stackoverflow.com/questions/288546/connect-to-exchange-mailbox-with-python
   http://msdn.microsoft.com/en-us/library/bb204119.aspx
   https://pypi.python.org/pypi/exchangelib/
   https://github.com/ecederstrand/exchangelib
   Supports autodiscover, calendars, inbox, tasks and contacts, delegation, impersonation access to other accounts.

from exchangelib import DELEGATE, Account, Credentials

credentials = Credentials(
    username='MYWINDOMAIN\\myusername', 
    password='topsecret'
)
account = Account(
    primary_smtp_address='john@example.com', 
    credentials=credentials, 
    autodiscover=True, 
    access_type=DELEGATE
)
# Print first 100 inbox messages in reverse order
for item in account.inbox.all().order_by('-datetime_received')[:100]:
    print(item.subject, item.body, item.attachments)

[nicolas33]

You're right. I've read more about this. I don't think we want offlineimap to become a client of HTTP webservices.

[chris001]

I understand the reluctance to take on web services. It's maybe a dilemma. However, if we could let the libraries fully handle the protocol communications, and only create/read/update/delete encapsulated abstract email message related objects produced or consumed by these libraries, as long as the mails sync properly with all imap as well as all newer EWS-enabled microsoft mail servers, I'm OK with offlineimap using a library which uses any protocol, such as EWS which happens to use SOAP/JSON/XML, to sync mails. All we care about is the mails and sync'ing them with the top mail servers on the internet.

[nicolas33]

Sure! My point is more about the source code. It's highly tied to IMAP. Actually, offlineimap is written as a (big) wrapper of IMAP. I believe this would be hard to add any other protocol to the code base.

I might be wrong, though. If you'd like to try this, please do! I wonder setting up a test suites would be great to track regressions while making strong changes like that in the code.

[chris001]

I'm a huge supporter of test suites, without them, you're flying blind in the fog for sure going to accidentally release versions with preventable bugs.

offlienimap (or maybe imapfw) could be refactored into a top layer which has no knowledge or preference of protocols, like an orchestra conductor, compare two mailboxes, decide a sync strategy, produce a sync action list, and execute it.. And a deeper layers, one of which talks IMAP to imaplib2. Another equal layer talking to exchangelib. Each deep layer would consuming login credentials hostname port and protocol, and produce abstract portable objects of messages, mailbox views, mailbox lists, flag sets, tag sets, etc, for the top layer to work on. Only missing element to make this happen is funding.

[nicolas33]

Yes, that's what I've started to do in imapfw.

[chris001]

Three layers would do this perfectly.
Top layer: orchestra conductor runs the generic sync algorithm. User Input: email addresses. Data input from Middle layer: generic abstract portable mail related objects necessary for sync. Output: sync success or fail.
Middle layer: Converts email addresses to full server connection based on autodiscover/autoconfig. Talks to Lowest layer protocol libraries using their APIs (imaplib2 and exchangelib). Output: generic abstract portable message related objects needed by Top layer. Produces the same type of objects, no matter what the lowest layer protocol library.
Lowest layer protocol libraries: imaplib2, exchangelib, etc.