[EPIC] Automatic account setup / server settings autodiscovery

[cketti]

Thanks to @wiktor-k (PR #3879 + #3980) we have code to retrieve server settings from Thunderbird's ISPDB. Thanks to @livmackintosh (PR #4719) we'll soon have code to retrieve server settings via DNS SRV records (see RFC 6186).
But in order to actually use these methods to automatically set up an account, K-9 Mail's code needs to be changed quite a bit. In this issue I'll try to list all requirements and propose a plan for an automatic account setup feature.

Goal

The goal is to make it as easy as possible for users to add their email account to K-9 Mail. Ideally, that means they only need to provide their email address and the app is figuring out the rest. In practice the authentication process also requires some kind of user interaction (provide password, allow access to an account using an OAuth workflow etc).
It should still be possible to manually provide server settings, but we want to avoid asking users for their server settings if possible at all.

Requirements

• The process needs to be secure. It should not be possible for a man-in-the-middle attacker to get the app to automatically send the user secret to an untrusted server. When server settings are found, but can't be trusted automatically, we ask the user to confirm using them.
• The process should be reasonably fast. In practice that means not trying all supported method sequentially, but in parallel.

Autodiscovery methods we want to support (ordered by preference)

• bundled providers.xml (we'll remove all entries that are covered by another method)
• RFC 6186 DNS SRV records
• Configuration server at ISP (Thunderbird method)
• Thunderbird's ISPDB
• Use MX record to look up provider in the ISPDB (useful when using a custom domain with a known provider)
• Guessing (imap.example.com, smtp.example.com etc)

Suggested approach

I think the process can be divided into these steps:

1. Asking for the user's email address
2. Finding server settings
3. Testing server settings and finding out which supported authentication methods are offered by the server
4. Asking the user for authentication credentials
5. Using the credentials to log in and maybe retrieve additional server capabilities to make the final decision on whether or not K-9 Mail supports this email account

Most methods to retrieve server settings hit the network and might therefore be slow. That's why we should run them simultaneously. Step 3 can be considered as part of the previous step. But different autodiscovery methods might return the same server settings and we don't want to test them twice. So we need a mechanism to collect the results and only test unique server settings once. But we also don't want to wait for all autodiscovery methods to finish step 2 to test server settings in step 3. Ideally, we respect the order in which the autodiscovery methods return server settings. But there should be a timeout value X after which we take the most preferable server settings that tested successfully without waiting for any, potentially better, matches.

Example:
Step 2: The DNS SRV lookup returns "imap" and "imaps" results.
Step 3: The test for the "imaps" result completes successfully in 2 seconds. The test for the "imap" result is still running after X (e.g. 10) seconds.
In this case we'd cancel the remaining tests in step 3 and select the "imaps" server settings.

In practice we probably want to delay some of the autodiscovery methods in step 2. We for example don't want to start with guessing server names right away.

My suggestion is to group the methods:

• Group 1:
  • bundled providers.xml
• Group 2:
  • RFC 6186 DNS SRV records
  • Configuration server at ISP
  • Thunderbird's ISPDB
• Group 3:
  • Use MX record to look up provider in the ISPDB
• Group 4:
  • Guessing

Step 2 for the next group is started once the current group finishes step 3 or X seconds have passed.

For all of this to work smoothly we have to move away from the current UI code where checking the server settings is performed in a separate Activity. We also don't want to ask for the password right away, because we might not need it, e.g. when using OAuth.

Related: #865

[wiktor-k]

Hi @cketti, one thing that may be relevant: I see you want to have a consistent onboarding experience but when the user wants to use their existing Android account (via OAuth flow) asking them to type their e-mail address in full (e.g. venkatanarasimharajuvaripeta@gmail.com) and then offering OAuth would take longer than just clicking a button and selecting the account from the list.

Maybe this could be approached by offering an e-mail autocompletion (where the suggestions would be account names, I don't remember right now what permissions are necessary to get the list of accounts).

[cketti]

@wiktor-k: At the moment I'm heavily leaning towards not supporting the Android account based authentication flow. A generic OAuth workflow using the browser would work for providers other than Google, also with Gmail accounts not set up on the device.
Auto-completing email addresses is something we could try to do regardless. But if feels like this should be discussed in another issue.

[wiktor-k]

Okay, sounds good 👍 I hope this webview can be used with hardware tokens but I think this should be possible.

[dilyanpalauzov]

Concerning the Thunderbird autodiscovery, the description at https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration is incomplete. When the ISP does not offer <https://autoconfig.example.com/mail/config-v1.1.xml?emailaddress=fred@example.com> address, but offers <https://example.com/.well-known/autoconfig/mail/config-v1.1.xml>, then Thunderbird in fact attaches ?emailaddress=fred@example.com to the query. But the persons behind the Wiki insist not to write this in the protocol description.

Note that the protocol description does not say what the output will be, if the input email address does not exist.

If you manage to change the wiki to write, that K-9 also supports this model, then mention also Evolution. But it does not send the email address. It sends instead EVOLUTIONUSER@ followed by the right domain, followed by &emailmd5= and then the md5 of the email address.

Note that until last year the autoconfig discovery was done over plain HTTP:// and until 2023 Thunderbird will check both https://autoconfig.domain and http://autoconfig.domain.

[cketti]

When the ISP does not offer <https://autoconfig.example.com/mail/config-v1.1.xml?emailaddress=fred@example.com> address, but offers <https://example.com/.well-known/autoconfig/mail/config-v1.1.xml>, then Thunderbird in fact attaches ?emailaddress=fred@example.com to the query.

I don't see a problem doing this. Our threat model considers all subdomains of the email domain as trusted. Our requests will all use transport encryption (HTTPS).

If you manage to change the wiki to write, that K-9 also supports this model, then mention also Evolution. But it does not send the email address. It sends instead EVOLUTIONUSER@ followed by the right domain, followed by &emailmd5= and then the md5 of the email address.

Is this done unconditionally or only when communicating over plain HTTP?

[dilyanpalauzov]

The only problem with the undocumented behaviour is, when different clients behave differently on the same configuration files.

The EVOLUTIONUSER replacement is done unconditionally at https://gitlab.gnome.org/GNOME/evolution/-/blob/master/src/mail/e-mail-autoconfig.c#L245

[cketti]

It feels like Evolution needlessly changed the protocol in a way that makes it incompatible with that of Thunderbird in some cases. I don't see a reason to adopt that change to the protocol.

We will document the exact mechanism used by K-9 Mail once we're done.

[boyska]

Hello! I know this is slightly off topic, but would it be possible to have a summary of the current status?

As a mail server administrator, I'd like to make life easier to users which use K9. What can I do to help them?

[linuxgokan]

Can we use rfc6186 for autodiscover? Is this working now?
If not, is there a planned date?

https://datatracker.ietf.org/doc/html/rfc6186

[cketti]

Current beta versions support Thunderbird's autoconfig mechanism, but not RFC 6186. We'll add support for it eventually. But we can't give a timeframe.

[linuxgokan]

So when will the full version be released? I'm planning to change servers. I will do a system update. We have 150+ users. I will recommend K9 to all of them.

[richardleger]

Hi,

Following confirmation from cketti that K-9 Mail 6.603 stable doesn't support autoconfig(*), I have managed to install the beta version 6.717 (via the F-Droid > K-9 Mail > Versions options) on Android 8.0.0.

When I open the app, enter my email address + Next, I get "Configuration Not Found" warning, inviting me to setup manually, while autoconfig works fine in Thunderbird and I can access the configuration file on our server via direct URL in web browser.

On our server the access.log show the following:
[21/Feb/2024:11:10:48 +0000] "GET /mail/config-v1.1.xml HTTP/1.1" 302 594 "-" "okhttp/4.12.0"

Which seems to suggest that the Autoconfig process is omitting the required "?emailaddress=richard@mydomain.co.uk" parameter in the HTTP request... which is necessary to identify the user and get the proper related configuration file for that user.

(*): #7655 (comment)
Fyi, I was mislead by this article (https://blog.thunderbird.net/2023/06/thunderbird-for-android-k-9-mail-may-2023-progress-report) that seems to say that Thunderbird’s Autoconfiguration (https://mzla.link/autoconfig) was integrated since May 2023. The Release Notes not clearly stating either at which version autoconfig was made available. Also difficult to find in the commit list. I would suggest you add a line Release Notes to indicate in which version feature was introduced.

I thought to report the issue in the hope it might help fix it, as this is an important long awaited feature ;-)

Regards,

[kewisch]

Hi folks. This issue has a number of great ideas on how to improve automatic account setup. However, it has fulfilled its purpose for the work that we've scoped for the project. Our focus has been mainly towards aligning with Thunderbird Desktop's autoconfig and making sure that the majority of use cases are covered.

There are a few features that didn't quite make it in, if you are particularly excited about one of the things listed here I'd suggest checking other open issues to see if we have an individual issue, or file a new one. Bonus points if you'd like to try them out as a contributor, we'd love to have your support!

[richardleger]

Hi,

Is there any plan to fix the autoconfig bug reported previously #4721 (comment), I recently updated my beta version and it seems the bug is still present.

Cheers,

[cketti]

See https://forum.k9mail.app/t/autoconfig-url-emailaddress-not-supported/8336