feat: Auth - OAuth2 (Dovecot PassDB)

[thechubbypanda]

Description

I decided to start on an implementation of Oauth2 as described in #2713.

I am testing this locally with my Authentik instance and a roundcube instance configured as per their documentation.

Original details prior to PR feedback

I'm setting the following new variables as such:

ACCOUNT_PROVISIONER=OAUTH2
OAUTH2_CLIENT_ID=id
OAUTH2_CLIENT_SECRET=secret
OAUTH2_TOKENINFO_URL=https://authentik.domain.com/application/o/userinfo/?access_token=

The current oauth2 login (that's all I've done so far) accepts xoauth2 tokens from something like roundcube using the dovecot passdb xoauth2 mechanism.

• It disables the default passdb and so users can't login with login or plain despite the fact that the configuration allows it and so all email clients that don't support generic oauth cannot login.
• This could have been mitigated by using password grant but this has been "deprecated" (is no longer best practice).

With the current changes, I am able to:

• Login with roundcube
• Send emails (albeit with a few errors discussed below)

Implementation TODOs

• Figure out a system for provisioning accounts such that they can receive emails before login
• Figure out a system for querying account changes on a periodic basis
• Figure out a way to define aliases in the oauth2 provider such that the query can read them
• Figure out a way to define quotas in the oauth2 provider such that the query can read them

Questions

Out of the 3 options listed in the tracking issue regarding account provisioning, which is going to be the best solution?

The first option is implemented here, using a static userdb.

The third option sounds like it would be the optimal one however there have been points made against this.

• However Mailcow has recently implemented oauth2 (in their nightly branch and they have gone about it in an interesting way.
• It appears they are adding userinfo and querying that using the already supplied credentials on a cron job or equivalent. This sounds like a very generic and simple solution to the problem and one I believe we can yoink.

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

PR Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (README.md or the documentation under docs/)
• If necessary I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[polarathene]

I'll possibly be able to contribute feedback / review sometime this month. The other maintainers are also a bit busy, so just a heads-up if it takes a bit of time 😅 (I've got a bit of a backlog to address first in DMS and elsewhere)

[thechubbypanda]

I'm realising that there is really no way to solely use oauth2 if you also want to use any generic imap client like thunderbird or almost everything on mobile. I'm aware this was mentioned in the original issue.
The solution for this sounds like finding a way to do password authentication via the oauth2 provider or requiring a second form of auth such as ldap or static...
Authentik for example does have an LDAP provider but the whole idea for this was to not have to use LDAP.

Essentially this is just an update comment saying that I'm looking into proxying passwords to the auth backend. Unfortunately it doesn't look promising as grant password has been deprecated... although there might be some light at the end of the tunnel.

Until the next update o7

[polarathene]

Hey sorry for the silence about this, caught up working elsewhere on the project.

While working with the LDAP support, I have been wondering if the OAuth2 / OIDC support is meant to be an account provisioner or just an alternative authentication feature DMS could support?

---

Probably nothing below that you don't already know; However, I am forgetful this is more of a refresher for me when I return to this 😝

I don't have time to go over the existing discussion for the feature or this PR right now, but flow wise we have:

• Inbound mail delivered to Postfix, which queries Dovecot if the user exists (UserDB) to accept the mail.
• If user exists and mail is accepted, Dovecot will create the mailboxes in /var/mail as configured to do so.
• Mail submission to Postfix requires authentication, which again delegates to Dovecot (even with LDAP by default, Dovecot will auth agains the LDAP provider). This is not the case when using ENABLE_SASLAUTHD=1, which could be to an IMAP provider (again Dovecot or any other alternative) or LDAP (no Dovecot indirection to LDAP this time).
• Mail retrieval from Dovecot storage (/var/mail) involves authentication, which like with Postfix delegating auth to Dovecot, involves Dovecots PassDB, which may have overlap with the UserDB but doesn't need to.
• Both PassDB and UserDB can be configured to have multiple backends, trying one and falling back to another lookup if unsuccessful.

With the current LDAP and FILE provisioners, both auth and accounts are configured to use that provisioner. ENABLE_SASLAUTHD=1 is only configured for Postfix at present, thus Dovecot is either configured by FILE lookups or LDAP queries, never a mix.

---

I'm realising that there is really no way to solely use oauth2 if you also want to use any generic imap client like thunderbird or almost everything on mobile. I'm aware this was mentioned in the original issue.
The solution for this sounds like finding a way to do password authentication via the oauth2 provider or requiring a second form of auth such as ldap or static...

From what I understand, the OAuth2 backend may be sufficient if mail clients are restricted to being compatible with that, however some deployments may want to support traditional means. Some IDPs like Keycloak can support both OIDC and LDAP IIRC, which should allow centralizing the account management/provisioning through that service, as you don't really want to try keep two different account sources in sync (eg: static) 😅

Thus DMS probably needs to rethink the account management support, refactoring how we configure Dovecot PassDB/UserDB? I can try give that some thought after I bring LDAP support into better shape.

[thechubbypanda]

Good consolidation, yeah. Most of that rings true, I have some ideas but life has caught up with me and I'll be busy for a while. I'll keep an eye out for mentions here in case others want some help taking over if I'm not back to finish this soon.

[MohammedNoureldin]

Thank you for this PR. I am wondering if using LDAP in this case will keep working with OIDC? Because you mentioned that the standard long is not working anymore.

[thechubbypanda]

The thing stopping oAuth2 from being it's own fully blown auth provider is that lack of query-ability. As such, for now it will likely just be a layer over top for something like roundcube to make use of, but still falling back to something like LDAP.

I'm thinking of just reducing the scope of this PR to that for now and creating another for the final form;

The final form being implementing adapters for different oAuth2 provider APIs such as Authentik, KeyCloak, etc. to allow user querying. This would allow for oAuth2 to be its own fully blown auth provider.

[MohammedNoureldin]

Being its own fully OAuth2 provider and falling back to other providers like LDAP is the best solution and so should it be, IMO. Is there any time estimation when you will working on it?

[thechubbypanda]

I have some time this weekend, hopefully to get the minimum use case done in this PR.

[MohammedNoureldin]

Great! I am excited! Let me know if I can somehow support.

[MohammedNoureldin]

The comment here is interesting:

nextcloud/mail#5933 (comment)

Would this change the way you are going to implement this PR? It just attracted my attention, because it is talking about the most modern way of authenticating E-Mail users.

@polarathene could you please also take a look on it?

[polarathene]

@polarathene could you please also take a look on it?

I don't know what you want me to take from that. I don't see it adding anything helpful to the discussion here.

• This PR is tackling the Dovecot support for OAuth2 via passdb. That handles authentication.
• Dovecot userdb is the other part with actual accounts provisioned. SSO provides the authentication, and sometimes additional data about a user, but AFAIK not something we use to manage Dovecot specific account data such as quota limit, maildir, etc.

You would provision accounts and have same auth as we do currently AFAIK. This PR will provide an alternative authentication mechanism. There is another PR contributing Dovecot auth via Lua scripts as another alternative (but won't be officially supported by DMS, only via a community guide in the docs), it's been discussed there that in future DMS may need to rework this support to better separate the account provisioning choice and enabled auth backends for Dovecot to use.

It may be a couple months before I have time to spare for any of this. Backlog is already full.

[thechubbypanda]

I'd agree with @polarathene, that doesn't really change the situation. For now it's just going to be enabling oauth2 in dovecot as an alternate login method allowing for things like SSO webmail login.
Unfortunately most mail clients implement specific workarounds for each major provider and not a generic redirect that we could use here. If this were the case, I would likely have it implemented and working already.

The Lua scripts sound ideal for the potential next phase of this implementation which is to make oAuth2 a full blown auth provider. They would be used to retrieve user lists for mailbox creation etc.

[polarathene]

Unfortunately most mail clients implement specific workarounds for each major provider and not a generic redirect that we could use here.

It's been a while since I looked into OAuth / OIDC, but I remember OIDC resolving that issue (or it may have been OAuth2?). I do recall before that I would see different libraries for languages that would maintain a massive list of OAuth provider integration, but AFAIK that became much simpler with OIDC (OpenID Connect).

Perhaps one option is to add a bridge of sorts that provides the OAuth2 providers if needed, but ideally OIDC if that has good compatibility these days, then for integrating with Dovecot I suppose it might be possible to route the OAuth2 feature through that internal DMS service? 🤔

I've not worked on such myself before, so it may be too much work as-is.

I assume Dovecots mention of a Proxy Backend for OAuth2 might be related to this for offloading the auth to a separate service / container.

---

If it's easier taking the per provider route, then this would be less valuable to DMS and might be better suited similarly to the Lua support via our docs? We can add some integration and some provider support, but rather than DMS maintaining providers it may be better as a BYO guide from the docs? 🤷‍♂️

That way the community can contribute provider support more easily and the providers that DMS users express enough interest in can be part of DMS.

EDIT: I assume this is about the Dovecot OAuth2 "Backends" examples?

---

EDIT: Here's an old discussion about using KeyCloak for the OIDC service to connect with any other services, and Dovecot integration (which has some interesting information about their implementation). KeyCloak would be too big for DMS to bundle, so they'd need to use a separate image, but I believe there are some Go or Rust based projects that might be an option if there is any benefit bundling such a service into DMS.

I may have misunderstood that interaction and it changes nothing to the Dovecot OAuth2 provider issue 😅

UPDATE: Oh, there is already OIDC support in Dovecot? 🤔

• It was part of the 2.3.16 release in Aug 2021, but OIDC Discovery was apparently broken until a fix in 2.3.21 release in Sep 2023
• If that changes anything, perhaps we should hold-off until we can get a newer Dovecot release?

[thechubbypanda]

Ok I have this working as of now with authentik and roundcube. The user has to exist beforehand in dms for it to work as we are not providing dovecot a userdb, only a passdb. Ergo, either ldap or file provisioner must be used and the user added beforehand.
Regular login using username and password still works.
The 2 passwords are not linked in any way, they only grant access to the same account.

@MohammedNoureldin can you confirm this before I add tests and document?
My compose.override.yml looks like this for reference:

services:
  mailserver:
    image: docker.io/library/mailserver-testing:ci
    hostname: mail.domain.com
    environment:
      ENABLE_OAUTH2: 1
      OAUTH2_CLIENT_ID: verySecretId
      OAUTH2_CLIENT_SECRET: verySecretSecret
      OAUTH2_INTROSPECTION_URL: https://authentik.domain.com/application/o/userinfo/

and a roundcube configuration reference can be found here

[MohammedNoureldin]

Ok I have this working as of now with authentik and roundcube.

Sounds great! Thank you!

The user has to exist beforehand in dms for it to work as we are not providing dovecot a userdb, only a passdb. Ergo, either ldap or file provisioner must be used and the user added beforehand.

Should this be the case? I mean, in case of LDAP, we are also not storing anything related to the user, but rather fetching the data directly from the LDAP server, and the mails are accordingly stored in the folder of the user. Why can't this be the case with OIDC? I mean it should be somehow possible to log in to the mail also for the first time. Or did I misunderstood the point here?

[polarathene]

Should this be the case? I mean, in case of LDAP, we are also not storing anything related to the user, but rather fetching the data directly from the LDAP server

LDAP service has the account information already, including mail specific data such as aliases for that user.

Postfix and Dovecot talks to the LDAP service to retrieve this information.

and the mails are accordingly stored in the folder of the user.

That is a folder Dovecot manages responsibility for, nothing to do with LDAP.

To accept the mail, the account must exist from the provisioner. Dovecot UserDB provides that information, which for LDAP involves talking to the LDAP service to verify IIRC.

Why can't this be the case with OIDC?
I mean it should be somehow possible to log in to the mail also for the first time.

You can authenticate with Dovecot via the PassDB, this then optionally provides some information to UserDB IIRC.

Even if that works fine, if you have not logged in, then how should Dovecot handle the situation when Postfix provides some new mail for the account? Dovecot needs to know it exists to accept the mail, so how will it check via UserDB?

OIDC is just providing authentication AFAIK. Dovecot needs to know not only if a user exists, but any additional information such as mail storage location, quota limit, etc. Many of these we can provide defaults on. I know that OIDC allows supporting some complimentary user data, but not sure if it's that flexible for the Dovecot related user data.

LDAP support has Dovecot perform a lookup query to the LDAP service, usually with an admin account or similar permissions to grant querying user data.

LDAP also supports auth (PassDB), like our file provisioner. But AFAIK OIDC is only able to support the PassDB functionality, not UserDB, hence why you need something else providing that data to Dovecot.

[MohammedNoureldin]

Please excuse my delay. I had some private circumstances and thus I could not do any progress. Now I available again and would be very glad to help! I see that you made a lot of progress here. Well done guys! Please let me know if I can support somehow.

[polarathene]

Well done guys! Please let me know if I can support somehow.

You can give it a try if you like. Let me know if you run into any problems, like the docs not providing enough information to setup for you.

How important the ENV (OAUTH_*) is to you (since this only introduces one explicitly), vs just providing the config directly (slightly less convenient than it needs to be atm).

This feature is otherwise approved and will get merged for v14 🎉

---

One concern that might be worth raising awareness on is with this integration, if your third-party auth provider let's you configure the email, and there isn't any trust in that being verified by the user, that authentication to Dovecot grants access to the Dovecot account, even if you're not the actual owner of the address 🤔

Usually there should be additional information like email_verified, and I think Dovecot can additionally match that, or select other claims to source a value as part of passdb attributes (eg: some id-hash that Dovecot UserDB provides and the third-party auth provider needs to include in returned claims to establish trust. However that's not really something we can offer and would need custom integration AFAIK 😅 )

Non-issue if the auth service is self-hosted and this sort of account registration concern isn't applicable.

[github-actions]

Documentation preview for this PR is ready! 🎉

Built with commit: acfc44d

[polarathene]

Approving and merging for v14.

If I have time to revise an update to the oauth2 docs / ENV before then, I will but at this stage I may not have the time for it :(

---

Thanks for contributing this new feature! ❤️

[MohammedNoureldin]

Thanks guys. I have been trying to get it working with NextCloud, still I have some points to clarify.

Just one point regarding its release, this is a sensitive feature, I would release it in a separate minor version (13.3.0) ASAP and mention that it is under review rather than major version with many other big changes. What do you think?

[polarathene]

this is a sensitive feature

Why is it sensitive? You need to opt-in regardless.

---

I would release it in a separate minor version (13.3.0) ASAP

Unfortunately our release process isn't well suited for that if we've already merged changes that would be semver incompatible:

• PRs get merged to master branch, that publishes the latest :edge image.
• master branch gets tagged which triggers a release. We do this via the GH releases feature which will automatically tag the latest commit and trigger CI publishing workflows once we publish a release manually / explicitly.
• To support what you propose requires adapting the CI to handle release branches, which would be nice but no one has had enough time to warrant prioritizing that.

Thus once breaking changes are added to master, we can only release a major version. Likewise once features are released we cannot push a bug fix release.

Looking at the current CHANGELOG.md, if @georglauterbach doesn't consider the rspamd update a breaking change, it does look like we could publish a v13.3 release.

We have improved our review process to add changelog entries before merging now, so much of that friction to releases is now gone 💪

---

mention that it is under review rather than major version with many other big changes.

Presently the only big change due for v14 is the base image upgrade from Debian Bullseye to Debian Bookworm, and all the package updates that brings.

I would like to get the LDAP breaking change in as well, and I may be able to get that resolved in time.

Other than that, I don't think we have anything else that should be breaking / big.

---

What do you think?

It would be good to get v13.3 out for the jaq ARM fix, so those users get notified of the v14 release (presently broken with v13.2).

I'm comfortable with v13.3 if the other maintainers are too.

[georglauterbach]

I think the Rspamd changes are not breaking as we define it. I'm fine with releasing v13.3.

UPDATE: ref #3781

[MohammedNoureldin]

@polarathene thank you for the explanation.

I said sensitive because it is about authentication in general, which is the first part the user need to be able to interact with the email. And thus any small issue may potentially end up that users are not being able to login, for example. That is why it seemed sensitive to me.

@georglauterbach thanks for confirming this!

This sounds great, so we can soon have this in 13.3.0.

[polarathene]

I said sensitive because it is about authentication in general, which is the first part the user need to be able to interact with the email. And thus any small issue may potentially end up that users are not being able to login, for example. That is why it seemed sensitive to me.

Fair. In that case do you want the breaking changes LDAP PR to be it's own major release instead of bundled with the base image upgrade?

[georglauterbach]

I'd prefer to have it in one release; less work on our side 🤣

[polarathene]

less work on our side 🤣

I think the reasoning was the release separation benefits maintainers too when it comes to troubleshooting issues related to releases that introduced breaking changes.

The LDAP changes are large enough that it'd be easier to troubleshoot when you rule out the changes the base image upgrade PR introduces. Thus the separate releases is probably less work maintenance wise 😝

[MohammedNoureldin]

I am neutral here :D you can decide whether you would like to put them in one or two PRs.

Actually I have been trying to test, but for some reason I am not able to login even using LDAP, it is not related to this change as I am trying with 12.1.0. I get dovecot: auth error failed operations error. I will keep trying for a few hours yet, maybe I can get it working. Otherwise I will open a discussion asking for help ^^".

[polarathene]

I get dovecot: auth error failed operations error.

Set ENABLE_QUOTAS=0 to opt-out of this Dovecot feature. It may help avoid some confusion if it introduces additional logs about dovecot: auth error, although with LDAP that quotas feature isn't available AFAIK, so probably not relevant to you 😅

---

for some reason I am not able to login even using LDAP, it is not related to this change as I am trying with 12.1.0

Are you saying you cannot login with LDAP? Nothing should have changed there and our tests pass auth:

docker-mailserver/test/tests/serial/mail_with_ldap.bats

Lines 378 to 384 in 2bf5234

	_send_email \
	--port 465 -tlsc \
	--auth LOGIN \
	--auth-user some.user@localhost.localdomain \
	--auth-password secret \
	--quit-after AUTH
	assert_output --partial 'Authentication successful'

It's possibly related to your LDAP config and DMS related settings for query filters (and result attribute). You do not need to enable SASLAuthd, just ensure configuration for Postfix and Dovecot is correct, refer to how we've done so in our LDAP test (related LDAP config for users is by .ldif files here):

docker-mailserver/test/tests/serial/mail_with_ldap.bats

Lines 74 to 90 in 2bf5234

	# When using SASLAuthd for login auth (only valid to Postfix in DMS),
	# Postfix will pass on the login username and SASLAuthd will use it's backend to do a lookup.
	# The current default is `uniqueIdentifier=%u`, but `%u` is inaccurate currently with LDAP backend
	# as SASLAuthd will ignore the domain-part received (if any) and forward only the local-part as the query.
	# TODO: Fix this by having supervisor start the service with `-r` option like it does `rimap` backend.
	# NOTE: `%u` (full login with realm/domain-part) is presently equivalent to `%U` (local-part) only,
	# As the `userID` is not a mail address, we ensure any domain-part is ignored, as a login name is not
	# required to match the mail accounts actual `mail` attribute (nor the local-part), they are distinct.
	# TODO: Docs should better cover this difference, as it does confuse some users of DMS (and past contributors to our tests..).
	local SASLAUTHD_QUERY='(&(userID=%U)(mailEnabled=TRUE))'
	# Dovecot is configured to lookup a user account by their login name (`userID` in this case, but it could be any attribute like `mail`).
	# Dovecot syntax token `%n` is the local-part of the full email address supplied as the login name. There must be a unique match on `userID` (which there will be as each account is configured via LDIF to use it in their DN)
	# NOTE: We already have a constraint on the LDAP tree to search (`LDAP_SEARCH_BASE`), if all objects in that subtree use `PostfixBookMailAccount` class then there is no benefit in the extra constraint.
	# TODO: For tests, that additional constraint is meaningless. We can detail it in our docs instead and just use `userID=%n`.
	local DOVECOT_QUERY_PASS='(&(userID=%n)(objectClass=PostfixBookMailAccount))'
	local DOVECOT_QUERY_USER='(&(userID=%n)(objectClass=PostfixBookMailAccount))'

---

I've already verified that this feature is compatible with our LDAP auth tests, so I know it works :)

The LDAP config will be changed and the related docs page completely rewritten to accommodate it. Another change will affect SASLAuthd LDAP integration with Postfix as I'm not happy with it introducing an inconsistency.

As this is getting offtopic, feel free to open an issue/discussion when you have more information to share and ping me there 👍

[georglauterbach]

I think the reasoning was the release separation benefits maintainers too when it comes to troubleshooting issues related to releases that introduced breaking changes.

The LDAP changes are large enough that it'd be easier to troubleshoot when you rule out the changes the base image upgrade PR introduces. Thus the separate releases is probably less work maintenance wise 😝

I see, makes sense. Then let's please merge #3403 first because I really want to finally have Debian 12 (to change some things in my personal setup).

[MohammedNoureldin]

I have no clue anymore. I am not able to test this because, as mentioned, for some reason I cannot connect to LDAP anymore, which really frustrating after 20 hours of trying. I tested also 12.1.0, so it is not related to the changes made here. Though, any help would be great so I can do extensive test to this feature. To keep this PR clean I opened a new discussion for it. https://github.com/orgs/docker-mailserver/discussions/3785

[MohammedNoureldin]

Thanks guys. I have been trying to get it working with NextCloud, still I have some points to clarify.

As an update, after a lot of tests I was NOT able to get it working with NextCloud. However, it turned out (I got an official statement regarding this), that Nextcloud Mail does not forward the access tokens apparently the the mail server. So I stopped working on it at the moment.