Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting up a mail server with OpenSMTPD, Dovecot and Rspamd #17

Open
poolpOrg opened this issue Sep 14, 2019 · 35 comments

Comments

@poolpOrg
Copy link
Owner

commented Sep 14, 2019

No description provided.

@leo-unglaub

This comment has been minimized.

Copy link

commented Sep 14, 2019

Awesome article, thanks for all your work!

@mchack23

This comment has been minimized.

Copy link

commented Sep 14, 2019

neat, been waiting for this article for some days now since I read your "mail's not hard" article just when I was trying to set up an openbsd smtpd server myself.
Since I'm still flailing a bit (it is working but I managed to create local mail loops for instance), I'll try and recreate and tweak your setup next :) Hints on how to set up virtual users are most welcome (and how I can relay local user mail to those virtual users)

Anyway: Found a small typo it should say options.inc (not options.conf) in this line: "This is done by adding the dkim_signing filter to the filters key configuration in /etc/rspamd/options.conf:"

When I find more I'll edit this post. :) Thanks again Gilles!

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 14, 2019

@mchack23 typo fixed

the source markdown for the articles are in this repository, feel free to submit pull requests for typos and such :-)

@myfirstnameispaul

This comment has been minimized.

Copy link

commented Sep 14, 2019

I have been using Vultr 1GB VPS for about a year with a LEMP server spitting out transactional mail without issue. A few months ago I moved my MiaB over to the same service and haven't had any real issues, never submitted a ticket for any sort of filtering. Are you using a European DC?

Have you found dnswl.org to be useful? I see a lot of the big guys are there.

I look forward to your article on Microsoft because I'm not sure it can be done, as well as the DKIM key size as this is the first I've been aware of this.

While I originally taught myself how to configure my own mail server, I moved to MiaB in order on a simple VPS because as a personal mail server, it's fantastic and trouble-free, as much as a mail server can be.

However, I am wondering if you ever cover capacity limits? For some of the organizations I think you are targeting, they may quickly max out a VPS so capacity planning seems like a very relevant topic. What is considered for CPU limits in a mail server? What is using up memory and how painful is swap? How much storage should I plan on using, especially when forecasting growth rates? Are there some rules of thumb for any of these? Can I share block storage with more than one server? Reverse proxy? Etc...

I have also wondered about IPv6. What happens when spammers can just use a new IP address for every email and blast you a million times a second into eternity? Can I get by with just IPv4?

@benwaffle

This comment has been minimized.

Copy link

commented Sep 14, 2019

If reading about how you can build such setups is of any interest, I could also write about it to help people build MORE hosting alternatives which is ultimately what I hope for.

Yes, I'm interested.

@superkuh

This comment has been minimized.

Copy link

commented Sep 14, 2019

The hard part is 5 years later when your current mailserver's OS goes end of life and you have to port everything to a new setup using different versions of things.

@cmouse

This comment has been minimized.

Copy link

commented Sep 15, 2019

I'd like to suggest incorporating this in your howto:

OpenSMTPD could use LMTP (or execute dovecot-lda, not sure which works in OpenSTMPD) to deliver mail via dovecot, the benefit is that then dovecot updates it's indexes immediately, and you can benefit from sieve filtering for incoming mail. Then you can make dovecot filter e.g. spam messages to Spam folder directly.

@ynakao

This comment has been minimized.

Copy link

commented Sep 15, 2019

Thanks for the great writeup. As @benwaffle mentioned, I'm also interested in large-scale email server setup especially about virtual users and multiple domain managements.

I will write instructions assuming the upcoming 6.6.0 release which is due in a few weeks now. It brings a ton of improvements and I don’t want the instructions to be obsoleted and need a rewrite in October.

Out of curiosity, which is the OpenSMTPD v6.6.0 specific settings? Will there be similar breaking changes on OpenSMTPD in the future?

@omoerbeek

This comment has been minimized.

Copy link
Contributor

commented Sep 15, 2019

PR #18, I think the flag does not match the text.

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 15, 2019

@myfirstnameispaul

I have been using Vultr 1GB VPS for about a year with a LEMP server spitting out transactional mail without issue. A few months ago I moved my MiaB over to the same service and haven't had any real issues, never submitted a ticket for any sort of filtering. Are you using a European DC?

For my own needs, I'm using multiple DC (Amsterdam, Frankfurt, London, Paris) all in Europe yes.

Have you found dnswl.org to be useful? I see a lot of the big guys are there.

In all honesty no, I know dnswl.org but I can't even recall if I ever subscribed mine, so they may be useful but you can certainly do without.

I'm 100% positive that I didn't subscribe most of the domains I plugged.

However, I am wondering if you ever cover capacity limits? For some of the organizations I think you are targeting, they may quickly max out a VPS so capacity planning seems like a very relevant topic.
What is considered for CPU limits in a mail server?
What is using up memory and how painful is swap?
How much storage should I plan on using, especially when forecasting growth rates?

I don't because it really depends on what you are going after.

I documented a very simple setup which would work fine for a lot of people on the cheapest VPS, because that's how I test these setups and even load them a little, but how people need to handle capacity planning depends on many factors.

On my own setup, Rspamd instances run on machines of their own so I can adjust them depending on CPU and RAM usage, whereas Dovecot runs on a separate machine with decent disks, whereas my OpenSMTPD instances are basically CPU-bound, delivering the messages to a remote machine and not the local disk. My planning here is per-service and since services are per-machine, this is something I'd expect a company to do but not a lambda user, my recipes won't work for them.

Then, even on a simpler setup, some users will want MariaDB to handle their accounts, others will want an elk stack to monitor, others will want a webmail and maybe some fancy log analysers, and depending on where you run these, it changes radically how you do your planning. A machine could run very fine with your mail infra and load like hell just with elastic installed, so you might need to go for a real dedicated server or much higher-frequency CPUs just for one of the components.

I think capacity planning is not very different with mail than with any other service and, as far as I'm concerned, it goes by graphing and monitoring how things evolve in time to adjust, I don't have a secret works-for-all recipe. Some more competent SREs surely have better solutions though.

A very cheap VPS will do as they are more powerful than most mail servers from the last decade and the software have not grown more hungry in resources (only speaking about opensource solutions, not the commercial ones which will plain not work on a VPS).

For Rspamd, I have no idea what the minimal resources are, but I had never had a problem running it on the cheapest VPS I could get my hands on. It mostly does CPU, that's about all.

For Dovecot, this is VERY dependant on how many users you have, how they consume their mail, if they search mailboxes and such, so again I had never had a problem running on the cheapest VPS BUT your mileage may vary and graphing resources consumption to adjust is key. It mostly does CPU and DISK but is also mostly idle if not on a busy server.

For OpenSMTPD, not only the cheapest VPS will do but SMTP servers are mostly idle, so testing the limits is quite difficult. It can run (not just build but actually run) on very cheap machines, people use Raspberry boards, as well as on some ancient or not-so-powerful architectures supported by OpenBSD (https://www.openbsd.org/plat.html). It does CPU and DISK but it can also deliver remotely (what I do) in which case it mostly does CPU when it's not idle, even on a relatively busy server.

A lot of people thing that a lot of CPU is needed for all the TLS handling but this is not correct, I could share stories about that ;-p

I don't observe variations in terms of memory usage across time and on a server that has been running for a while, the memory consumption of just mapping the Go runtime is an order of magnitude higher than the memory consumption of any process from Dovecot or OpenSMTPD for instance.

When looking into how good I'm with resources, my main concern is if I should add disk or not.

Are there some rules of thumb for any of these? Can I share block storage with more than one server? Reverse proxy? Etc...

The sharing of block storage is a tricky part and depends on what system you're using.
If using Maildir backend, you're supposedly safe when sharing block storage because Dovecot keeps its state internal to a Maildir, and OpenSMTPD delivers atomically and lockless inside Maildir.
Then questions arise if you're not only sharing block storage but also clustering / cloning the SMTP instances or the IMAP instances. Both are technically doable, not necessarily in a very tricky way, but I would not say it's easy.

Reverse proxying is simple in both cases, both Dovecot and OpenSMTPD can support proxy-v2 and as a matter of fact I'm using an haproxy myself.

I have also wondered about IPv6. What happens when spammers can just use a new IP address for every email and blast you a million times a second into eternity? Can I get by with just IPv4?

That's a problem.

But since it's not globally solved by Big Mailer Corps, IPv6 remains insignificant when it comes to mail as of today and you can certainly just get by with IPv4.

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 15, 2019

@cmouse

I'd like to suggest incorporating this in your howto:

OpenSMTPD could use LMTP (or execute dovecot-lda, not sure which works in OpenSTMPD) to deliver mail via dovecot, the benefit is that then dovecot updates it's indexes immediately, and you can benefit from sieve filtering for incoming mail. Then you can make dovecot filter e.g. spam messages to Spam folder directly.

I'd rather see people write such howtos ;-)

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 15, 2019

@ynakao

Thanks for the great writeup. As @benwaffle mentioned, I'm also interested in large-scale email server setup especially about virtual users and multiple domain managements.

I will write instructions assuming the upcoming 6.6.0 release which is due in a few weeks now. It brings a ton of improvements and I don’t want the instructions to be obsoleted and need a rewrite in October.

Out of curiosity, which is the OpenSMTPD v6.6.0 specific settings?

All the filter stuff comes with v6.6.0 so while you could have rspamd, dkim and such, it was done through a slighly trickier method of "queue reinjection" where you tell OpenSMTPD to relay to an Rspamd proxy or a DKIM proxy and you tell the proxy to re-enqueue in the daemon.

It was not necessarily much more complex in terms of configuration but it was a bit more complex in terms of understanding how it works and far less elegant than just adding a filter keyword to a listen line :-)

Will there be similar breaking changes on OpenSMTPD in the future?

In this case the change is not breaking, the configurations that worked before would still work, but they would be far less elegant and efficient at solving the issue, or would even be missing some of the improvements.

That being said, OpenSMTPD is a very active project with a release cycle of six months and we'll have breaking changes every now and then because we'd rather have users change a keyword than let technical debt accumulate.

We have had a very big breaking change a year ago where the configuration file was completely revamped, this was the first time since the project was first started ten years ago, so a huge breaking change should not happen before another decade :-) but it's likely that within the next year, we'll change a keyword here or there to help improve the project.

@poolpOrg poolpOrg self-assigned this Sep 15, 2019
@cmouse

This comment has been minimized.

Copy link

commented Sep 15, 2019

I'd like to suggest incorporating this in your howto:

OpenSMTPD could use LMTP (or execute dovecot-lda, not sure which works in OpenSTMPD) to deliver mail via dovecot, the benefit is that then dovecot updates it's indexes immediately, and you can benefit from sieve filtering for incoming mail. Then you can make dovecot filter e.g. spam messages to Spam folder directly.

I'd rather see people write such howtos ;-)

Perhaps. But I think it would make sense to involve dovecot on the delivery phase always, as it would enable much more functionality in dovecot and would allow dovecot to update it's indexes during delivery.

@jirib

This comment has been minimized.

Copy link

commented Sep 16, 2019

My only concern for HA mail setup are mails waiting in a queue to be delivered, eg. a mail server in DMZ with paused delivery aka "deffered relaying" (cf 9.2.1 Postponing Mail Delivery in Postfix: Definitive Guide; would that be doable with smtpd -P mta?) and which waits for a manual trigger to try to deliver via SMTP to a mail server in intranet. If the mail server with this setup would fail, the mails will be lost. IIRC 'extras' used to allow Python plugins which could save mails into redis etc. to have a queue "distributed"...

@epsilon-0

This comment has been minimized.

Copy link

commented Sep 16, 2019

sweet as fuck!

@jungle-boogie

This comment has been minimized.

Copy link

commented Sep 17, 2019

Great article and very detailed. Thanks for taking the hours to craft this together and get everything in a functioning state.

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 17, 2019

@jirib

My only concern for HA mail setup are mails waiting in a queue to be delivered, eg. a mail server in DMZ with paused delivery aka "deffered relaying" (cf 9.2.1 Postponing Mail Delivery in Postfix: Definitive Guide; would that be doable with smtpd -P mta?) and which waits for a manual trigger to try to deliver via SMTP to a mail server in intranet. If the mail server with this setup would fail, the mails will be lost. IIRC 'extras' used to allow Python plugins which could save mails into redis etc. to have a queue "distributed"...

I don't understand the "if the mail server with this setup would fail, the mails would be lost" part because unless it got a clear rejection or acceptation, it would not lose them, you are describing what a backup MX does: buffer mail until primary host is back again.

If what you are scared about is if the backup MX dies while the primary MX is up, then I would say that there is a simpler approach than distributed queue for this: delocalized queue. If you can setup a distributed filesystem, then its easier to point OpenSMTPD to that mounted share.

You should not try to make OpenSMTPD use a plugin to distribute queue across multiple instances, as it would also mean that the scheduler needs to know about these, so you'll need to write a distributed scheduler plugin too.

@kr1pt0ph0b14

This comment has been minimized.

Copy link

commented Sep 17, 2019

Great article! Thanks for all your work! <3 I just wish it was covering a Roundcube installation/integration as well. :/

@jirib

This comment has been minimized.

Copy link

commented Sep 17, 2019

@jirib

My only concern for HA mail setup are mails waiting in a queue to be delivered, eg. a mail server in DMZ with paused delivery aka "deffered relaying" (cf 9.2.1 Postponing Mail Delivery in Postfix: Definitive Guide; would that be doable with smtpd -P mta?) and which waits for a manual trigger to try to deliver via SMTP to a mail server in intranet. If the mail server with this setup would fail, the mails will be lost. IIRC 'extras' used to allow Python plugins which could save mails into redis etc. to have a queue "distributed"...

I don't understand the "if the mail server with this setup would fail, the mails would be lost" part because unless it got a clear rejection or acceptation, it would not lose them, you are describing what a backup MX does: buffer mail until primary host is back again.

I wanted to say that mail is already accepted by primary MX acting as incoming mail gateway/proxy.

If what you are scared about is if the backup MX dies while the primary MX is up, then I would say that there is a simpler approach than distributed queue for this: delocalized queue. If you can setup a distributed filesystem, then its easier to point OpenSMTPD to that mounted share.

Yes, a distributed filesystem would be ok but there's no one on OpenBSD. (Yes, there's NFS or (i)SCSI from a NAS/SAN box attached to multiple mail servers but that's not "distributed" FS.)

You should not try to make OpenSMTPD use a plugin to distribute queue across multiple instances, as it would also mean that the scheduler needs to know about these, so you'll need to write a distributed scheduler plugin too.

Or... a kind of resource management/clustering, where only one instance would consume this distributed queue. Or... using some kind of DB based on raft.

@jdinel

This comment has been minimized.

Copy link

commented Sep 18, 2019

I'd love to see an addendum to this article that shows how to setup virtual users for dovecot to use. I'm currently building an OpenBSD/OpenSMTPD mail server for my whole family, using your excellent article, but I'm very reticent to creating local user accounts for every member of my family.

@jdinel

This comment has been minimized.

Copy link

commented Sep 18, 2019

The hard part is 5 years later when your current mailserver's OS goes end of life and you have to port everything to a new setup using different versions of things.

You mean a year, right? OpenBSD provides security fixes for a release for one year, not 5.

Also, configuring everything through an automator like Ansible makes re-creating your setup every year pretty painless.

@superkuh

This comment has been minimized.

Copy link

commented Sep 18, 2019

You mean a year, right? OpenBSD provides security fixes for a release for one year, not 5.

Yikes, but no. I was referencing my own current problems with my 5 year old debian mailserver on a VPS and the problems of running your own mailserver long term in general. Back then I followed the guides from https://workaround.org/ispmail much like people will likely follow the guides here now. But in some years what's in the guides will no longer apply. And I severely doubt that any automater is automagic enough to account for that kind of thing unless you know the configuration yourself first.

It's easy enough to set up a new clean setup every $x years. It's hard, and painful, to port existing config and data.

@jamesdtyler

This comment has been minimized.

Copy link

commented Sep 19, 2019

Hurrah, for this document. I believe all of the answers that kept me from self-hosting my own server have been addressed. Email, pure and simple, is the key to my shackles. I look forward to the other documents that you propose. They are also needed.

@kr1pt0ph0b14

This comment has been minimized.

Copy link

commented Sep 20, 2019

Would you add few addon lines, words regarding how to setup virtual users for dovecot? Many thanks once again!

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 20, 2019

I'll write other articles to tackle specific setups, this one was just to show that you could get a general setup done :-)

@julienXX

This comment has been minimized.

Copy link

commented Sep 20, 2019

Hey great article! I follwed it with OpenBSD 6.5 on vultr and can successfully login manually via openssl s_client -connect mail.t***.***:993 or openssl s_client -connect mail.t***.***:imap -crlf -starttls imap.
But impossible using thunderbird or iOS Mail. I always get dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<> for thunderbird and I can login but the connection is closed immediately using iOS Mail. Does it ring a bell on what the issue could be by any chance?

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 20, 2019

mh, doesn't ring a bell, I'll check with an ipad later today :-)

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 20, 2019

47947F6C-9D85-4586-8370-FB81421B34CE

Works fine for me :-/

You should enable debug logging in Dovecot, this might help understanding where the issue occurs.

@iodous

This comment has been minimized.

Copy link

commented Sep 21, 2019

Is dkim_signing actually needed in options.inc? I'm running on -current and my logs say:

2019-09-21 19:38:41 #1265(main) <8k84xk>; cfg; rspamd_init_filters: requested unknown module dkim_signing
2019-09-21 19:38:41 #1265(main) <8k84xk>; cfg; rspamd_init_lua_filters: init lua module dkim_signing

@vinnyt

This comment has been minimized.

Copy link

commented Sep 23, 2019

awesome article! Would love any pointer to setting up virtual users. Thanks again!

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 30, 2019

@iodous well if you want dkim signing it is needed for sure :-) what version of Rspamd are you using ?

@vinnyt thanks, it's a popular request so I'll write about it at some point :-)

@vstakhov

This comment has been minimized.

Copy link

commented Sep 30, 2019

filters is used just for C modules not for Lua ones. Lua modules are automatically enabled if their's section is represented in the config file. In fact, this filters stuff is nothing but legacy (as well as C modules in general). dkim_signing is a Lua module, so it shouldn't be added in filters.

@poolpOrg

This comment has been minimized.

Copy link
Owner Author

commented Sep 30, 2019

oh nice, wasn't aware so this means it'll simplify instructions further, will update :-)

Thanks @vstakhov !

@mikejsavage

This comment has been minimized.

Copy link

commented Oct 17, 2019

I think you should add a note about rspamd log rotation. Just found that mine had grown to 90MB :)

echo '/var/log/rspamd/rspamd.log      _rspamd:_rspamd 640  7     *    24    Z "pkill -USR1 -u root -U root -x rspamd"' >> /etc/newsyslog.conf
@mt7479

This comment has been minimized.

Copy link

commented Oct 17, 2019

Hey great article! I follwed it with OpenBSD 6.5 on vultr and can successfully login manually via openssl s_client -connect mail.t***.***:993 or openssl s_client -connect mail.t***.***:imap -crlf -starttls imap.
But impossible using thunderbird or iOS Mail. I always get dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<> for thunderbird and I can login but the connection is closed immediately using iOS Mail. Does it ring a bell on what the issue could be by any chance?

Following the howto on 6.6 I had to change the following bits in 10-auth-conf to get dovecot working:

disable_plaintext_auth = no
auth_username_format = %n

IMO this is due to the local username authentication. You also have to specify the bare username (not username@domain) for both imap/smtp.

/cc @poolpOrg ^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.