November 2018: OpenSMTPD released and upcoming filters preview

[poolpOrg]

https://poolp.org/posts/2018-11-03/opensmtpd-released-and-upcoming-filters-preview/

[kaey]

Filters protocol looks very similar to communigate
https://www.communigate.com/communigatepro/Helpers.html#Filters
https://www.communigate.com/communigatepro/VirusScan.html

Congrats on releasing it!

[leo-unglaub]

The new filter design is absolutley amazing. I love it. It allows people to quickly write a filter in a simple shell script but it also allows you to redirect the filter-requests to a different (central filtering server) and process there. It's simply amazing.

I have already written a SPF filter in Rust. Works perfect.
Thanks for all your work!

[poolpOrg]

@kaey I wasn't aware but it's not really surprising, I didn't do rocket science, the interesting part is not so much the protocol but rather the split between events reporting and filtering, where a filter may or not depend on reporting to build its state, this and the read from stdin / write to stdout logic which makes it trivial to plug with any language. Thanks for your comment.

@leo-unglaub oh my, I created a monster... :-)

[ghewgill]

This looks great! Some requests:

• Filter on link-connect event (for connection rejection before welcome message)
• Some way to invoke spamd-like tarpit behaviour (slow responses), even if the remainder of the session has to be handled within the filter
• A way to pass a tag back from a filter to smtpd
• Consider a filter data format option, eg. JSON might make more sense for some filter implementations than pipe-delimited

[poolpOrg]

This looks great! Some requests:

• Filter on link-connect event (for connection rejection before welcome message)

This is already implemented, I use it myself

• Some way to invoke spamd-like tarpit behaviour (slow responses), even if the remainder of the session has to be handled within the filter

This can be done using proc filters which actively slow down responses.

I have implemented a PoC filter which paused 5 seconds for each reply to
a particular IP address.

• A way to pass a tag back from a filter to smtpd

This will be done, it was even discussed yesterday on IRC but I want the
DATA part to be finished first.

Tags are trickier because we already have a tagging mechanism which does
not operate at the same level so, technicity set apart, we need to think
how these two mechanisms will impact each other.

• Consider a filter data format option, eg. JSON might make more sense for some filter implementations than pipe-delimited

That's a nope :-)

I don't want to deal with multiple formats in OpenSMTPD itself.

If a filter wants to work with json, in most languages that provide json
facilities it is trivial to convert |-delimited lines into json objects,
even using one-liners:

>>> import json
>>> json.dumps('report|smtp-in|protocol-server|1541271222|3189ac6874354895|250 poolp.org Hello localhost [127.0.0.1], pleased to meet you'.split('|'))
'["report", "smtp-in", "protocol-server", "1541271222", "3189ac6874354895", "250 poolp.org Hello localhost [127.0.0.1], pleased to meet you"]'
>>>

It doesn't make sense to me that OpenSMTPD should start supporting json,
otherwise next week someone will want XML and there will be no reason to
not implement that format, until someone comes with a thirds, etc...

Filters get to convert the simple format into what they want, and this way
our MTA gets to not depend on a json library :-)

[jdelic]

@poolpOrg

After you pointed me to your blog from my well intentioned, but utterly misguided PR at OpenSMTPD/OpenSMTPD-extras#58, I guess this is the right forum to ask a follow-up question.

From a quick reading of lka_filter.c, I was unable to find out what the process model for filters actually is. The same is true from reading the smtpd man pages and OpenSMTPd. Perhaps you can just help clear this up for me:

• What is the opensmtpd process model (async green threads/one thread per email (with/without pooling)/one process per email (with/without pooling)?
• How does that process model relate to filters? Are they spawned per-message? Can a filter assume that it will process one message per invocation?
• If not, especially for "costly startup" like a filter written in Python would have, what would you suggest?
• Your example at https://github.com/poolpOrg/filter-rspamd/blob/master/filter-rspamd.go seems to be single-threaded and long running. Is OpenSMTPd spawning new instances as needed for parallel deliveries?

I apologize if I overlooked some easily accessible information that would have answered those questions. Thank you for your work and all that you do.

[poolpOrg]

@jdelic

Very simple:

A filter is simply a process fork()-ed by OpenSMTPD at startup and which is expected to run an infinite loop reading reporting events and filtering requests from stdin, writing filtering responses to stdout.

What your process does internally is up to you: you can run with a single thread if you don't do blocking tasks, you can run with a single thread and event-based programming, you can run with a single thread and co-routines (or related), you can run with multiple threads, etc... the only rule is that you're not allowed to block.

One process is enough to handle concurrent deliveries so a single mono-thread filter, like the filter-rspamd you gave as example, is enough to handle all concurrent sessions, there is no need to spawn multiple instances.

If you write a cpu-bound filter, one instance is not enough, which I doubt, it is up to your filter to spawn subinstances.