Implement SPF/DMARC checks, add spam weight to those mails

[jvolkenant]

Fixes #1755

Comments welcome.

[yodax]

I've installed this on my machine. Works like a charm! Thanks for this 👍

[JoshData]

Looks like it's operating correctly for me. I've gotten two false positives so far (which for me is a lot, not surprisingly the previous Mail-in-a-Box defaults worked flawlessly for me), in the sense that they were legitimate emails but seemed to have improperly configured SPF.

[downtownallday]

The default score for softfail appears to be 1.0.

X-Spam-Report: 
	* -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
	*  1.0 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail)

[downtownallday]

A score of 5.0 for SPF_FAIL won't always send the message to spam due to negative scoring like in the above. Not sure if that was intended or not.

[downtownallday]

If I drop off a message to the submission port, there is no SPF information at all in the delivered message.

Is a DKIM signature required for SPF checks?

Return-Path: <test@guest.com>
Delivered-To: alice@somedomain.com
Received: from vanilla.local ([127.0.0.1])
	by vanilla.local with LMTP id uM2eA+w1p18rIQAACeoDfA
	for <alice@somedomain.com>; Sun, 08 Nov 2020 00:03:56 +0000
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on vanilla.local
X-Spam-Level: 
X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=ham
	autolearn_force=no version=3.4.2
X-Spam-Report: 
	* -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
X-Spam-Score: -1.0
Received: from authenticated-user (vanilla.local [10.0.2.15])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by vanilla.local (Postfix) with ESMTPS id E0F2982B63
	for <alice@somedomain.com>; Sun,  8 Nov 2020 00:03:54 +0000 (UTC)
Authentication-Results: vanilla.local; dkim=none; dkim-atps=neutral
From: test@guest.com
To: alice@somedomain.com
Subject: Test
Message-Id: <20201108000355.E0F2982B63@vanilla.local>
Date: Sun,  8 Nov 2020 00:03:54 +0000 (UTC)

This is a test

[Detect syslog errors]
[   OK] Nov  8 00:03:54 vanilla postfix/submission/smtpd[8485]: connect from unknown[10.0.2.15]
[   OK] Nov  8 00:03:55 vanilla postgrey[1619]: action=pass, reason=triplet found, client_name=unknown, client_address=
10.0.2.15/32, sender=test@guest.com, recipient=alice@somedomain.com
[   OK] Nov  8 00:03:55 vanilla postfix/submission/smtpd[8485]: E0F2982B63: client=unknown[10.0.2.15]
[   OK] Nov  8 00:03:55 vanilla postfix/cleanup[8489]: E0F2982B63: replace: header Received: from vanilla.local (unknow
n [10.0.2.15])??(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))??(No client certificate requested)??b
y vanilla.local (Postfix) with ESMTPS id E0F2 from unknown[10.0.2.15]; from=<test@guest.com> to=<alice@somedomain.com> 
proto=ESMTP helo=<vanilla.local>: Received: from authenticated-user (vanilla.local [10.0.2.15])??(using TLSv1.3 with ci
pher TLS_AES_256_GCM_SHA384 (256/256 bits))??(No client certificate requested)??by vanilla.local (Postfix) with ESMTPS 
id E0F2982B63??for <alice@somedomain.com>; Sun,  8 Nov 2020 00:03:54 +0000 (UTC)
[   OK] Nov  8 00:03:55 vanilla postfix/cleanup[8489]: E0F2982B63: message-id=<20201108000355.E0F2982B63@vanilla.local>
[   OK] Nov  8 00:03:56 vanilla postfix/qmgr[5907]: E0F2982B63: from=<test@guest.com>, size=634, nrcpt=1 (queue active)
[   OK] Nov  8 00:03:56 vanilla postfix/submission/smtpd[8485]: disconnect from unknown[10.0.2.15] ehlo=2 starttls=1 ma
il=1 rcpt=1 data=1 quit=1 commands=7
[   OK] Nov  8 00:03:56 vanilla spampd[2393]: processing message <20201108000355.E0F2982B63@vanilla.local> for <alice@s
omedomain.com>
[   OK] Nov  8 00:03:57 vanilla spampd[2393]: clean message <20201108000355.E0F2982B63@vanilla.local> (-1.00/5.00) from
 <test@guest.com> for <alice@somedomain.com> in 1.21s, 669 bytes.
[   OK] Nov  8 00:03:57 vanilla postfix/lmtp[8490]: E0F2982B63: to=<alice@somedomain.com>, relay=127.0.0.1[127.0.0.1]:1
0025, delay=2.9, delays=1.5/0.03/0.01/1.4, dsn=2.0.0, status=sent (250 2.0.0 <alice@somedomain.com> uM2eA+w1p18rIQAACeo
DfA Saved)
[   OK] Nov  8 00:03:57 vanilla postfix/qmgr[5907]: E0F2982B63: removed

[downtownallday]

What happens if a milter timeout occurs (http://www.postfix.org/MILTER_README.html#timeouts)? Which timeouts apply for SPF checks? Even if you summed up all the milter timeouts they're still significantly shorter than what is recommended by Ubuntu with policyd-spf-python (https://help.ubuntu.com/community/Postfix/SPF).

[myfirstnameispaul]

@downtownallday

What about the other SPF result types, especially "softfail"?

The default score for softfail appears to be 1.0.

X-Spam-Report: 
	* -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
	*  1.0 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail)

The PR configuration does not differentiate between ~ and - in the results. In my observation, this is common for most mail servers. In my server, a fail of a record with ~ does not result in spam score points quoted in your post.

A simple failure of SPF does not need to cause the message to be declared as spam. An SPF failure is usually accompanied with other modifiers, so will either be declared not spam or spam based on other scores in the message.

If I drop off a message to the submission port, there is no SPF information at all in the delivered message.

The primary purpose of this change, as referenced in the issue linked to this PR (which I would request you please review every post there), is to make spoofed emails landing in an inbox non-trivial. Issues related to trusted hosts I would consider outside the scope of my original issue.

Is a DKIM signature required for SPF checks?

No.

What happens if a milter timeout occurs (http://www.postfix.org/MILTER_README.html#timeouts)? Which timeouts apply for SPF checks? Even if you summed up all the milter timeouts they're still significantly shorter than what is recommended by Ubuntu with policyd-spf-python (https://help.ubuntu.com/community/Postfix/SPF).

Why does policyd-spf-python make those recommendations? I have been using this configuration for close to 6 months now. No problems to report and reviewing recent logs it seems every lookup takes less than 2 logged seconds. Please specify under what conditions would a user experience these issues and how they would be identified.

[downtownallday]

SPF_SOFTFAIL score (1.0) occurs when "Received-SPF: Softfail" is present in the mail headers. For me, that occurs because I'm using policyd-spf, which adds it. If you don't care to differentiate, cool.

The dkim milter appears to be the only milter attached to submission. I guess this means incoming mail from submission is not SPF checked. That appears to be true in this configuration. policyd-spf does SPF checks on submission mail, btw.

Regarding timeouts. I don't have answers. I presume they're related to bind timeouts. No idea what those are. Maybe 30 seconds for a single dns query, worst case according to this serverfault post (#1836 (comment)) on bind10. There might be more than 1 query, right?

But forgetting all that. The question I posed was "what happens if a milter timeout occurs?" I don't know, but I thought it might be nice to know.

Thanks for this great work

[myfirstnameispaul]

I also used policyd-spf-python when I configured my own mail servers, but with MiaB I was trying to use the tools already installed in the project. I did not discover this difference of the timeout length when coming up with a suggested configuration.

I may be misunderstanding the postfix page, but it seems the timeout refers to each request to a milter, so the OpenDMARC requests are separate from other requests. When I run grep "mail opendmarc" /var/log/mail.log.1 and scroll down, I can't find a single instance of a lookup taking longer than 2 seconds. I haven't tried to analyze all of my logs more precisely (not really sure how to easily do this), but I've done this a few times on different days, and the results are the same, assuming this is a valid analysis anyway.

[JoshData]

As I said in #1755 but repeating here, I think folks are in agreement that this is ready to be merged. Speak now or forever hold your peace. :) If there are no objections, I'll merge. I really appreciate all the efforts here both in figuring out the configuration changes and in the discussions vetting its reliability.