bigmailer: OpenBSD smtpctl spf walk starter kit

This facility is listed as OpenBSD-specific as it depends on both the pf(8) facility and the smtpd(8) daemon’s helper program, smtpctl(8). That said, the output of smtpctl spf walk is text, and text is easy to reform. If you find a way to use this on other platforms, great - we’d love to be cross-platform.

Bit o' history

When the internet was young, everybody ran their own mail server - singular. SPAM came along. GreyListing was one of several good responses to it. Then big mailers grew multiple servers AND were too lazy to implement "server affinity" - keep trying to send a given message from a given IP. Instead, they send a message from a randomly-chosen server each time they retry after the greylisting delays it…​ This broke greylisting, as the message would come from a different IP each time, which would get added to the ever-growing grey list, but the message would rarely get through.

Fortunately, there’s a solution to this problem too. Sender Policy Framework (SPF) requires mail senders to list all the hosts (or groups, using CIDR notation) that their mail can come from. So now, smtpd’s helper program smtpctl has the capability to read the SPF information from each of a list of "big mailer" domains, and list them in a form which our pfctl can accept.

HowTo for OpenBSD users

TL:DR; for OpenBSD-current as of 2018-01-11, or 6.3+:

1. copy bigmailers.txt to /etc/mail/;

2. Set a pf rule to let table spf-white bypass the spamd redirection;

3. Install the body of scripts/run in a weekly root cron entry or in /etc/weekly.local

4. Optionally use the scripts in scripts for debugging/analysis.

Assuming you have installed file called bigmailers.txt (from this repo) into /etc/mail, and maintain your local changes in bigmailers-local.txt, you can just execute something like this, either by hand or out of cron (weekly?):

# cat /etc/mail/bigmailers*.txt | smtpctl spf walk | pfctl -t spf-white -T add -f -

or, if you don’t want a local set:

# smtpctl spf walk < /etc/mail/bigmailers.txt | pfctl -t spf-white -T add -f -

The first form is what the run script in the scripts folder does. Assuming you have pf redirect incoming SMTP into greylisting, and you have a filter rule that bypasses hosts in the table spf-white, mail from the big mailers will come right in. Along with any SPAM their customers send, of course.

pass in quick on egress proto tcp from <spf-white> to any port smtp
pass in quick on egress proto tcp from any to any port smtp \
			 rdr-to 127.0.0.1 port spamd

It is good that pf is very efficient at handling tables, as the very first version of bigmailers.txt in this repo, 12 domains, generated almost 500 IP netgroups. After removing one redundant entry:

$ doas scripts/walk | wc -l
	 451
$

HowTo for non-OpenBSD users

This will be added when you contribute it.

Contribute

Pull requests for "major" bigmailers that don’t do server affinity will be gratefully accepted, as long as they:

1. keep the list in alphabetical order;

2. don’t list your local public school or Cadillac dealer that has two mail servers, as most of the world will never get email from them;

3. provide proof in the comments field, such as spamd output showing two more more different IPs trying to send the same message, like so:

Jan 13 17:32:13 darwinsys spamd[24416]: (GREY) 1.2.3.123: <spammer@bigmailer.com> -> <covfefe123@darwinsys.com>
Jan 13 17:38:63 darwinsys spamd[24416]: (GREY) 1.2.3.456: <spammer@bigmailer.com> -> <covfefe123@darwinsys.com>

Alternately, big or important sites may be listed just to stop greylisting them.

Pulls for making things work on other platforms will also be considered, as long as they don’t detract from the main focus.

Pull requests constitute acceptance of the (un)license in LICENSE.txt. Contributions become public domain.

The original code contributions that made this possible are documented here.