SPF record is too long

[kelson42]

We would like to inform you that our scan has identified one problem with your SPF record. These issue prevent SPF from working correctly and as a result, emails sent from your domain can be forged.
Specifically, we have found the following issue:

Too many DNS lookups

The SPF record requires more than 10 DNS queries and is therefore invalid [3]. Currently, the SPF record triggers 34 requests. We recommend replacing some domain names with IP addresses.
If you are uncertain whether your SPF entry is correct, there are online tools that can help you, such as:

• https://www.mailhardener.com/tools/spf-validator
• https://www.spf-record.com/spf-lookup/

[kelson42]

I had briefly a look to:

• https://www.autospf.com/
• https://github.com/smck83/expurgate

[rgaudin]

That's for kiwix.org

https://mxtoolbox.com/SuperTool.aspx?action=spf%3akiwix.org&run=toolpage

[benoit74]

The main problem seems to be that we are using many services to send emails on behalf of our domain, and each services is using many sub-services / IPs:

• gaggle.email
• email.freshdesk.com
• cyon.ch
• _mailcust.gandi.net

Is there any service we do not use anymore? Or any we could switch to use gandi SMTP servers (with proprer authentication of course...)?

I digged a bit into the SPF records (with https://gist.github.com/benoit74/35cb8b01d3a6aa4a91ad985a5de9ed57) and this is what I found:

kiwix.org: 11 instructions, among which 4 includes
  gaggle.email: 5 instructions, among which 3 includes
    _spf.firebasemail.com: 4 instructions, among which 2 includes
      _spf.google.com: 5 instructions, among which 3 includes
        _netblocks3.google.com: 12 instructions, among which 0 includes
        _netblocks2.google.com: 8 instructions, among which 0 includes
        _netblocks.google.com: 13 instructions, among which 0 includes
      sendgrid.net: 12 instructions, among which 1 includes
        ab.sendgrid.net: 6 instructions, among which 0 includes
    helpscoutemail.com: 8 instructions, among which 0 includes
    _spf.gaggle.email: 14 instructions, among which 1 includes
      amazonses.com: 13 instructions, among which 0 includes
  email.freshdesk.com: 7 instructions, among which 5 includes
    fdspfaus.freshemail.io: 12 instructions, among which 0 includes
    fdspfind.freshemail.io: 12 instructions, among which 0 includes
    fdspfeuc.freshemail.io: 13 instructions, among which 0 includes
    fdspfus.freshemail.io: 13 instructions, among which 1 includes
      fdspfus2.freshemail.io: 6 instructions, among which 0 includes
    sendgrid.net: 12 instructions, among which 1 includes
      ab.sendgrid.net: 6 instructions, among which 0 includes
  cyon.ch: 9 instructions, among which 4 includes
    _spf.mailrelay.rrpproxy.net: 3 instructions, among which 1 includes
      spf.key-systems.net: 5 instructions, among which 0 includes
    servers.mcsv.net: 5 instructions, among which 0 includes
    helpscoutemail.com: 8 instructions, among which 0 includes
    spf.patchman.co: 3 instructions, among which 0 includes
  _mailcust.gandi.net: 3 instructions, among which 1 includes
    _nblcust.gandi.net: 10 instructions, among which 0 includes

gaggle.email seems to be pretty deeply nested / using many subservices.

cyon.ch and freshdesk are also not very good.

[rgaudin]

We use them all ; I cleaned it up already in May

[rgaudin]

I think there are at least those options:

• Pay a fee to a flattening service. That's the easiest. We pay (autofps is $324/y), change the SPF once initially and that's it.
• Setup a dynamic solution like expurgate which adds a SPOF for emails on our infrastructure as with this setup, every recipient's email provider triggers at least one DNS call to this service. We'd also need to make the initial SPF record change and we have to maintain it (keep it alive, mostly).
• Bake something ourselves. There are probably pieces of code doing this but it's relatively simple anyway: recusrively loop over our sources, query/parse SPF and get IPs (de-duping). Could run on GH CI and either update record (I believe GANDI has an API) or just fail CI so we manually update records (I imagine those 20 names dont change IPs every week). Could be an intern/student task to build.

---

From what I understand, we don't need Gaggle in our SPF record. We do use it but it doesn't send from an @kiwix.org address. Gaggle is a mailing-list/groups service and we have a handful of those (@Popolechien please share Gaggle credentials btw). We thus has address list shortcut@kiwix.org that gets redirected into an @gaggle.email address and Gaggle sends all email with a From of his own.

I've thus removed it from the SPF record. We're still way above the limit (at 20 instead of 10).

I've also reconfigured (via a Plugin) WP to send emails via Mailgun and I've also configured the MailPoet Plugin to send via Gandi SMTP. Those changes allows us to remove the cyon include from the SPF record.

We're now at 12 records instead of 10 (arg!). The remaining ones are: Gandi (for SMTP), Mailgun and Freshdesk.

Freshdesk is our helpdesk and we can't spare it but… I just checked and we can configure it to use Gandi's IMAP and SMTP instead of the current setup (SPF to use From: xxx@kiwix.org and incoming via a redirect to xxx@kiwix.freshdesk.com).

Given how nested freshdesk records are, it's clear we can stay within 10 once we remove it. I haven't done it (not my call) but it's an easy switch.

---

For future reference, here is an exhaustive list (best effort) of our services sending email:

Service	From	Via
Freshdesk	contact@kiwix.org	FreshDesk
Regular Emails	*@kiwix.org	Gandi SMTP
Wordpress MailPoet Plugin	hello@kiwix.org	Gandi SMTP
Wordpress	stephane@kiwix.org	Mailgun
Cardshop-scheduler	hotspot@kiwix.org	Mailgun
Cardshop-manager	hotspot@kiwix.org	Mailgun
Matomo	-	-
Kiwix Wiki	wiki@wiki.kiwix.org	Mailgun
openZIM Wiki	wiki@wiki.openzim.org	Mailgun
Zimfarm API	info@farm.openzim.org	Mailgun
youzim.it Zimfarm API	-	-
youzim.it frontend	info@youzim.it	Mailgun

[rgaudin]

Freshdesk has been migrated to a new IMAP BAL. Its include statement has been removed and now our SPF is valid 🎉

Check for yourself