Rspamd DKIM unsigned #3598

ceeim · 2023-10-23T09:21:16Z

📝 Preliminary Checks

I tried searching for an existing issue and followed the debugging docs advice, but still need assistance.

👀 What Happened?

Rspamd DKIM unsigned, sending emails to mail-tester prompts this issue. Can someone tell me where the problem is? I have no issues when using OpenDKIM.

dkim_signing.conf:

# documentation: https://rspamd.com/doc/modules/dkim_signing.html

enabled = true;

sign_authenticated = true;
sign_local = false;
try_fallback = false;

use_domain = "header";
use_redis = false; # don't change unless Redis also provides the DKIM keys
use_esld = true;
allow_username_mismatch = true;

check_pubkey = true; # you want to use this in the beginning

domain {
    mail-1.com {
        path = "/tmp/docker-mailserver/rspamd/dkim/rsa-2048-mail-mail-1.com.private.txt";
        selector = "mail";
    }
    mail-2.com {
        path = "/tmp/docker-mailserver/rspamd/dkim/rsa-2048-mail-mail-2.com.private.txt";
        selector = "mail";
    }
}

👟 Reproduction Steps

No response

🐋 DMS Version

edge

💻 Operating System and Architecture

ARM64

⚙️ Container configuration files

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:edge
    container_name: mailserver
    # If the FQDN for your mail-server is only two labels (eg: example.com),
    # you can assign this entirely to `hostname` and remove `domainname`.
    hostname: mx
    domainname: test.com
    env_file: mailserver.env
    # More information about the mail-server ports:
    # https://docker-mailserver.github.io/docker-mailserver/edge/config/security/understanding-the-ports/
    # To avoid conflicts with yaml base-60 float, DO NOT remove the quotation marks.
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
    dns:
      - 1.1.1.1
      - 1.0.0.1
      - 2606:4700:4700::1111
      - 2606:4700:4700::1001
    volumes:
      - ./docker-data/mail-data/:/var/mail/
      - ./docker-data/mail-state/:/var/mail-state/
      - ./docker-data/mail-logs/:/var/log/mail/
      - ./docker-data/config/:/tmp/docker-mailserver/
      - ./docker-data/custom-certs/:/tmp/dms/custom-certs/:ro
    environment:
      - OVERRIDE_HOSTNAME=hostname.test.com
      - ONE_DIR=1
      - ENABLE_DNSBL=1
      - ENABLE_FAIL2BAN=1
      - ENABLE_FETCHMAIL=1
      - FETCHMAIL_POLL=60
      - ENABLE_RSPAMD=1
      - ENABLE_OPENDKIM=0
      - ENABLE_OPENDMARC=0
      - ENABLE_POLICYD_SPF=0
      - ENABLE_AMAVIS=0
      - ENABLE_SPAMASSASSIN=0
      - RSPAMD_CHECK_AUTHENTICATED=1
      - RSPAMD_GREYLISTING=1
      - RSPAMD_LEARN=1
      - RSPAMD_HFILTER=1

    restart: always
    stop_grace_period: 1m
    cap_add:
      - NET_ADMIN
      - SYS_PTRACE
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0

📜 Relevant log output

mailserver  | Oct 23 16:59:35 mx postfix/submissions/smtpd[11904]: 977791825AB: client=ec1.2.3.4.ap-northeast-1.compute.amazonaws.com[1.2.3.4], sasl_method=PLAIN, sasl_username=postmaster@mailserver.com
mailserver  | Oct 23 16:59:35 mx postfix/sender-cleanup/cleanup[12193]: 977791825AB: message-id=<79c47a2c-a106-4b4f-99cc-530d65441a83@mailserver.com>
mailserver  | Oct 23 16:59:35 mx postfix/sender-cleanup/cleanup[12193]: 977791825AB: replace: header MIME-Version: 1.0 from ec1.2.3.4.ap-northeast-1.compute.amazonaws.com[1.2.3.4]; from=<postmaster@mailserver.com> to=<test-hckvzdc1l@srv1.mail-tester.com> proto=ESMTP helo=<[10.0.0.118]>: MIME-Version: 1.0
mailserver  | Oct 23 16:59:35 mx postfix/qmgr[955]: 977791825AB: from=<postmaster@mailserver.com>, size=8137, nrcpt=1 (queue active)
mailserver  | Oct 23 16:59:36 mx dovecot: imap-login: Login: user=<postmaster@mailserver.com>, method=PLAIN, rip=1.2.3.4, lip=172.27.0.2, mpid=12254, TLS, session=<dmsIcV4IPL0N5t7u>
mailserver  | Oct 23 16:59:37 mx dovecot: imap(postmaster@mailserver.com)<12254><dmsIcV4IPL0N5t7u>: Logged out in=8213 out=626 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
mailserver  | Oct 23 16:59:37 mx dovecot: imap-login: Login: user=<postmaster@mailserver.com>, method=PLAIN, rip=1.2.3.4, lip=172.27.0.2, mpid=12263, TLS, session=<gsEScV4ISr0N5t7u>
mailserver  | Oct 23 16:59:38 mx postfix/smtp[12223]: Anonymous TLS connection established to reception.mail-tester.com[94.23.206.89]:25: TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)
mailserver  | Oct 23 16:59:39 mx postfix/smtp[12223]: 977791825AB: to=<test-hckvzdc1l@srv1.mail-tester.com>, relay=reception.mail-tester.com[94.23.206.89]:25, delay=4.4, delays=0.33/0.02/3.4/0.65, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 706E4A0BEE)
mailserver  | Oct 23 16:59:39 mx postfix/qmgr[955]: 977791825AB: removed
mailserver  | Oct 23 16:59:43 mx postfix/smtpd[10600]: warning: milter inet:localhost:11332: can't read SMFIC_OPTNEG reply packet header: Connection timed out
mailserver  | Oct 23 16:59:43 mx postfix/smtpd[10600]: warning: milter inet:localhost:11332: read error in initial handshake

Improvements to this form?

No response

The text was updated successfully, but these errors were encountered:

georglauterbach · 2023-10-23T19:17:19Z

The issue is here:

postfix/smtpd[10600]: warning: milter inet:localhost:11332: can't read SMFIC_OPTNEG reply packet header: Connection timed out
postfix/smtpd[10600]: warning: milter inet:localhost:11332: read error in initial handshake

So after Googling this for a while, I will take a big guess and say that TCP on port 53 may be blocked? Please double-check; I know, DNS over TCP is cursed, but this is the mail world, not the normal world... Make sure port 53 is completely open for outbound (and of course established) traffic.

ref: https://forum.howtoforge.com/threads/dkim-filter-problem-cant-read-smfic_optneg-reply-packet-header.32681/

georglauterbach · 2023-10-23T19:18:02Z

If this turns out to be a DNS issue, I will update the docs.

ceeim · 2023-10-23T19:28:31Z

The issue is here:
postfix/smtpd[10600]: warning: milter inet:localhost:11332: can't read SMFIC_OPTNEG reply packet header: Connection timed out
postfix/smtpd[10600]: warning: milter inet:localhost:11332: read error in initial handshake
So after Googling this for a while, I will take a big guess and say that TCP on port 53 may be blocked? Please double-check; I know, DNS over TCP is cursed, but this is the mail world, not the normal world... Make sure port 53 is completely open for outbound (and of course established) traffic.

ref: https://forum.howtoforge.com/threads/dkim-filter-problem-cant-read-smfic_optneg-reply-packet-header.32681/

I use Oracle Cloud, my firewall has all ports open, not sure if Oracle Cloud is blocking port 53, but I had no issues when using OpenDKIM before.

georglauterbach · 2023-10-23T19:36:19Z

When you exec into your container (docker exec -ti <CONTAINER NAME> bash) and execute dig google.com +tcp, what is the result?

ceeim · 2023-10-23T19:41:50Z

When you exec into your container (docker exec -ti <CONTAINER NAME> bash) and execute dig google.com +tcp, what is the result?

; <<>> DiG 9.16.44-Debian <<>> google.com +tcp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38704
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1372
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             154     IN      A       142.250.199.110

;; Query time: 4 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Tue Oct 24 03:41:14 HKT 2023
;; MSG SIZE  rcvd: 55

georglauterbach · 2023-10-23T20:11:22Z

I see, this is then not the problem. Can you please disable IPv6 shortly and see whether it works without IPv6?

georglauterbach · 2023-10-23T20:12:07Z

Ah, and please also try removing the explicit DNS servers set in compose.yml and report what happens :)

reneploetz · 2023-10-23T21:20:29Z

Just a general thought: SMFIC_OPTNEG is usually the first command being sent by the MTA (here: postfix) to the Milter (rspamd). This command does not include any actual mail data, so it should basically return immediately with a list of actions that the Milter can perform [1]. As the command reply is taking too long I would suggest checking the logs of rspamd if the process is indeed running correctly. Also increasing the logging level for rspamd via overriding /etc/rspamd/local.d/logging.inc might also be a good idea - see https://rspamd.com/doc/configuration/logging.html

[1] https://github.com/emersion/go-milter/blob/master/milter-protocol.txt#L100 / https://github.com/avar/sendmail-pmilter/blob/master/doc/milter-protocol.txt#L100 (hard to find the official document ...)

polarathene · 2023-10-23T23:27:53Z

Below is unlikely going to help, but I thought I'd mention it.

If a process is running yet appears stalled/unresponsive, then you'll want to set ulimits: #3374 (comment)

This is known to affect PostSRSd, previously Fail2Ban (fixed), Rsyslogd (only recently fixed upstream, so probably not fixed in DMS), but could affect other processes if unlucky.

This would only be relevant though if the following outputs a number around 1 billion:

docker run --rm -it alpine ash -c 'ulimit -Sn && ulimit -Hn'

ceeim · 2023-10-24T05:59:32Z

I see, this is then not the problem. Can you please disable IPv6 shortly and see whether it works without IPv6?

It seems that the problem has been identified. I tried disabling the DNS settings in compose.yml, but it still couldn't perform DKIM signing. However, after disabling IPv6, I noticed that Rspamd DKIM is working fine. Now I am trying to fix the IPv6 connection for DMS.

ceeim · 2023-10-24T06:41:16Z

I configured IPv6 based on this document, but it seems that there are new issues. Is there a more comprehensive IPv6 configuration document available?

polarathene · 2023-10-24T06:56:00Z

Is there a more comprehensive IPv6 configuration document available?

Yes that has been revised for upcoming v13 release of the docs: https://docker-mailserver.github.io/docker-mailserver/edge/config/advanced/ipv6/#enable-proper-ipv6-support

Let me know how you find that rewrite :)

ceeim · 2023-10-24T09:20:01Z

Is there a more comprehensive IPv6 configuration document available?

Yes that has been revised for upcoming v13 release of the docs: https://docker-mailserver.github.io/docker-mailserver/edge/config/advanced/ipv6/#enable-proper-ipv6-support

Let me know how you find that rewrite :)

I followed the documentation of edge to configure Docker's IPv6 settings, but when I ping -6 google.com from the container, it shows "No route". I think I might have done something wrong in the steps because I have never tried Docker IPv6 before.

root@mx:/# ping -6 google.com
PING google.com(nrt20s21-in-x0e.1e100.net (2404:6800:4004:81f::200e)) 56 data bytes
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=1 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=2 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=3 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=4 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=5 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=6 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=7 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=8 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=9 Destination unreachable: No route
From fd00:cafe:face:feed::1 (fd00:cafe:face:feed::1) icmp_seq=10 Destination unreachable: No route

polarathene · 2023-10-24T11:12:45Z

Are you using the latest Docker? It might not be until v25 is released, IPv6 had an issue with the ping command and was resolved around mid 2023 IIRC.

I'm not sure if configuring your /etc/docker/daemon.json to have:

{
  "userland-proxy": false
}

will help with the ping issue. It was related to an ip6tables rule being too restrictive.

EDIT: Here's the IPv6 ping issue, seems related to container to container though. It was backported to Docker 24.0.6 in August.

Do you have a firewall that might also be blocking the traffic?

ceeim · 2023-10-24T14:06:41Z

Are you using the latest Docker? It might not be until v25 is released, IPv6 had an issue with the ping command and was resolved around mid 2023 IIRC.

I'm not sure if configuring your /etc/docker/daemon.json to have:
{
  "userland-proxy": false
}
will help with the ping issue. It was related to an ip6tables rule being too restrictive.

EDIT: Here's the IPv6 ping issue, seems related to container to container though. It was backported to Docker 24.0.6 in August.

Do you have a firewall that might also be blocking the traffic?

my compose.yaml:

services:
  mailserver:
    networks:
      - dms-ipv6

networks:
  dms-ipv6:
    enable_ipv6: true
    ipam:
      config:
        - subnet: fd00:cafe:face:feed::/64

/etc/docker/daemon.json

{
  "ip6tables": true,
  "experimental" : true,
  "userland-proxy": false
}

docker version

Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:36 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

This will result in the unavailability of IPv6 for my host machine and containers. At which step did I make a mistake?
For testing purposes, my host machine has not set up any firewall to eliminate the possibility of a firewall causing issues.

reneploetz · 2023-10-24T20:31:10Z

Some guesses ...

Generally speaking, you likely want to use the userland-proxy - which is also on per default. Your configuration - apart from the userland-proxy setting looks correct to me at first glance. You can also check with docker network inspect dms-ipv6 if the settings were applied.

Which firewall frontend are you using? Plain iptables or nft or something else?
Depending on your system you need to check if you have ip6tables or ip6tables-nft installed at all / correctly. Rules for Netfilter (aka: the Linux Kernel Network Filter system) will be created by Docker regardless of whether you have a specific firewall frontend running because they are needed for the network routing from/to your containers.
You can check the current rules via iptables -L or nft list ruleset or (insert your firewall/netfilter frontend). Check if the policies are indeed "accept" per default (you want to change that later though).

You might also want to check if you have net.ipv6.conf.all.accept_ra=2 as sysctl if you rely on router advertisements to autoconfigure your IPv6 address. This is because turning on IPv6 in Docker will turn on IPV6 forwarding which will then turn off listening for router advertisements. That can also explain the connectivity loss when enabling IPv6.
Also check for net.ipv6.conf.default.forwarding=1.

polarathene · 2023-10-24T20:56:42Z

Generally speaking, you likely want to use the userland-proxy - which is also on per default.

It's mostly for localhost connectivity. There's a few quirks depending on if it's enabled or not, but the intent is to eventually default to disabled and then drop it. I spent a good chunk of time contributing to the upstream tracking issue on that 😅

Plain iptables or nft or something else?

Regardless, Docker only supports managing iptables rules, it doesn't natively support managing rules for nftables.

When I asked about firewall, I meant the frontends ufw / firewalld as these have quirks.

The remaining advice is good IIRC, on an IPv6 host I recall having to manage NDP proxy for using GUA with containers, but ULA like the DMS docs advise was fine. If you need actual IPv6 reachability you'd publish the ports to the public IPv6 interface with it's GUA address. This works like the containers IPv4 bridged network from private address to public, which is fine.

@ceeim ping issue aside, were you able to verify IPv6 is working correctly?:

https://docker-mailserver.github.io/docker-mailserver/edge/config/advanced/ipv6/#verify-remote-ip-is-correct

reneploetz · 2023-10-24T22:36:43Z

Generally speaking, you likely want to use the userland-proxy - which is also on per default.

It's mostly for localhost connectivity. There's a few quirks depending on if it's enabled or not, but the intent is to eventually default to disabled and then drop it. I spent a good chunk of time contributing to the upstream tracking issue on that 😅

I agree, it would be good to get rid of it, but there are still some issues that are unresolved. In the end it's probably a matter of whether you would have to work around some issues if you would be affected.

Plain iptables or nft or something else?

Regardless, Docker only supports managing iptables rules, it doesn't natively support managing rules for nftables.

I did reference iptables6-nft specifically to make clear that there are Linux distributions (RHEL etc) that use the alternatives system to switch from the old iptables to the nftables iptables-translation. Yes, Docker can only use an iptables-compatible command, but iptables6-nft provides this and can be used with Docker.

ceeim · 2023-10-28T11:45:01Z

Forget about it, at least I have IPv4 available now, I gave up on configuring IPv6. :(

georglauterbach · 2023-10-28T13:10:26Z

You did well, believe me. In my not-so-very-humble opinion, IPv6 is useless at best, or incurring problems at worst 😆

ceeim added kind/bug/report A report about a bug meta/needs triage This issue / PR needs checks and verification from maintainers labels Oct 23, 2023

georglauterbach added service/security/rspamd area/networking labels Oct 23, 2023

georglauterbach self-assigned this Oct 23, 2023

ceeim closed this as completed Oct 28, 2023

tchapi mentioned this issue Nov 5, 2023

[Possible bug?] Using Rspamd, outbound mails are not DKIM-signed #3621

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rspamd DKIM unsigned #3598

Rspamd DKIM unsigned #3598

ceeim commented Oct 23, 2023 •

edited

georglauterbach commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

ceeim commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

ceeim commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

reneploetz commented Oct 23, 2023 •

edited

polarathene commented Oct 23, 2023

ceeim commented Oct 24, 2023

ceeim commented Oct 24, 2023

polarathene commented Oct 24, 2023

ceeim commented Oct 24, 2023

polarathene commented Oct 24, 2023

ceeim commented Oct 24, 2023 •

edited

reneploetz commented Oct 24, 2023 •

edited

polarathene commented Oct 24, 2023 •

edited

reneploetz commented Oct 24, 2023

ceeim commented Oct 28, 2023

georglauterbach commented Oct 28, 2023

Rspamd DKIM unsigned #3598

Rspamd DKIM unsigned #3598

Comments

ceeim commented Oct 23, 2023 • edited

📝 Preliminary Checks

👀 What Happened?

👟 Reproduction Steps

🐋 DMS Version

💻 Operating System and Architecture

⚙️ Container configuration files

📜 Relevant log output

Improvements to this form?

georglauterbach commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

ceeim commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

ceeim commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

georglauterbach commented Oct 23, 2023

reneploetz commented Oct 23, 2023 • edited

polarathene commented Oct 23, 2023

ceeim commented Oct 24, 2023

ceeim commented Oct 24, 2023

polarathene commented Oct 24, 2023

ceeim commented Oct 24, 2023

polarathene commented Oct 24, 2023

ceeim commented Oct 24, 2023 • edited

reneploetz commented Oct 24, 2023 • edited

polarathene commented Oct 24, 2023 • edited

reneploetz commented Oct 24, 2023

ceeim commented Oct 28, 2023

georglauterbach commented Oct 28, 2023

ceeim commented Oct 23, 2023 •

edited

reneploetz commented Oct 23, 2023 •

edited

ceeim commented Oct 24, 2023 •

edited

reneploetz commented Oct 24, 2023 •

edited

polarathene commented Oct 24, 2023 •

edited