Mailu deploy openly relaying non-authenticated email

[fiskhest]

After successfully running mailu for a couple of weeks a botnet detected that my stack was openly relaying any email thrown at it. The server is behind a router (asus AC68u with asuswrt-merlin firmware), thus traffic coming from the internet is NATed to the internal server. I've run a custom setup postfix+dovecot instance previously under this network configuration, so therefore I think that the network translations should be going correctly, but I haven't gotten the time(/figured out) how to verify that yet. (Would it be wise to tcpdump this?)

When I test against https://www.appmaildev.com/ everything passes except for RBL. (Sorry, some bug with my screenshot taking application so I can't attach an image)

Running swaks yields authentication required:

swaks -server domain.tld:587 -4 -tls --to user@gmail.com --from fiskhest@domain.tld
=== Trying domain.tld:587...
=== Connected to domain.tld.
<-  220 domain.tld ESMTP ready
 -> EHLO fisk.hest
<-  250-domain.tld
<-  250 STARTTLS
 -> STARTTLS
<-  220 2.0.0 Start TLS
=== TLS started with cipher TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
=== TLS no local certificate set
=== TLS peer DN="/CN=domain.tld"
 ~> EHLO fisk.hest
<~  250-domain.tld
<~  250 AUTH PLAIN
 ~> MAIL FROM:<fiskhest@domain.tld>
<~* 530 5.7.1 Authentication required
 ~> QUIT
<~  221 2.0.0 Bye
=== Connection closed with remote host.

Testing against https://mxtoolbox.com/diagnostic.aspx:

220 domain.tld ESMTP ready [708 ms]
EHLO keeper-us-east-1c.mxtoolbox.com
250-domain.tld
250 STARTTLS [755 ms]
MAIL FROM:<supertool@mxtoolbox.com>
250 2.0.0 OK [725 ms]
RCPT TO:<test@mxtoolboxsmtpdiag.com>
250 2.1.5 Ok [848 ms]

My configuration files,

# docker-compose.yml
# This file is auto-generated by the Mailu configuration wizard.
# Please read the documentation before attempting any change.
# Generated for compose flavor

version: '3.5'

services:

  # External dependencies
  redis:
    image: redis:alpine
    restart: always
    volumes:
      - "/mailu/redis:/data"
    
  # Core services
  front:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    logging:
      driver: json-file
    ports:
      - "192.168.2.x:8080:80"
      - "192.168.2.x:8443:443"
      - "192.168.2.x:25:25"
      - "192.168.2.x:465:465"
      - "192.168.2.x:587:587"
      - "192.168.2.x:110:110"
      - "192.168.2.x:995:995"
      - "192.168.2.x:143:143"
      - "192.168.2.x:993:993"
    volumes:
    #  - "/mailu/certs:/certs"
    #  - "/etc/letsencrypt:/certs/letsencrypt"
      - "/mailu/overrides/nginx:/overrides"
      - type: bind
        source: /etc/letsencrypt/live/domain.tld/cert.pem
        target: /certs/cert.pem
        read_only: true
      - type: bind
        source: /etc/letsencrypt/live/domain.tld/privkey.pem
        target: /certs/key.pem
        read_only: true


  resolver:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}unbound:${MAILU_VERSION:-1.6}
    env_file: mailu.env
    restart: always
    networks:
      default:
        ipv4_address: 192.168.203.254

  admin:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/data:/data"
      - "/mailu/dkim:/dkim"
    depends_on:
      - redis

  imap:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/mail:/mail"
      - "/mailu/overrides:/overrides"
    depends_on:
      - front

  smtp:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/overrides:/overrides"
    depends_on:
      - front
      - resolver
    dns:
      - 192.168.203.254

  antispam:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/filter:/var/lib/rspamd"
      - "/mailu/dkim:/dkim"
      - "/mailu/overrides/rspamd:/etc/rspamd/override.d"
    depends_on:
      - front
      - resolver
    dns:
      - 192.168.203.254

  # Optional services
  antivirus:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}clamav:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/filter:/data"
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  webdav:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}radicale:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/dav:/data"

  fetchmail:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}fetchmail:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  # Webmail
  webmail:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}roundcube:${MAILU_VERSION:-1.6}
    restart: always
    env_file: mailu.env
    volumes:
      - "/mailu/webmail:/data"
    depends_on:
      - imap


networks:
  default:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 192.168.203.0/24

# mailu.env
# Mailu main configuration file
#
# Generated for compose flavor
#
# This file is autogenerated by the configuration management wizard.
# For a detailed list of configuration variables, see the documentation at
# https://mailu.io

###################################
# Common configuration variables
###################################

# Set this to the path where Mailu data and configuration is stored
# This variable is now set directly in `docker-compose.yml by the setup utility
# ROOT=/mailu

# Mailu version to run (1.0, 1.1, etc. or master)
#VERSION=1.6

# Set to a randomly generated 16 bytes string
SECRET_KEY=<redacted>

# Address where listening ports should bind
# This variables are now set directly in `docker-compose.yml by the setup utility
# PUBLIC_IPV4= 192.168.2.246 (default: 127.0.0.1)
# PUBLIC_IPV6= fe80::9da:5306:2520:edb9 (default: ::1)

# Subnet of the docker network. This should not conflict with any networks to which your system is connected. (Internal and external!)
SUBNET=192.168.203.0/24

# Main mail domain
DOMAIN=domain.tld

# Hostnames for this server, separated with comas
HOSTNAMES=domainx.tld,domainy.tld

# Postmaster local part (will append the main mail domain)
POSTMASTER=admin

# Choose how secure connections will behave (value: letsencrypt, cert, notls, mail, mail-letsencrypt)
TLS_FLAVOR=mail

# Authentication rate limit (per source IP address)
AUTH_RATELIMIT=10/minute;1000/hour 

# Opt-out of statistics, replace with "True" to opt out
DISABLE_STATISTICS=False

###################################
# Optional features
###################################

# Expose the admin interface (value: true, false)
ADMIN=true

# Choose which webmail to run if any (values: roundcube, rainloop, none)
WEBMAIL=roundcube

# Dav server implementation (value: radicale, none)
WEBDAV=radicale

# Antivirus solution (value: clamav, none)
#ANTIVIRUS=clamav

#Antispam solution
ANTISPAM=none

###################################
# Mail settings
###################################

# Message size limit in bytes
# Default: accept messages up to 50MB
# Max attachment size will be 33% smaller
MESSAGE_SIZE_LIMIT=133000000

# Networks granted relay permissions
# Use this with care, all hosts in this networks will be able to send mail without authentication!
RELAYNETS=

# Will relay all outgoing mails if configured
RELAYHOST=

# Fetchmail delay
FETCHMAIL_DELAY=600

# Recipient delimiter, character used to delimiter localpart from custom address part
RECIPIENT_DELIMITER=+

# DMARC rua and ruf email
DMARC_RUA=admin
DMARC_RUF=admin

# Welcome email, enable and set a topic and body if you wish to send welcome
# emails to all users.
WELCOME=false
WELCOME_SUBJECT=Welcome to your new email account
WELCOME_BODY=Welcome to your new email account, if you can read this, then it is configured properly!

# Maildir Compression
# choose compression-method, default: none (value: bz2, gz)
COMPRESSION=
# change compression-level, default: 6 (value: 1-9)
COMPRESSION_LEVEL=

###################################
# Web settings
###################################

# Path to redirect / to
WEBROOT_REDIRECT=/webmail

# Path to the admin interface if enabled
WEB_ADMIN=/admin

# Path to the webmail if enabled
WEB_WEBMAIL=/

# Website name
SITENAME=epost

# Linked Website URL
WEBSITE=https://mail.domain.tld



###################################
# Advanced settings
###################################

# Log driver for front service. Possible values:
# json-file (default)
# journald (On systemd platforms, useful for Fail2Ban integration)
# syslog (Non systemd platforms, Fail2Ban integration. Disables `docker-compose log` for front!)
# LOG_DRIVER=json-file

# Docker-compose project name, this will prepended to containers names.
COMPOSE_PROJECT_NAME=mailu

# Default password scheme used for newly created accounts and changed passwords
# (value: BLF-CRYPT, SHA512-CRYPT, SHA256-CRYPT, MD5-CRYPT, CRYPT)
PASSWORD_SCHEME=BLF-CRYPT

# Header to take the real ip from
REAL_IP_HEADER=

# IPs for nginx set_real_ip_from (CIDR list separated by commas)
REAL_IP_FROM=

# choose wether mailu bounces (no) or rejects (yes) mail when recipient is unknown (value: yes, no)
REJECT_UNLISTED_RECIPIENT=

# Log level threshold in start.py (value: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET)
LOG_LEVEL=WARNING

###################################
# Database settings
###################################
DB_FLAVOR=mysql
DB_USER=username
DB_PW=password
DB_HOST=192.168.203.1
DB_NAME=mailu

There are some tweaks I've had to do as I'm running an apache front reverse proxying on the host as I have some other legacy stuff I had to quickly migrate over, it should only be load balancing for the roundcube instance and all other changes shouldn't affect the stack in such a way that it should be open to relaying external networks but I figured it's worth mentioning in case there's something I'm misunderstanding wrt this.

I've got no problem deep diving into containers and network troubleshooting but as I'm a huge jack of all trades I've run out of ideas where to look. Tips are greatly appreciated, thanks.

[fiskhest]

So it seems like it's back!

One possible issue mentioned on the chat yesterday was NAT masquerading. Here's a snippet from the source of a relayed email;

Received: from kindarium.ru (unknown [192.168.203.1])
by domain.tld (Postfix) with SMTP id 58F5E90F3A61;

indicating that postfix on the smtp instance is receiving the emails straight from the docker gateway and not passing through front.

Docker instance IP's:
gateway: 192.168.203.1
front: 192.168.203.3
smtp: 192.168.203.8

Snippet of tcpdumping the veth interface for front on Docker host, which to me indicates that the all three instances are participating in the delivery flow:

09:53:43.132929 02:42:52:60:8b:15 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 103: 192.168.203.1.46997 > 192.168.203.3.smtp: Flags [P.], seq 100:137, ack 43, win 115, options [nop,nop,TS val 153314414 ecr 134256827], length 37: SMTP: RCPT TO:<myasnikova.n@kanctanta.ru>
09:53:43.132969 02:42:c0:a8:cb:03 > 02:42:52:60:8b:15, ethertype IPv4 (0x0800), length 66: 192.168.203.3.smtp > 192.168.203.1.46997: Flags [.], ack 137, win 227, options [nop,nop,TS val 134257360 ecr 153314414], length 0
09:53:43.133162 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 103: 192.168.203.3.38400 > 192.168.203.8.smtp: Flags [P.], seq 100:137, ack 43, win 237, options [nop,nop,TS val 134257360 ecr 134256826], length 37: SMTP: RCPT TO:<myasnikova.n@kanctanta.ru>
09:53:43.133231 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 66: 192.168.203.8.smtp > 192.168.203.3.38400: Flags [.], ack 137, win 227, options [nop,nop,TS val 134257360 ecr 134257360], length 0
09:53:43.135051 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 74: 192.168.203.3.60144 > 192.168.203.8.smtp: Flags [S], seq 3832196620, win 29200, options [mss 1460,sackOK,TS val 134257362 ecr 0,nop,wscale 7], length 0
09:53:43.135158 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 74: 192.168.203.8.smtp > 192.168.203.3.60144: Flags [S.], seq 3372114128, ack 3832196621, win 28960, options [mss 1460,sackOK,TS val 134257362 ecr 134257362,nop,wscale 7], length 0
09:53:43.135180 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 66: 192.168.203.3.60144 > 192.168.203.8.smtp: Flags [.], ack 1, win 229, options [nop,nop,TS val 134257362 ecr 134257362], length 0
09:53:43.137597 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 96: 192.168.203.8.smtp > 192.168.203.3.60144: Flags [P.], seq 1:31, ack 1, win 227, options [nop,nop,TS val 134257364 ecr 134257362], length 30: SMTP: 220 radivoj.se ESMTP Postfix
09:53:43.137629 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 66: 192.168.203.3.60144 > 192.168.203.8.smtp: Flags [.], ack 31, win 229, options [nop,nop,TS val 134257364 ecr 134257364], length 0
09:53:43.137689 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 83: 192.168.203.3.60144 > 192.168.203.8.smtp: Flags [P.], seq 1:18, ack 31, win 229, options [nop,nop,TS val 134257364 ecr 134257364], length 17: SMTP: EHLO radivoj.se
09:53:43.137720 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 66: 192.168.203.8.smtp > 192.168.203.3.60144: Flags [.], ack 18, win 227, options [nop,nop,TS val 134257364 ecr 134257364], length 0
09:53:43.137954 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 262: 192.168.203.8.smtp > 192.168.203.3.60144: Flags [P.], seq 31:227, ack 18, win 227, options [nop,nop,TS val 134257365 ecr 134257364], length 196: SMTP: 250-radivoj.se
09:53:43.137995 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 113: 192.168.203.3.60144 > 192.168.203.8.smtp: Flags [P.], seq 18:65, ack 227, win 237, options [nop,nop,TS val 134257365 ecr 134257365], length 47: SMTP: XCLIENT ADDR=192.168.203.1 NAME=[UNAVAILABLE]
09:53:43.139069 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 96: 192.168.203.8.smtp > 192.168.203.3.60144: Flags [P.], seq 227:257, ack 65, win 227, options [nop,nop,TS val 134257366 ecr 134257365], length 30: SMTP: 220 radivoj.se ESMTP Postfix
09:53:43.139117 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 92: 192.168.203.3.60144 > 192.168.203.8.smtp: Flags [P.], seq 65:91, ack 257, win 237, options [nop,nop,TS val 134257366 ecr 134257366], length 26: SMTP: HELO dostavka-toplivo.ru
09:53:43.139274 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 82: 192.168.203.8.smtp > 192.168.203.3.60144: Flags [P.], seq 257:273, ack 91, win 227, options [nop,nop,TS val 134257366 ecr 134257366], length 16: SMTP: 250 radivoj.se
09:53:43.139311 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 99: 192.168.203.3.60144 > 192.168.203.8.smtp: Flags [P.], seq 91:124, ack 273, win 237, options [nop,nop,TS val 134257366 ecr 134257366], length 33: SMTP: MAIL FROM:<zot2005@boarding.dk>
09:53:43.141727 02:42:52:60:8b:15 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 66: 192.168.203.1.55006 > 192.168.203.3.smtp: Flags [.], ack 43, win 115, options [nop,nop,TS val 430182708 ecr 134257179], length 0
09:53:43.145120 02:42:52:60:8b:15 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 66: 192.168.203.1.34651 > 192.168.203.3.smtp: Flags [.], ack 43, win 1825, options [nop,nop,TS val 65089885 ecr 134257107], length 0
09:53:43.146373 02:42:c0:a8:cb:08 > 02:42:c0:a8:cb:03, ethertype IPv4 (0x0800), length 80: 192.168.203.8.smtp > 192.168.203.3.38602: Flags [P.], seq 99:113, ack 243, win 1432, options [nop,nop,TS val 134257373 ecr 134257096], length 14: SMTP: 250 2.1.5 Ok
09:53:43.146405 02:42:c0:a8:cb:03 > 02:42:c0:a8:cb:08, ethertype IPv4 (0x0800), length 66: 192.168.203.3.38602 > 192.168.203.8.smtp: Flags [.], ack 113, win 237, options [nop,nop,TS val 134257373 ecr 134257373], length 0
09:53:43.146443 02:42:c0:a8:cb:03 > 02:42:52:60:8b:15, ethertype IPv4 (0x0800), length 80: 192.168.203.3.smtp > 192.168.203.1.37666: Flags [P.], seq 99:113, ack 243, win 1658, options [nop,nop,TS val 134257373 ecr 43463490], length 14: SMTP: 250 2.1.5 Ok

[fiskhest]

After a long time of pondering what could be wrong I finally found that firewalld(on Centos7 atleast) and Docker does not play nice with each other, as per moby/moby#16137

After disabling firewalld (systemctl stop firewalld && systemctl disable firewalld) everything magically started working and packets are routed from my NAT gateway to the docker host, through front (and rejected if not authenticated) to smtp-instance. Testing against mxtoolbox yields a closed relay.

Closing issue.