Mailer seems to be working but no email are being sent

[alexd124641]

Issue Summary
I've installed alerta (which work fine) and alerta-mailer. When I do alerta-mailer (or execute mailer.py), everything seems to be working (connection to the database and heartbeat) I can see the heartbeat in the web UI but I don't receive any email.

Environment

• OS: Linux (Debian 10)

• API version: 8.0.3

• Deployment: self-hosted

• For self-hosted, WSGI environment: nginx/uwsgi

• Database: MongoDB

• Server config:
  Auth enabled? Yes
  Auth provider? Basic
  Customer views? No

• web UI version: [eg. 6.8.1]

• CLI version: [eg. 6.8.1]

To Reproduce
Steps to reproduce the behavior:

1. Execute alerta-mailer
2. Generate new alerts after the POST command (which results in 201)
3. No email are being sent (nothing in the logs)

Expected behavior
At least having a trace somewhere of the email in order to identify what is going wrong

Alerta config

Alerta.conf:

[DEFAULT]
endpoint = http://localhost:8080/api
key = 6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO

[alerta-mailer]
key = 6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO
mail_to = mailhog@mailhog.example
mail_from = alerta@mailhog.example
amqp_url = mongodb://localhost:27017/kombu
endpoint = http://localhost:8080/api
dashboard_url = http://localhost:8000
smtp_host = localhost
smtp_port = 1025
smtp_use_ssl = False
debug = True
skip_mta = False
email_type = text
amqp_topic = notify
config_file = /etc/alerta/alerta.conf

alertad.conf:

#!/usr/bin/bash
PLUGINS = ['remote_ip','reject','heartbeat','blackout', 'prometheus', 'alerta-mailer', 'amqp']
ALERTA_TIMEOUT = 14400
ALERT_TIMEOUT = 14400
SECRET_KEY = '6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO'
BASE_URL = '/api'
DEBUG = True
LOG_HANDLERS = ['console', 'file']
LOG_FILE = '/var/log/alerta/alertad.log'
LOG_MAX_BYTES = 5*1024*1024  # 5 MB
LOG_BACKUP_COUNT = 2
LOG_FORMAT = 'verbose'


DATABASE_URL = 'mongodb://127.0.0.1:27017/?replicaSet=test&connectTimeoutMS=300000'
DATABASE_NAME = 'monitoring'
DATABASE_RAISE_ON_ERROR = False

SEVERITY_MAP = {
    'security': 0,
    'critical': 1,
    'major': 2,
    'minor': 3,
    'warning': 4,
    'indeterminate': 5,
    'normal': 9,
    'ok': 9,
    'unknown': 9,
    'informational': 9,
}

DEFAULT_NORMAL_SEVERITY = 'ok'  # 'normal', 'ok', 'cleared'
DEFAULT_PREVIOUS_SEVERITY = 'indeterminate'

COLOR_MAP = {
    'severity': {
        'security': '#B3B3B3',
        'critical': 'red',
        'major': 'orange',
        'minor': '#9B870C',
        'warning': '#1E90FF',
        'indeterminate': 'lightblue',
        'normal': '#00CC00',
        'ok': '#00CC00',
        'unknown': 'silver',
        'informational': 'grey'
    },
    'text': 'black',
    'highlight': 'skyblue '
}

EMAIL_VERIFICATION = False
MAIL_FROM = 'alerta@mailhog.example'
SMTP_HOST = 'localhost'
SMTP_PORT = 1025
SMTP_STARTTLS = False

Mailer.py :

DEFAULT_OPTIONS = {
    'config_file':   '/etc/alerta.conf',
    'profile':       'alertamailer',
    'endpoint':      'http://127.0.0.1:8080/api',
    'key':           '6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO',
    'amqp_url':      'mongodb://127.0.0.1:27017/kombu',
    'amqp_topic':    'notify',
    'amqp_queue_name':    '', # Name of the AMQP queue. Default is no name (default queue destination).
    'amqp_queue_exclusive': True, # Exclusive queues may only be consumed by the current connection.
    'smtp_host':     '127.0.0.1',
    'smtp_port':     1025,
    'smtp_username': '', # application-specific username if it differs from the specified 'mail_from' user
    'smtp_password': '',  # application-specific password if gmail used
    'smtp_starttls': True,  # use the STARTTLS SMTP extension
    'smtp_use_ssl': False,  # whether or not SSL is being used for the SMTP connection
    'ssl_key_file': None, # a PEM formatted private key file for the SSL connection
    'ssl_cert_file': None, # a certificate chain file for the SSL connection
    'mail_from':     '',  # alerta@example.com
    'mail_to':       [],  # devops@example.com, support@example.com
    'mail_localhost': 'mailhog.example',  # fqdn to use in the HELO/EHLO command
    'mail_template':  os.path.dirname(__file__) + os.sep + 'email.tmpl',
    'mail_template_html': os.path.dirname(__file__) + os.sep + 'email.html.tmpl',  # nopep8
    'mail_subject':  ('[{{ alert.status|capitalize }}] {{ alert.environment }}: '
                      '{{ alert.severity|capitalize }} {{ alert.event }} on '
                      '{{ alert.service|join(\',\') }} {{ alert.resource }}'),
    'dashboard_url': 'http://127.0.0.1:3000',
    'debug':         True,
    'skip_mta':      False,
    'email_type':    'html'  # options are: text, html
}

Output of either ./mailer.py or alerta-mailer:

root@112501L01SUP:/etc/alerta/mailer# ./mailer.py
DEBUG:__main__:Looking for rules files in /etc/alerta/alerta.rules.d
INFO:kombu.mixins:Connected to mongodb://localhost:27017/kombu
Connected to mongodb://localhost:27017/kombu
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8080
Starting new HTTP connection (1): localhost:8080
DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 454
http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 454

If you need any additional info, file or command output let me know.

[decentral1se]

Hey, having the same issue. I see that 'alerta-mailer' shouldn't be in PLUGINS in your alertad.conf because it is not a plugin. That's one thing. Otherwise, I do not know, I have the exact same issue and I am not getting any logging at all. I suspect the logic is not even running because there are plenty of debug statements in the code...

[decentral1se]

It's been suggested that maybe mongodb is the source of the issue....I will try Redis soon.

Also, no "..." double quote lines in your alerta.conf was an issue I had to learn the hard way...

[decentral1se]

I will try Redis soon.

Ok, it's not simple and I am not sure how to do it since it now complains about a missing redis back-end and I see there is no documentation to show it is supported? Docs state that mongodb is supported but it is somehow problematic here for reasons unknown? I am sure that everything is working on my end now in terms of configuration and settings (see https://git.autonomic.zone/coop-cloud/alerta, docker swarm setup) and I am wondering how to go about investigating if Mongodb really is the issue?

[decentral1se]

OK fuck it, I couldn't resist more pain, using redis worked...

[satterly]

Kombu is the AMQP python package used to implement the mailer queue and it supports a number of different "transports" (aka. backends). See https://docs.celeryproject.org/projects/kombu/en/stable/introduction.html#transport-comparison

[decentral1se]

Well the MongoDB transport is broken for the mailer queue as far as I can see.

[jrevillard]

Dear @decentral1se, @satterly

I tried also but cannot make it working even with redis. I see that messages are published (amqp plugin logs)... The mailer is connected also... but no mail sent...

I don't know how to debug it....

Best.

[mhk001]

I tried Mongo transport for mailer, 2 weeks ago and is working.
OS ubuntu.bionic64
Docker environment, mongodb localy installed in web container
DB postgresql

ENV variable
ALERTA_CONF_FILE=/app/alerta.conf
ALERTA_SVR_CONF_FILE=/app/alertad.conf

part of the alertad.conf:
there is TOPIC notify with '
....
PLUGINS = ['debug','reject','blackout','enhance','normalise','acked_by','amqp']
AMQP_URL = 'mongodb://localhost:27017/kombu'
AMQP_TOPIC = 'notify'

part of the alerta.conf
there is amqp_topic notify without '
...
[alerta-mailer]
key = some_string ; must have "write:heartbeats" scope
mail_to =
mail_from = from@domain.com
amqp_url = mongodb://localhost:27017/kombu
amqp_topic = notify
dashboard_url = http://localhost:48080
smtp_host =
smtp_port =
smtp_username = smtp_login_name
smtp_password = smtp_login_pass
smtp_starttls = True
smtp_use_ssl = False
debug = True
skip_mta = False
email_type = html

alerta-mailer rules dir
/app/alerta.rules.d

simple rule file: t1_rules.conf

[
{
"name": "T001",
"fields": [
{"field": "resource", "regex": "Mailer_D11" }
],
"contacts": ["user@example.net"]
}
]

Start alerta-mailer in debug mode
#alerta-mailer

then send event from different shell
#alerta send -r Mailer_D11 -E Test -e MailerErr_33 -S postfix4 -s critical

and alert mailer try deliver message after 30 sec and writes debug log on console

--- debug log ---
DEBUG:mailer:Looking for rules files in /app/alerta.rules.d
DEBUG:mailer:Parsing t1_rules.conf
INFO:mailer:Adding rule {'name': 'T001', 'fields': [{'field': 'resource', 'regex': 'Mailer_D11'}], 'contacts': ['user@example.net']} to list of rules to be evaluated
INFO:kombu.mixins:Connected to mongodb://localhost:27017/kombu
Connected to mongodb://localhost:27017/kombu
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8080
Starting new HTTP connection (1): localhost:8080
DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 297
http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 297
DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 298
http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 298
DEBUG:mailer:Initial contact list: []
Initial contact list: []
DEBUG:mailer:Checking 1 group rules
Checking 1 group rules
INFO:mailer:Evaluating rule T001
Evaluating rule T001
DEBUG:mailer:Evaluating rule field {'field': 'resource', 'regex': 'Mailer_D11'}
Evaluating rule field {'field': 'resource', 'regex': 'Mailer_D11'}
DEBUG:mailer:Trying to match Mailer_D11 to Mailer_D11
Trying to match Mailer_D11 to Mailer_D11
DEBUG:mailer:Extending contact to include ['user@example.net']
Extending contact to include ['user@example.net']
send: 'ehlo [10.8.0.2]\r\n'
reply: b'250-XXXXXXXXX\r\n'

.......... SMTP conversation ....................

reply: retcode (235); Msg: b'2.7.0 Authentication successful'
send: 'mail FROM: size=3099\r\n'
reply: b'250 2.1.0 Ok\r\n'
reply: retcode (250); Msg: b'2.1.0 Ok'
send: 'rcpt TO:user@example.net\r\n'
reply: b'556 5.1.10 user@example.net: Recipient address rejected: Domain example.net does not accept mail (nullMX)\r\n'
reply: retcode (556); Msg: b'5.1.10 user@example.net: Recipient address rejected: Domain example.net does not accept mail (nullMX)'
send: 'rset\r\n'
reply: b'250 2.0.0 Ok\r\n'
reply: retcode (250); Msg: b'2.0.0 Ok'
---------------------

Read this https://github.com/alerta/alerta-contrib/tree/master/integrations/mailer ,
Alerts have to meet the some criterias

BR.

[jrevillard]

Hello,

I finally made it working also... the main problem is that I think that you are missing debug messages to know when a message is acknowledged for different reasons.

Best,
Jerome

[satterly]

@jrevillard Thanks!