alerta-mailer: usage docs need improving

[decentral1se]

Issue Summary
When I run https://github.com/alerta/alerta-contrib/tree/master/integrations/mailer#deployment I see the following logs. Unfortunately, I have no idea at which point the authentication is failing: amqp, api, smtp? As for the 403 error, I have manually validated that my key works by running Client(...) in the Python REPL.

I have no files in alerta.rules.d.

alerta@440297312f61:/$ env | grep -i smtp
SMTP_PASSWORD=...my smtp password...
alerta@440297312f61:/$ alerta-mailer
DEBUG:mailer:Looking for rules files in /app/alerta.rules.d
Authentication failed.
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8080
Starting new HTTP connection (1): localhost:8080
DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 403 139
http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 403 139

Here's my alerta.conf:

[DEFAULT]
debug = True
endpoint = http://localhost:8080/api
key = {{ alerta_cli_api_key }}
output = json
sslverify = False
timezone = Europe/London

[alerta-mailer]
amqp_topic = notify
amqp_url = {{ db_type }}://{{ db_user }}:{{ db_passwd }}@{{ db_loc }}/kombu
config_file = /app/alerta.conf
dashboard_url = https://{{ domain }}
debug = True
email_type = text
endpoint = http://localhost:8080/api
key = {{ alerta_mailer_api_key }}
mail_from = {{ mail_from }}
mail_to = {{ mail_to }}
skip_mta = False
smtp_host = {{ smtp_host }}
smtp_password = {{ smtp_passwd }}
smtp_port = {{ smtp_port }}
smtp_starttls = True
smtp_username = {{ mail_from }}

Here's my alertad.conf:

SITE_LOGO_URL = "myurl"

DEBUG = True

SECRET = "{{ alerta_secret_key }}"

ALLOWED_ENVIRONMENTS = [
  "myenv"
]

BASE_URL = "/api"
USE_PROXYFIX = True

AUTH_REQUIRED = True
AUTH_PROVIDER = "keycloak"

ADMIN_USERS = ["someone"]
USER_DEFAULT_SCOPES = ["admin"]

SIGNUP_ENABLED = False

KEYCLOAK_URL = "{{ keycloak_url }}"
KEYCLOAK_REALM = "{{ keycloak_realm }}"
ALLOWED_KEYCLOAK_ROLES = ["{{ keycloak_role }}"]
OAUTH2_CLIENT_ID = "{{ oauth_client_id }}"
OAUTH2_CLIENT_SECRET = "{{ oauth_client_secret }}"

SEVERITY_MAP = {
    "fatal": 0,
    "critical": 1,
    "major": 2,
    "minor": 3,
    "warning": 4,
    "indeterminate": 5,
    "cleared": 5,
    "normal": 5,
    "ok": 5,
    "informational": 6,
    "debug": 7,
    "trace": 8,
    "unknown": 9
}

DEFAULT_NORMAL_SEVERITY = "normal"
DEFAULT_PREVIOUS_SEVERITY = "indeterminate"

PLUGINS = [
    "alerta-mailer",
    "amqp",
    "blackout",
    "normalise",
    "reject",
]

AMQP_URL = "{{ db_type }}://{{ db_user }}:{{ db_passwd }}@{{ db_loc }}/kombu"
AMQP_TOPIC = "notify"

DATABASE_URL = "{{ db_type }}://{{ db_user }}:{{ db_passwd }}@{{ db_loc }}/{{ db_name }}"
DATABASE_NAME = "{{ db_name }}"

EMAIL_VERIFICATION = False
MAIL_FROM = "{{ mail_from }}"
SMTP_HOST = "{{ smtp_host }}"
SMTP_PASSWORD = "{{ smtp_passwd }}"
SMTP_PORT = "{{ smtp_port }}"
SMTP_STARTTLS = True
SMTP_USERNAME = "{{ smtp_username }}"

CORS_ORIGINS = ["*"]

Here's my Dockerfile:

FROM alerta/alerta-web:7.4.5

EXPOSE 8080

RUN /venv/bin/pip install git+https://github.com/alerta/alerta-contrib.git#subdirectory=integrations/mailer

RUN /venv/bin/pip install dnspython==1.16.0

COPY . ${WORKDIR}

Environment

• OS: Linux

• API version: 7.4.5

• Deployment: Docker

• Database: MongoDB

• Server config:
  Auth enabled? Yes
  Auth provider? Keycloak
  Customer views? No

• web UI version: 7.4.5

• CLI version: 7.4.0

To Reproduce
Steps to reproduce the behavior:

1. Install AMQP and mailer plugins
2. Configure (see files above) ^^^
3. Run alerta-mailer

For web app issues, include any web browser JavaScript console errors.

• I see no JS errrors

Expected behavior
Alerta-mailer can authenticate and query the heartbet successfully

Screenshots
N/A

Additional context
Here's a gist of the configs when I asked on the Gitter: https://gist.github.com/decentral1se/8d6551909a7d6b25826d4ff8ca32c364.

[decentral1se]

Oh wait...

>>> api = Client(endpoint="http://localhost:8080/api", key="...mykey...")
>>> api.heartbeat("alerta-mailer/...mysmtphost...")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib/python3.7/site-packages/alertaclient/api.py", line 211, in heartbeat
    r = self.http.post('/heartbeat', data)
  File "/venv/lib/python3.7/site-packages/alertaclient/api.py", line 515, in post
    return self._handle_error(response)
  File "/venv/lib/python3.7/site-packages/alertaclient/api.py", line 543, in _handle_error
    raise UnknownError(resp['message'])
alertaclient.exceptions.UnknownError: Missing required scope: write:heartbeats

So, I had the wrong scope! I only gave "read". So, this is something that can be documented then if you'll accept the PR? We could also perhaps improve the error message for this heartbeats request error message?

alerta-contrib/integrations/mailer/mailer.py

Line 197 in edf6326

api.heartbeat(origin, tags=[__version__])

However, the "Authentication failed" notice remains, so that now narrows it down to amqp or smtp... I will keep digging...

[decentral1se]

Aha,

>>> conn = Connection("mongodb://alerta:mypw@myhost/kombu")
>>> import mailer
>>> consumer = mailer.FanoutConsumer(connection=conn)
Traceback (most recent call last):
  File "/venv/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 921, in create_channel
    return self._avail_channels.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib/python3.7/site-packages/mailer.py", line 91, in __init__
    self.channel = self.connection.channel()
  File "/venv/lib/python3.7/site-packages/kombu/connection.py", line 288, in channel
    chan = self.transport.create_channel(self.connection)
  File "/venv/lib/python3.7/site-packages/kombu/connection.py", line 839, in connection
    self._connection = self._establish_connection()
  File "/venv/lib/python3.7/site-packages/kombu/connection.py", line 794, in _establish_connection
    conn = self.transport.establish_connection()
  File "/venv/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 941, in establish_connection
    self._avail_channels.append(self.create_channel(self))
  File "/venv/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 923, in create_channel
    channel = self.Channel(connection)
  File "/venv/lib/python3.7/site-packages/kombu/transport/mongodb.py", line 120, in __init__
    self.client
  File "/venv/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/venv/lib/python3.7/site-packages/kombu/transport/mongodb.py", line 359, in client
    return self._create_client()
  File "/venv/lib/python3.7/site-packages/kombu/transport/mongodb.py", line 351, in _create_client
    database = self._open()
  File "/venv/lib/python3.7/site-packages/kombu/transport/mongodb.py", line 310, in _open
    version_str = mongoconn.server_info()['version']
  File "/venv/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1623, in server_info
    session=session)
  File "/venv/lib/python3.7/site-packages/pymongo/database.py", line 611, in command
    read_preference) as (sock_info, slave_ok):
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/venv/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1101, in _socket_for_reads
    with self._get_socket(server) as sock_info:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/venv/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1058, in _get_socket
    with server.get_socket(self.__all_credentials) as sock_info:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/venv/lib/python3.7/site-packages/pymongo/pool.py", line 1006, in get_socket
    sock_info.check_auth(all_credentials)
  File "/venv/lib/python3.7/site-packages/pymongo/pool.py", line 677, in check_auth
    auth.authenticate(credentials, self)
  File "/venv/lib/python3.7/site-packages/pymongo/auth.py", line 563, in authenticate
    auth_func(credentials, sock_info)
  File "/venv/lib/python3.7/site-packages/pymongo/auth.py", line 540, in _authenticate_default
    return _authenticate_scram(credentials, sock_info, 'SCRAM-SHA-1')
  File "/venv/lib/python3.7/site-packages/pymongo/auth.py", line 262, in _authenticate_scram
    res = sock_info.command(source, cmd)
  File "/venv/lib/python3.7/site-packages/pymongo/pool.py", line 579, in command
    unacknowledged=unacknowledged)
  File "/venv/lib/python3.7/site-packages/pymongo/network.py", line 150, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/venv/lib/python3.7/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Authentication failed.

So, this consumer invocation could also bring a better error message. Still investigating how to solve this one...

[decentral1se]

Ok, so since I have a "alerta" channel created on the mongodb side and not "kombu", then I was finally able to connect by specifying my amqp_url for both mailer and amqp to "/alerta". I am now manually testing this works.

So, I guess this is a documentation issue.

[decentral1se]

Also, to note, I'm running it by overriding the entrypoint and doing nohup alerta-mailer &. I wonder if that is the best way? I am not sure but it would certainly be good to document this approach or?

[satterly]

The best approach would be to install alerta-mailer into a dedicated container and not into the same container as the Alerta API and web UI.

Fixed 4387599