Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Problem Sending Mail Without Smtp Auth on OpenSearch #890

Open
espala opened this issue Feb 21, 2024 · 5 comments
Open

[BUG] Problem Sending Mail Without Smtp Auth on OpenSearch #890

espala opened this issue Feb 21, 2024 · 5 comments
Labels
bug Something isn't working

Comments

@espala
Copy link

espala commented Feb 21, 2024

Hello everyone,

I am using Opensearch cluster and my cluster is running smoothly. I don’t get any error messages or logs in the standard run cycle.

I wanted to define an alarm specifically, so I defined a new “Monitors” definition from the “Alerting” menu. Then I define “Trigger” and select my “Smtp Channel” definition.

Everything works correctly in the Alerting mechanism, it finds the word I filtered and runs the trigger function. But it cannot send an email using my smtp information I entered in the “Notification” section. Because it is trying to do smtp authentication.

But I selected the “Encryption Method” as “none” in the “Email Senders” section under the “Notifications” menu. When I click the “send test email” button both in the “Channel” I defined and in the Alerting > Monitors > Triggers I defined, I see that it sends emails. I can also see the emails in my inbox.

But when “Alerting > Monitors > Triggers > Action” automatically sends an email, it tries to establish smtp authentication connection.

I want to send email without smtp auth. There is also “none” option on OpenSearch Panel. I think we have such a possibility. But I see in the log that I get a 503 error. 503 here is an error code representing smtp auth.

I am using an internal smtp relay server and I am not using the smtp auth mechanism. I need to be able to send emails without smtp auth as the Opensearch system allows me.

I think this problem is a bug. How can I solve this problem?

My first OpenSearch forum case;
https://forum.opensearch.org/t/about-mail-sending-problem-on-alerting-notification-section/17786

My second case detailing the problem;
https://forum.opensearch.org/t/problem-sending-mail-without-smtp-auth-on-opensearch-critical/17993

screenshot of my smtp definition;
Screen Shot 2024-02-21 at 07 58 21
Screen Shot 2024-02-21 at 08 17 14

screenshot of my Alerting/Monitors definition;
Screen Shot 2024-02-21 at 08 13 47
Screen Shot 2024-02-21 at 08 13 55

Log;
Error: Failed running action: OpenSearchStatusException[{"event_status_list": [{"config_id":"Qzjsho0B2TBnxA0NWL9r","config_type":"email","config_name":"Test Email Notification Channel","email_recipient_status":[{"recipient":"myaddress@mydomain.com","delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}],"delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}]}]


Opensearch and my linux versions

[root@server6 ~]# yum list installed |grep opensearch
opensearch.x86_64                             2.11.1-1                                  @opensearch-2.x
opensearch-dashboards.x86_64                  2.11.1-1                                  @opensearch-dashboards-2.x

[root@server6 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)

Related sections in my logs

...
[2024-02-21T07:59:52,113][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.112Z, periodEnd: 2024-02-21T04:59:52.112Z, dryrun: true, executionId: _2024-02-21T04:59:52.113348729_cf70ca80-2682-4222-936f-b830f3ba42c1
[2024-02-21T07:59:52,162][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.161Z, periodEnd: 2024-02-21T04:59:52.161Z, dryrun: true
[2024-02-21T07:59:52,163][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.161Z, periodEnd: 2024-02-21T04:59:52.161Z, dryrun: true, executionId: _2024-02-21T04:59:52.163613799_ed76123f-6835-4285-8d35-3967a9c53af0
[2024-02-21T07:59:52,302][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.301Z, periodEnd: 2024-02-21T04:59:52.301Z, dryrun: true
[2024-02-21T07:59:52,302][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.301Z, periodEnd: 2024-02-21T04:59:52.301Z, dryrun: true, executionId: _2024-02-21T04:59:52.302626749_4da076b3-d59b-4128-9666-5af0273ce234
[2024-02-21T07:59:52,339][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.339Z, periodEnd: 2024-02-21T04:59:52.339Z, dryrun: true
[2024-02-21T07:59:52,340][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.339Z, periodEnd: 2024-02-21T04:59:52.339Z, dryrun: true, executionId: _2024-02-21T04:59:52.339981829_405f8627-c4d2-45c8-95be-9f636852f566
[2024-02-21T07:59:56,935][INFO ][o.o.a.t.TransportIndexMonitorAction] [server6] Updating monitor, hq0Xl40BzHB4GwYe75E5, from: {"monitor":{"type":"monitor","schema_version":8,"name":"opensearch Disaster  Error Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707914606685,"schedule":{"period":{"interval":5,"unit":"MINUTES"}},"inputs":[{"search":{"indices":["opensearch-disaster-*"],"query":{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"{{period_end}}||-3h","to":"{{period_end}}","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"level":{"query":"error","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"CzSzp40BPraQSw1iuOEJ","name":"Email Trigger","severity":"1","condition":{"script":{"source":"ctx.results[0].hits.total.value > 1","lang":"painless"}},"actions":[{"id":"notification915423","name":"act","destination_id":"Qzjsho0B2TBnxA0NWL9r","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"Alerting Notification action","lang":"mustache"}}]}}],"last_update_time":1708424762064,"ui_metadata":{"schedule":{"cronExpression":"0 */1 * * *","period":{"unit":"MINUTES","interval":5},"timezone":null,"daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"interval"},"search":{"searchType":"graph","bucketValue":3,"timeField":"@timestamp","bucketUnitOfTime":"h","filters":[{"fieldName":[{"label":"level","type":"text"}],"fieldValue":"error","operator":"is"}],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"Email Trigger":{"value":1,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
 to: {"monitor":{"type":"monitor","schema_version":8,"name":"opensearch Disaster  Error Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707914606685,"schedule":{"period":{"interval":5,"unit":"MINUTES"}},"inputs":[{"search":{"indices":["opensearch-disaster-*"],"query":{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"{{period_end}}||-7h","to":"{{period_end}}","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"level":{"query":"error","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"CzSzp40BPraQSw1iuOEJ","name":"Email Trigger","severity":"1","condition":{"script":{"source":"ctx.results[0].hits.total.value > 1","lang":"painless"}},"actions":[{"id":"notification915423","name":"act","destination_id":"Qzjsho0B2TBnxA0NWL9r","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"Alerting Notification action","lang":"mustache"}}]}}],"last_update_time":1708491595830,"ui_metadata":{"schedule":{"cronExpression":"0 */1 * * *","period":{"unit":"MINUTES","interval":5},"timezone":null,"daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"interval"},"search":{"searchType":"graph","bucketValue":7,"timeField":"@timestamp","bucketUnitOfTime":"h","filters":[{"fieldName":[{"label":"level","type":"text"}],"fieldValue":"error","operator":"is"}],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"Email Trigger":{"value":1,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
[2024-02-21T07:59:58,104][INFO ][o.o.a.r.RestSearchMonitorAction] [server6] The monitor parsing failed. Will return response as is.
...

2. Subject
At the same time, when it can’t find the word I filtered for the alarm in “Alerting” > “Monitor”, it says that the “Trigger” and “Action” functions are working in the Alerts section and shows that it sent the email. But this is not true either.

My filter works correctly. I am looking for the word “error” in the log data for the last 3 hours. When it really can’t find the word “error” in the last 3 hours, the “History” section goes to green “no alerts”. This works correctly.

But when I change the status of the alarms under the “Alerts” heading at the bottom of the dashboard to “Completed”, it shows it as “Email Trigger”.

In this case, it does not seem to reflect an true result. Because even though there was an alarm trigger, the action gave an error and could not send the email.

Screen Shot 2024-02-21 at 07 59 34

@ghost
Copy link

ghost commented Feb 22, 2024

which OS version do you use?

@kavilla
Copy link
Member

kavilla commented Feb 22, 2024

Hello @espala,

Thank you for opening. I have pinged the team as well.

@opensearch-project/admin please re-route to https://github.com/opensearch-project/alerting-dashboards-plugin

@espala
Copy link
Author

espala commented Feb 23, 2024

which OS version do you use?

Hi, I actually wrote above.
I am using Redhat version 8.8, OpenSearch and OpenSearch Dashboards version 2.11.1-1

[root@server6 ~]# yum list installed |grep opensearch
opensearch.x86_64                             2.11.1-1                                  @opensearch-2.x
opensearch-dashboards.x86_64                  2.11.1-1                                  @opensearch-dashboards-2.x

[root@server6 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)

Hello @espala,

Thank you for opening. I have pinged the team as well.

@opensearch-project/admin please re-route to https://github.com/opensearch-project/alerting-dashboards-plugin

@kavilla thank you. We will be waiting for information from you.

@wbeckler wbeckler transferred this issue from opensearch-project/OpenSearch-Dashboards Feb 23, 2024
@AWSHurneyt
Copy link
Collaborator

Hi @espala,
Would you be able to provide the JSON payloads for the monitor, and the notification channel (scrubbed of any sensitive info of course)?

To retrieve the monitor JSON, you can run the following command in devtools.

GET _plugins/_alerting/monitors/_search
{
  "query": {
    "match" : {
      "monitor.name": "Orion Disaster Error Monitor"
    }
  }
}

And from the logs provided above, it looks like the ID for your Smtp Channel is Qzjsho0B2TBnxA0NWL9r. The following command can be used to retrieve the JSON for that channel.

GET _plugins/_notifications/configs/Qzjsho0B2TBnxA0NWL9r

If it looks like the channel returned by that command isn't the correct one, you can use the following command to get all of the notification channels configured on the cluster; I would just need the JSON for your Smtp Channel.

GET _plugins/_notifications/configs

@espala
Copy link
Author

espala commented Feb 26, 2024

@AWSHurneyt Thank you very much for your response and interest. I'm sorry I've had a rough weekend and I'm responding late.

I queried all the paths you forwarded and wrote them all below. Can you check it?

By the way, I changed the sensitive data in the output. It may be different from the information I gave when opening the case before. If there is a difference, please ignore those parts (mail address, domain, etc.)

query 1

GET _plugins/_alerting/monitors/_search
{
  "query": {
    "match" : {
      "monitor.name": "Orion Disaster Error Monitor"
    }
  }
}

results

{
  "took": 19,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 2.1507282,
    "hits": [
      {
        "_index": ".opendistro-alerting-config",
        "_id": "hq0Xl40BzHB4GwYe75E5",
        "_version": 16,
        "_seq_no": 81,
        "_primary_term": 2,
        "_score": 2.1507282,
        "_source": {
          "type": "monitor",
          "schema_version": 8,
          "name": "Orion Disaster  Error Monitor",
          "monitor_type": "query_level_monitor",
          "enabled": true,
          "enabled_time": 1707914606685,
          "schedule": {
            "period": {
              "interval": 5,
              "unit": "MINUTES"
            }
          },
          "inputs": [
            {
              "search": {
                "indices": [
                  "orion-disaster-*"
                ],
                "query": {
                  "size": 0,
                  "query": {
                    "bool": {
                      "filter": [
                        {
                          "range": {
                            "@timestamp": {
                              "from": "{{period_end}}||-7h",
                              "to": "{{period_end}}",
                              "include_lower": true,
                              "include_upper": true,
                              "format": "epoch_millis",
                              "boost": 1
                            }
                          }
                        },
                        {
                          "match_phrase": {
                            "level": {
                              "query": "error",
                              "slop": 0,
                              "zero_terms_query": "NONE",
                              "boost": 1
                            }
                          }
                        }
                      ],
                      "adjust_pure_negative": true,
                      "boost": 1
                    }
                  }
                }
              }
            }
          ],
          "triggers": [
            {
              "query_level_trigger": {
                "id": "CzSzp40BPraQSw1iuOEJ",
                "name": "Email Trigger",
                "severity": "1",
                "condition": {
                  "script": {
                    "source": "ctx.results[0].hits.total.value > 1",
                    "lang": "painless"
                  }
                },
                "actions": [
                  {
                    "id": "notification915423",
                    "name": "act",
                    "destination_id": "Qzjsho0B2TBnxA0NWL9r",
                    "message_template": {
                      "source": """Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.
  - Trigger: {{ctx.trigger.name}}
  - Severity: {{ctx.trigger.severity}}
  - Period start: {{ctx.periodStart}}
  - Period end: {{ctx.periodEnd}}""",
                      "lang": "mustache"
                    },
                    "throttle_enabled": false,
                    "subject_template": {
                      "source": "Alerting Notification action",
                      "lang": "mustache"
                    }
                  }
                ]
              }
            }
          ],
          "last_update_time": 1708491595830,
          "data_sources": {
            "query_index": ".opensearch-alerting-queries",
            "findings_index": ".opensearch-alerting-finding-history-write",
            "findings_index_pattern": "<.opensearch-alerting-finding-history-{now/d}-1>",
            "alerts_index": ".opendistro-alerting-alerts",
            "alerts_history_index": ".opendistro-alerting-alert-history-write",
            "alerts_history_index_pattern": "<.opendistro-alerting-alert-history-{now/d}-1>",
            "query_index_mappings_by_type": {},
            "findings_enabled": false
          },
          "owner": "alerting"
        }
      }
    ]
  }
}

Query 2
GET _plugins/_notifications/configs/

{
  "start_index": 0,
  "total_hits": 3,
  "total_hit_relation": "eq",
  "config_list": [
    {
      "config_id": "e3s6Wo0BJ92cGwouTc9p",
      "last_updated_time_ms": 1706615786856,
      "created_time_ms": 1706615786856,
      "config": {
        "name": "myaddress",
        "description": "",
        "config_type": "email_group",
        "is_enabled": true,
        "email_group": {
          "recipient_list": [
            {
              "recipient": "myaddress@mydomain.com"
            }
          ]
        }
      }
    },
    {
      "config_id": "Qzjsho0B2TBnxA0NWL9r",
      "last_updated_time_ms": 1708491388112,
      "created_time_ms": 1707365652587,
      "config": {
        "name": "Test Email Notification Channel",
        "description": "",
        "config_type": "email",
        "is_enabled": true,
        "email": {
          "email_account_id": "9Djrho0B2TBnxA0NZb6i",
          "recipient_list": [],
          "email_group_id_list": [
            "e3s6Wo0BJ92cGwouTc9p"
          ]
        }
      }
    },
    {
      "config_id": "9Djrho0B2TBnxA0NZb6i",
      "last_updated_time_ms": 1708491434491,
      "created_time_ms": 1707365590433,
      "config": {
        "name": "email-relay-server",
        "description": "",
        "config_type": "smtp_account",
        "is_enabled": true,
        "smtp_account": {
          "host": "smtp.mydomain.com",
          "port": 25,
          "method": "none",
          "from_address": "opensearch@mydomain.com"
        }
      }
    }
  ]
}

Query 3
GET _plugins/_notifications/configs/Qzjsho0B2TBnxA0NWL9r

{
  "start_index": 0,
  "total_hits": 1,
  "total_hit_relation": "eq",
  "config_list": [
    {
      "config_id": "Qzjsho0B2TBnxA0NWL9r",
      "last_updated_time_ms": 1708491388112,
      "created_time_ms": 1707365652587,
      "config": {
        "name": "Test Email Notification Channel",
        "description": "",
        "config_type": "email",
        "is_enabled": true,
        "email": {
          "email_account_id": "9Djrho0B2TBnxA0NZb6i",
          "recipient_list": [],
          "email_group_id_list": [
            "e3s6Wo0BJ92cGwouTc9p"
          ]
        }
      }
    }
  ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants