Alerts not RFC 5322 compliant. Google's gmail blocks them

[JTMosaic]

Wazuh 4.3.7 | wazuh-maild | Linux (Amazon2)

When maild.grouping =1 in internal_options.conf and an alert for Shellshock is sent (rule id 31168), the resulting email is not RFC 5322 compliant and Google rejects it:

relay=aspmx.l.google.com[142.250.31.27]:25, delay=0.57, delays=0.05/0.02/0.15/0.35, dsn=5.7.1, status=bounced (host aspmx.l.google.com[142.250.31.27] said: 550-5.7.1 [52.44.133.207] Our system has detected that this message is not RFC 550-5.7.1 5322 compliant: duplicate headers. To reduce the amount of spam sent 550-5.7.1 to Gmail, this message has been blocked. Please review 550 5.7.1 RFC 5322 specifications for more information. eq1-20020ad45961000000b00498ff1cac52si697465qvb.479 - gsmtp (in reply to end of DATA command))

However, other alerts, such as rule id 100005, do not run afoul of this issue.

If maild.grouping is set to 0, the problem goes away so there is a workaround, though the default is 1 and this could cause alerts to be missed.

There isn't a debug setting for maild as far as I can see.

[FrancoRivero]

Hi @JTMosaic,
Thank you for using Wazuh!
Well we can use to debug maild its daemon, with this configuration we are able to see more information about the maild module. However this can be a bug if you want to collect this information it would be a good catch to debug and fix this problem.
Maybe it can be a lot of information but you can use wazuh-control to debug each module(read this document ).
Regards

[JTMosaic]

Thank you for the link to using wazuh-control.

However, this led me to a very strange situation.

With debug enabled, the problem does not happen and the mail is sent correctly.
With debug off, the mail is blocked by Google.

Here's what I did:

1. Generate the alert by echoing the pattern into a watched log
2. Did not receive the alert though Wazuh indicated a level 12 or higher alert
3. Checked the maillog on the Wazuh server and found the blocked message
4. Ran /var/ossec/wazuh-control enable debug
5. systemctl restart wazuh-manager
6. Generate the alert again
7. Received the email

This certainly does sound like a bug but I'm not sure how to gather trace or error information

Can you tell me where wazuh-maild should log to?

[JTMosaic]

I did some more troubleshooting, including killing wazuh-maild and starting it in the foreground with debug flags.

Unfortunately, in that mode I cannot get the mail to be blocked. However, as soon as I restart all of Wazuh with debug disabled, I can replicate the issue.

I've also narrowed down the issue a bit further and found that it only occurs if there are more than one <email_to> entries in ossec.conf's global section.

[JTMosaic]

There is very little data in the log and there is no difference between a message which is blocked and one that is not:

Non-Blocked message

2022/09/08 09:41:14 wazuh-maild[16842] os_maild_client.c:218 at OS_RecvMailQ(): DEBUG: OS_RecvMailQ: mail->body[
Wazuh Notification.
2022 Sep 08 09:41:10

Received From: (xxxxx) any->/var/log/nginx/access.log
Rule: 31168 fired (level 15) -> "Shellshock attack detected"
Src IP: 10.22.11.28
Portion of the log(s):

10.22.11.28 - - [06/Sep/2022:17:19:26 -0400] "GET /cgi-bin/slogin/login.py HTTP/1.1" 200 100 "-" "() { :; }; echo ; echo ; /bin/cat /etc/passwd" "123.123.123.123"

--END OF NOTIFICATION

]

Sep 8 09:41:24 roc-log postfix/smtp[16875]: 26EC9BDDA5: to=, relay=aspmx.l.google.com[172.253.63.26]:25, delay=0.49, delays=0.05/0/0.05/0.4, dsn=2.0.0, status=sent (250 2.0.0 OK 1662644484 g14-20020a05620a40ce00b006b949c6203csi12283435qko.38 - gsmtp)
Sep 8 09:41:24 roc-log postfix/smtp[16875]: 26EC9BDDA5: to=, relay=aspmx.l.google.com[172.253.63.26]:25, delay=0.49, delays=0.05/0/0.05/0.4, dsn=2.0.0, status=sent (250 2.0.0 OK 1662644484 g14-20020a05620a40ce00b006b949c6203csi12283435qko.38 - gsmtp)
Sep 8 09:41:24 roc-log postfix/qmgr[3160]: 26EC9BDDA5: removed

Blocked message

2022/09/08 09:41:49 wazuh-maild[16842] os_maild_client.c:218 at OS_RecvMailQ(): DEBUG: OS_RecvMailQ: mail->body[
Wazuh Notification.
2022 Sep 08 09:41:46

Received From: (xxxxxx) any->/var/log/nginx/access.log
Rule: 31168 fired (level 15) -> "Shellshock attack detected"
Src IP: 10.22.11.28
Portion of the log(s):

10.22.11.28 - - [06/Sep/2022:17:19:26 -0400] "GET /cgi-bin/slogin/login.py HTTP/1.1" 200 100 "-" "() { :; }; echo ; echo ; /bin/cat /etc/passwd" "123.123.123.123"

--END OF NOTIFICATION

]

Sep 8 09:41:59 roc-log postfix/smtp[16875]: 2EDC6BDDA5: to=, relay=aspmx.l.google.com[172.253.63.26]:25, delay=0.66, delays=0.04/0/0.04/0.57, dsn=5.7.1, status=bounced (host aspmx.l.google.com[172.253.63.26] said: 550-5.7.1 [52.44.133.207] Our system has detected that this message is not RFC 550-5.7.1 5322 compliant: duplicate headers. To reduce the amount of spam sent 550-5.7.1 to Gmail, this message has been blocked. Please review 550 5.7.1 RFC 5322 specifications for more information. do13-20020a05620a2b0d00b006cbd617d7c7si669532qkb.9 - gsmtp (in reply to end of DATA command))
Sep 8 09:41:59 roc-log postfix/smtp[16875]: 2EDC6BDDA5: to=, relay=aspmx.l.google.com[172.253.63.26]:25, delay=0.66, delays=0.04/0/0.04/0.57, dsn=5.7.1, status=bounced (host aspmx.l.google.com[172.253.63.26] said: 550-5.7.1 [52.44.133.207] Our system has detected that this message is not RFC 550-5.7.1 5322 compliant: duplicate headers. To reduce the amount of spam sent 550-5.7.1 to Gmail, this message has been blocked. Please review 550 5.7.1 RFC 5322 specifications for more information. do13-20020a05620a2b0d00b006cbd617d7c7si669532qkb.9 - gsmtp (in reply to end of DATA command))

[FrancoRivero]

I will test this problem in order to detect what is the bug.
Can you share me your settings?
I will set up the environment to test this problem.
Regards

[JTMosaic]

I can probably do that. Do you want ossec.conf, then?

Not sure if it matters but this was a 4.2.5 with open distro for elasticsearch upgraded to 4.3.7 and the wazuh components

[FrancoRivero]

Hi @JTMosaic,
I need to test the same environment if we want to debug the problem, we need to know the applied configuration (ossec.conf).
So the update is an important detail, did you face this error after the update?
Regards

[JTMosaic]

Correct, @FrancoRivero. The problem only started after the upgrade.

I've attached our ossec.conf. Note that I've removed our email addresses and replaced them with placeholders

Our organization's mail domain is hosted on gmail and that is where the issue with the RFC comes in.

Thank you for looking into this!

ossec.conf.zip

[FrancoRivero]

Hi @JTMosaic,
Sorry for the late reply.
I had a lot of work this week, I'm trying to reproduce the bug so I can't reproduce it yet!
Have you run the test again? Because when you used the debug flag the mail was being sent, however, checking the code the debug flag only adds debug messages. This is a strange case so I want to help you and fix the problem if it is a bug in our module. Is there any alert that works fine? If only this event is failing the alert header could be the problem here!
Regards

[JTMosaic]

@FrancoRivero, thank you for all your effort!

I have noticed that the issue only occurs when there are multiple <email_to> entries in ossec.conf and maild.grouping=0 in internal_options.conf. We had two but have switched to a group email address and one entry as a solution. Maybe this is the issue? If Wazuh is creating an email with multiple TO fields rather than serializing the addresses in one field that may violate the RFC

As for testing, it is possible the debug flag was a false indicator as I was able to send a few alerts with it off and a few were blocked with it on.

Unfortunately, Wazuh doesn't have a "test alert" functionality that I am aware of so I can't test all possible alerts. I know it happened for the Shellshock alert mentioned above so that is the only one I tested with.

[FrancoRivero]

Hi @JTMosaic,
Sorry for the delay in replying.
I have been working on this issue with some colleagues, however we have not been able to reproduce it.
We tested configuring maild to send alerts with a shell shock attack in these scenarios:

• Updated from version 4.2.5 to 4.3.7
• Clean install 4.2.5 version
• Clean install 4.3.7 version

Although I cannot reproduce the issue, unfortunately we need more time to analyse it, so we are planning to add it to our roadmap.
I would need as much information as possible, as I may or may not pick up the issue, and the next person who picks it up will need information to try to solve the problem.
Can you repeat to me in the next message the information we discussed in this topic, so that I can keep the information tidy?

• Manager's OS
• Email server
• Settings used in your system
• Detailed steps to reproduce the problem

With this information I or someone working on this topic may be able to try to reproduce and fix the possible bug.
As a workaround to this issue, for the time being I recommend that you disable the problematic settings to get a clean log.

I am sorry for not being able to solve your problem at the moment but we will do our best to correct the problem and have a product with as few errors as possible.
Thank you very much !

Regards

[JTMosaic]

Thanks you for the update, @FrancoRivero. I appreciate all your efforts!

As I mentioned, we have workarounds (only using one email_to or setting maild.grouping to 0) so we are in a good working configuration at the moment.

The manager is running on Amazon Linux 2 (kernel 4)
The email server is on the same box and is postfix 2.10.1. We send mail directly from this box
In ossec.conf, in the section we have 2 <email_to> addresses configured. Both addresses point to domains hosted by gmail
In internal_options.conf, we have left the default maild settings of:

maild.strict_checking=1
maild.grouping=1
maild.full_subject=0
maild.geoip=1

To recreate the issue I am pushing a log entry for a shellshock attack into the webserver log on another box with wazuh agent.

It took me several tries this morning to replicate the issue with the first 3 alerts successfully sending but the 4th and 5th tries bounced and the 6th send successfully. The issue appears to be transient. Logs of the failure message from maillog are below (highlight added)

Sep 22 09:02:02 roc-log postfix/smtp[27455]: 86048BFD5D: to=<jt.shyman@>, relay=aspmx.l.google.com[142.251.16.27]:25, delay=0.45, delays=0.05/0/0.07/0.33, dsn=5.7.1, status=bounced (host aspmx.l.google.com[142.251.16.27] said: 550-5.7.1 [] Our system has detected that this message is not RFC 550-5.7.1 5322 compliant: duplicate headers. To reduce the amount of spam sent 550-5.7.1 to Gmail, this message has been blocked. Please review 550 5.7.1 RFC 5322 specifications for more information. i6-20020a05620a248600b006ced53b80e8si3697364qkn.131 - gsmtp (in reply to end of DATA command))

Sep 22 09:04:32 roc-log postfix/smtp[27706]: B2F65BFD5D: to=<jt.shyman@>, relay=aspmx.l.google.com[142.251.16.27]:25, delay=0.66, delays=0.05/0.01/0.1/0.49, dsn=5.7.1, status=bounced (host aspmx.l.google.com[142.251.16.27] said: 550-5.7.1 [] Our system has detected that this message is not RFC 550-5.7.1 5322 compliant: duplicate headers. To reduce the amount of spam sent 550-5.7.1 to Gmail, this message has been blocked. Please review 550 5.7.1 RFC 5322 specifications for more information. c17-20020a05622a059100b0035cd0208541si3174461qtb.364 - gsmtp (in reply to end of DATA command))

However, with only 1 <email_to> value I cannot get the failure to occur at all.

Thanks for your continued efforts on this!

[if-jeremy]

We are facing this same issue currently. Here's some additional information on our setup:

Wazuh Manager 4.3.7 running on CentOS 7.
Local MTA is Postfix, with a standard CentOS 7 configuration

Our current ossec.conf on the manager truncated for brevity looks like:

<ossec_config>
  <global>
    <jsonout_output>yes</jsonout_output>
    <alerts_log>yes</alerts_log>
    <logall>no</logall>
    <logall_json>no</logall_json>
    <email_notification>yes</email_notification>
    <smtp_server>localhost</smtp_server>
    <email_from>ossec@my.domain.com</email_from>
    <email_to>wazuh@my.domain.com</email_to>
    <email_maxperhour>1000000</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    <memory_size>2048</memory_size>
  </global>

  <syslog_output>
    <server>172.31.92.15</server>
    <port>514</port>
    <level>7</level>
  </syslog_output>

  <syslog_output>
    <server>172.31.92.52</server>
    <port>514</port>
    <level>7</level>
  </syslog_output>

  <syslog_output>
    <server>172.31.92.53</server>
    <port>514</port>
    <level>7</level>
  </syslog_output>

  <alerts>
    <log_alert_level>7</log_alert_level>
    <email_alert_level>7</email_alert_level>
  </alerts>

  <email_alerts>
    <email_to>pager@my.domain.com</email_to>
    <level>10</level>
    <do_not_delay />
    <do_not_group />
  </email_alerts>
...

The internal_options.conf file is unmodified from it's stock configuration.

The idea behind this setup is that alerts of level 7 and above are sent to "wazuh@my.domain.com", and level 10 and above are also sent to "pager@my.domain.com".

However, for the case of level 10 and above, the resulting messages are not compliant with RFC 5322, as they result in multiple "To" lines in the header:

Header contents of a level 7 email (excluding MTA added data):

To: <wazuh@my.domain.com>
From: Wazuh <ossec@my.domain.com>
Date: Thu, 26 Jan 2023 15:34:40 -0500
Subject: Wazuh notification - (host.domain.com) any - Alert level 7
Message-Id: <20230126203440.A19984D4@my.domain.com>

While the same from a level 10 message looks as follows:

To: <wazuh@my.domain.com>
From: Wazuh <ossec@my.domain.com>
To: <pager@my.domain.com>
Date: Thu, 26 Jan 2023 15:39:40 -0500
Subject: Wazuh notification - (host.domain.com) any - Alert level 10
Message-Id: <20230126203940.F1AE9131@my.domain.com>

Notice the multiple To: lines in the level 10 email - this is a violation of RFC 5322, which specifies only one To line in the headers. If multiple addresses are listed, they should be all on a single To line, separated by commas. Also, this is the "Header-To", and not "Envelope-To" - so it is provided by the Mail User Agent (MUA), and not the MTA - so this is defined by Wazuh, and not Sendmail/Postfix.

We've been able to easily replicate this on our host by setting email_alert_level to 7, and email_alerts level 10 sending to "root", then triggering a level 10 alert (like a SSH password failure). The result will be a message in /var/mail/root, so you can do cat /var/mail/root to see the message and it's headers easily.

This appears to be a change made to Google's email validation sometime after Sept 22, 2022 - as we received one of these emails on that date, and none after that date. But, as my analysis clearly shows, the problem is definately with how Wazuh composes their Emails when level-based granular email alerts are enabled, and not with the MTA or Google themselves.

This is a major problem when using level-based granular email alerts and at least Google mailboxes as recipients. IMHO, this should be considered as a Severe bug.

[wilfredt]

I have the same issue of being emails blocked by Gmail. Bounced back emails by Gmail as follows:

Original-Recipient: rfc822; XXXX@gmail.com
Final-Recipient: rfc822; XXXX@gmail.com
Status: 550
Action: failed
Last-Attempt-Date: 29 Jan 2023 11:06:59 GMT
Diagnostic-Code: 5.7.1 [136.143.188.12] This message is not RFC 5322 compliant, the issue is:
5.7.1 duplicate To headers. To reduce the amount of spam sent to Gmail,
5.7.1 this message has been blocked. Please review
5.7.1 RFC 5322 specifications for more information. r13-20020a17090a1bcd00b0022bfb225af4si10146289pjr.53 - gsmtp

Original-Recipient: rfc822; YYYY@gmail.com
Final-Recipient: rfc822; YYYY@gmail.com
Status: 550
Action: failed
Last-Attempt-Date: 29 Jan 2023 11:06:59 GMT
Diagnostic-Code: 5.7.1 [136.143.188.12] This message is not RFC 5322 compliant, the issue is:
5.7.1 duplicate To headers. To reduce the amount of spam sent to Gmail,
5.7.1 this message has been blocked. Please review
5.7.1 RFC 5322 specifications for more information. r13-20020a17090a1bcd00b0022bfb225af4si10146289pjr.53 - gsmtp

This happens whenever a email is set in <email_alerts> which is different from the main global section:

<ossec_config>

<email_notification>yes</email_notification>
...
<email_to>XXXX@gmail.com</email_to>
...

...

<email_alerts>
<email_to>YYYY@gmail.com</email_to>
...
</email_alerts>
...

This effectively makes the <email_alerts> unusable. I also agree this is a major bug to fix.

[Icare-github]

Same issue here with the

Ubuntu 22.02 with v4.4.5

Email header:
Received: from notify.ossec.net (localhost [127.0.0.1]) by xxx (Postfix) with SMTP id 49E; Wed, 2 Aug 2023 14:29:03 +0000 (UTC)
To: xxx@xxx.com
To: xxx@xxx.com
Date: Wed, 02 Aug 2023 14:29:03 +0000

Standard email_alerts configuration.

This makes the <email_alerts> unusable which is an important part of the XDR. Waiting for a fix.

[joschneid]

And the same issue here.

Wazuh Manager 4.4.5 running on Debian 11 with the following ossec.conf on the manager:

  <global>
    <jsonout_output>yes</jsonout_output>
    <alerts_log>yes</alerts_log>
    <logall>no</logall>
    <logall_json>no</logall_json>
    <email_notification>yes</email_notification>
    <email_to>me@xxx.xx</email_to>
    <smtp_server>localhost</smtp_server>
    <email_from>wazuh-server@xxx.xx</email_from>
    <email_maxperhour>300</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    <agents_disconnection_time>20s</agents_disconnection_time>
    <agents_disconnection_alert_time>100s</agents_disconnection_alert_time>
  </global>

  <alerts>
    <log_alert_level>4</log_alert_level>
    <email_alert_level>7</email_alert_level>
  </alerts>

  <logging>
    <log_format>plain</log_format>
  </logging>

  <email_alerts>
    <email_to>me@xxx.xx</email_to>
    <format>full</format>
    <rule_id>550, 553, 554</rule_id>
  </email_alerts>

with an additional overwrite in local_rules.yml

<!--
     overwrite severity for file added
-->
     
<group name="ossec,">
  <rule id="554" level="7" overwrite="yes">
    <category>ossec</category>
    <decoded_as>syscheck_new_entry</decoded_as>
    <description>File added to the system.</description>
    <group>syscheck,pci_dss_11.5,gpg13_4.11,gdpr_II_5.1.f,hipaa_164.312.c.1,hipaa_164.312.c.2,nist_800_53_SI.7,</group>
  </rule>
</group>

The idea is that we get all FIM related changes via mail.

Maybe a hint where to search:

The error / double to-header occures only on the mails send via the <email_alerts> section. All other Wazuh related mails like Daily report, SCA-Mails or for example an alert for postfix restart on the Wazuh Server do work fine and have only one to-header.

[jrpedrianes]

Same problem here using wazuh v4.5.4 version. Any update on this error?